DEV Community: cheng zhang

Qwen3 Embedding on Cloud TPU: Production Long-Context Retrieval with vLLM

cheng zhang — Fri, 28 Aug 2026 01:54:23 +0000

Article Summary

Google Cloud published native vLLM TPU support for embedding inference on August 26, 2026, targeting production retrieval rather than chat generation. The engineering work focuses on Qwen3-Embedding-8B and Qwen3-VL-Embedding-8B with long text and multimodal contexts, including 16K-class text sequences and 15K+ multimodal inputs. Google addressed TPU tensor alignment, lazy loading, JAX/XLA compilation warm-up, chunked prefill, and pooling-state preservation through a hybrid StepPool design. In one published Qwen3-Embedding-8B configuration using bf16, 16K+ sequences, and TP=4, TPU Ironwood reached 83,996 total tokens/s and 5.13 requests/s. Google also validates cross-hardware vector parity with cosine-similarity thresholds of at least 0.999 for text and 0.995 for multimodal inputs.

Embedding infrastructure is easy to underestimate.

A prototype may look like:

documents
→ embedding API
→ vector database

Production can involve hundreds of millions of chunks, images, reindexing jobs, online queries, and multiple tenants.

At that point, embedding inference becomes a real serving platform.

Two different workloads

Indexing prioritizes token throughput.

Online query embedding prioritizes latency.

A mature platform needs both.

Why long-context embeddings matter

Modern retrieval increasingly wants long documents, multimodal pages, slide sections, and image-text pairs rather than 512-token snippets.

Google discusses text workloads above 4K tokens and multimodal inputs above 15K.

Long sequences increase memory pressure and make pooling correctness more difficult.

Why native vLLM TPU support matters

vLLM is already a mainstream open-source serving engine.

Adding TPU support lets teams use a more consistent serving stack across accelerator types instead of operating a separate TPU-only system.

Heterogeneous elasticity with GKE

Google describes prioritized capacity where TPU can be the primary pool and GPU capacity can serve as secondary fallback.

This is especially useful for bursty indexing workloads.

Embedding correctness is stricter than generation correctness

Small generation differences across hardware are often acceptable.

Embedding differences can alter nearest-neighbor ranking.

If vectors change materially, search results can change simply because the hardware backend changed.

Golden-reference testing

Let:

v_ref = reference embedding
v_tpu = TPU embedding

Then evaluate cosine similarity.

Google uses target thresholds of:

text >= 0.999
multimodal >= 0.995

That is a strict migration standard.

Do not benchmark only QPS

Before moving embedding inference across hardware, measure vector parity, Recall@K, NDCG, top-K overlap, and downstream business quality.

Faster infrastructure is not useful if retrieval quality silently changes.

Why chunked prefill is difficult for embeddings

Long inputs can exhaust accelerator memory.

Chunked prefill reduces peak memory by splitting the input across steps.

But embedding models still require one final pooled representation across the full sequence.

If pooling state is not accumulated correctly across chunks, the vector can be wrong without an obvious failure.

StepPool and cached state

Google’s hybrid StepPool design preserves pooling state across chunk boundaries and request preemption using cached request metadata.

This is an important example of the difference between code that runs and inference that remains mathematically correct.

Tensor alignment

TPU matrix units impose strict divisibility constraints during tensor parallel sharding.

Google added vocabulary padding so sharded execution remains hardware-safe while preserving logical output.

JAX/XLA warm-up

TPU serving frequently depends on compilation.

A production pod should not let its first real user pay the JIT cost.

A safer lifecycle is:

pod starts
→ model loads
→ compilation warm-up
→ health ready
→ traffic

Published throughput result

For one Qwen3-Embedding-8B configuration:

bf16
16K+ sequence
TP=4

Google reports:

83,996 total tokens/s
5.13 requests/s

This is a specific benchmark point, not a universal TPU number.

Why requests/s may look modest

Each request can contain thousands of tokens.

For long-context indexing, total token throughput can be more useful than raw request count.

Multimodal serving is harder

Qwen3-VL-Embedding combines text and image inputs. The current vLLM-TPU design chunks only the text portion of multimodal prefill, which highlights the extra complexity around visual features, pooling, and memory.

Recommended enterprise architecture

document pipeline
→ parser / chunker
→ embedding gateway
→ vLLM
   ├── TPU pool
   └── GPU fallback
→ vector database

Online query traffic should ideally use a separate low-latency pool.

Separate batch and online capacity

Large reindexing jobs can destroy online P99 latency if they share the same accelerator queue.

Use separate batch and online embedding pools with different scheduling objectives.

The embedding gateway should standardize

Track model version, vector dimension, normalization, maximum length, pooling method, and hardware backend.

Embedding versioning matters because different model versions produce different vector spaces.

Use dual indexes for model upgrades

Prefer:

old model → old index
new model → new index

Run shadow traffic, compare retrieval, reindex, and then cut over.

Do not mix a new query embedding with an old index blindly.

Is TPU always better?

No.

The decision depends on cloud platform, model support, workload shape, cost, and team expertise.

The strategic value of this release is that TPU becomes a first-class vLLM serving option.

Metrics that matter

Performance: tokens/s, requests/s, latency, queue time.

Quality: cosine parity, Recall@K, top-K overlap, NDCG.

Infrastructure: HBM, compile time, preemption, autoscaling.

Business: retrieval success and downstream answer quality.

Conclusion

The important change is not simply that Qwen3 embeddings can run on TPU.

Embedding inference is becoming independent production infrastructure with requirements for:

high throughput
+ long context
+ mathematical parity
+ elastic scaling
+ reproducibility

Google’s published configuration reaches 83,996 total tokens/s and 5.13 requests/s while applying strict cross-hardware cosine thresholds.

For production RAG, the key question is not “can the model run on another accelerator?” It is:

Can the system scale and change hardware without silently changing retrieval quality?

For more RAG, embedding, vLLM, and inference-infrastructure guidance, visit Zyentor Picks: https://www.zyentorpicks.com/.

Originally published on Zyentor Picks.

Qwen3 Embedding on Cloud TPU: Production Long-Context Retrieval with vLLM

cheng zhang — Thu, 27 Aug 2026 07:35:59 +0000

Article Summary

Embedding infrastructure is easy to underestimate.

A prototype may look like:

documents
→ embedding API
→ vector database

Production can involve hundreds of millions of chunks, images, reindexing jobs, online queries, and multiple tenants.

At that point, embedding inference becomes a real serving platform.

Two different workloads

Indexing prioritizes token throughput.

Online query embedding prioritizes latency.

A mature platform needs both.

Why long-context embeddings matter

Modern retrieval increasingly wants long documents, multimodal pages, slide sections, and image-text pairs rather than 512-token snippets.

Google discusses text workloads above 4K tokens and multimodal inputs above 15K.

Long sequences increase memory pressure and make pooling correctness more difficult.

Why native vLLM TPU support matters

vLLM is already a mainstream open-source serving engine.

Adding TPU support lets teams use a more consistent serving stack across accelerator types instead of operating a separate TPU-only system.

Heterogeneous elasticity with GKE

Google describes prioritized capacity where TPU can be the primary pool and GPU capacity can serve as secondary fallback.

This is especially useful for bursty indexing workloads.

Embedding correctness is stricter than generation correctness

Small generation differences across hardware are often acceptable.

Embedding differences can alter nearest-neighbor ranking.

If vectors change materially, search results can change simply because the hardware backend changed.

Golden-reference testing

Let:

v_ref = reference embedding
v_tpu = TPU embedding

Then evaluate cosine similarity.

Google uses target thresholds of:

text >= 0.999
multimodal >= 0.995

That is a strict migration standard.

Do not benchmark only QPS

Before moving embedding inference across hardware, measure vector parity, Recall@K, NDCG, top-K overlap, and downstream business quality.

Faster infrastructure is not useful if retrieval quality silently changes.

Why chunked prefill is difficult for embeddings

Long inputs can exhaust accelerator memory.

Chunked prefill reduces peak memory by splitting the input across steps.

But embedding models still require one final pooled representation across the full sequence.

If pooling state is not accumulated correctly across chunks, the vector can be wrong without an obvious failure.

StepPool and cached state

Google’s hybrid StepPool design preserves pooling state across chunk boundaries and request preemption using cached request metadata.

This is an important example of the difference between code that runs and inference that remains mathematically correct.

Tensor alignment

TPU matrix units impose strict divisibility constraints during tensor parallel sharding.

Google added vocabulary padding so sharded execution remains hardware-safe while preserving logical output.

JAX/XLA warm-up

TPU serving frequently depends on compilation.

A production pod should not let its first real user pay the JIT cost.

A safer lifecycle is:

pod starts
→ model loads
→ compilation warm-up
→ health ready
→ traffic

Published throughput result

For one Qwen3-Embedding-8B configuration:

bf16
16K+ sequence
TP=4

Google reports:

83,996 total tokens/s
5.13 requests/s

This is a specific benchmark point, not a universal TPU number.

Why requests/s may look modest

Each request can contain thousands of tokens.

For long-context indexing, total token throughput can be more useful than raw request count.

Multimodal serving is harder

Recommended enterprise architecture

document pipeline
→ parser / chunker
→ embedding gateway
→ vLLM
   ├── TPU pool
   └── GPU fallback
→ vector database

Online query traffic should ideally use a separate low-latency pool.

Separate batch and online capacity

Large reindexing jobs can destroy online P99 latency if they share the same accelerator queue.

Use separate batch and online embedding pools with different scheduling objectives.

The embedding gateway should standardize

Track model version, vector dimension, normalization, maximum length, pooling method, and hardware backend.

Embedding versioning matters because different model versions produce different vector spaces.

Use dual indexes for model upgrades

Prefer:

old model → old index
new model → new index

Run shadow traffic, compare retrieval, reindex, and then cut over.

Do not mix a new query embedding with an old index blindly.

Is TPU always better?

No.

The decision depends on cloud platform, model support, workload shape, cost, and team expertise.

The strategic value of this release is that TPU becomes a first-class vLLM serving option.

Metrics that matter

Performance: tokens/s, requests/s, latency, queue time.

Quality: cosine parity, Recall@K, top-K overlap, NDCG.

Infrastructure: HBM, compile time, preemption, autoscaling.

Business: retrieval success and downstream answer quality.

Conclusion

The important change is not simply that Qwen3 embeddings can run on TPU.

Embedding inference is becoming independent production infrastructure with requirements for:

high throughput
+ long context
+ mathematical parity
+ elastic scaling
+ reproducibility

Google’s published configuration reaches 83,996 total tokens/s and 5.13 requests/s while applying strict cross-hardware cosine thresholds.

For production RAG, the key question is not “can the model run on another accelerator?” It is:

Can the system scale and change hardware without silently changing retrieval quality?

For more RAG, embedding, vLLM, and inference-infrastructure guidance, visit Zyentor Picks: https://www.zyentorpicks.com/.

Originally published on Zyentor Picks.

In the AI Coding Era, Go’s Advantage Is Verifiability

cheng zhang — Tue, 25 Aug 2026 01:26:16 +0000

Article Summary

Google recently argued that AI-assisted software engineering changes what matters in a programming language. When coding agents can generate hundreds of lines in seconds, typing speed is no longer the dominant bottleneck. Reviewing, verifying, and maintaining generated code becomes more important. Go’s design aligns surprisingly well with that environment: gofmt, static typing, fast compilation, go test, native fuzzing, govulncheck, the module checksum database and mirror, gopls, and modernized go fix all provide deterministic feedback loops that agents can use to self-correct. The argument is not that Go is universally better than Python, Java, or Rust. It is that high-velocity AI generation increases the value of languages and toolchains that make errors cheap to detect.

Programming-language debates historically focus on how pleasant or fast code is to write.

AI coding changes the economics.

The cost of producing code is falling rapidly.

The expensive part becomes proving that generated code is correct, secure, and maintainable.

Generation is no longer the bottleneck

A coding agent can quickly create:

APIs;
services;
tests;
deployment files;
documentation.

But production teams still need to answer:

Does it compile?
Are the APIs real?
Are dependencies maintained?
Are error paths correct?
Does concurrency behave safely?
Will the code remain maintainable?

The new bottleneck is increasingly:

Generation: fast
Verification: expensive

`gofmt` reduces review noise

Go intentionally removes formatting choice.

Human-written and AI-written code pass through the same formatter:

human code
AI code
→ gofmt
→ consistent output

This sounds minor until many agents contribute to one repository.

Consistency lets reviewers focus on behavior rather than stylistic drift.

The compiler is a cheap deterministic critic

LLMs frequently invent methods, properties, and interfaces that do not exist.

In Go, many of those errors immediately fail compilation.

That creates a useful loop:

generate
→ go build / go test
→ compiler error
→ agent fixes
→ compile again

The feedback is deterministic, fast, and does not require another LLM judge.

Fast compilation also affects AI cost

An agent may iterate ten or twenty times.

Each cycle includes:

edit
→ validate
→ read result
→ reason again

Slower validation increases wall-clock time and can enlarge the model context and token bill.

Build performance therefore becomes part of agent-task economics.

A strong standard library reduces dependency hallucination

Coding agents often suggest third-party packages based on training-memory patterns.

Those dependencies may be stale, renamed, abandoned, or even nonexistent.

Go’s relatively comprehensive standard library lets many common server tasks avoid external packages entirely.

Fewer dependencies mean fewer opportunities for supply-chain mistakes.

Module checksums matter more with autonomous dependency changes

Go’s module checksum database and mirror provide integrity guarantees for downloaded dependencies.

An agent may modify go.mod, but the toolchain verifies expected module content.

This does not prevent choosing a malicious dependency intentionally, but it reduces risks from silent tampering and disappearing upstream artifacts.

`govulncheck` gives actionable vulnerability feedback

Many scanners report every CVE in the dependency graph.

govulncheck goes further by considering whether the program actually calls vulnerable symbols.

That produces a more useful agent loop:

vulnerability
→ reachable symbol
→ upgrade or replace dependency
→ test again

Lower-noise feedback is easier to automate safely.

Testing has a clear default path

Go ships with:

go test ./...

The agent does not need to reason first about which default testing framework the project should use.

A consistent toolchain reduces unnecessary agent decisions.

Native fuzzing is especially useful for AI-generated code

Models are good at producing plausible happy paths.

Boundary cases remain difficult.

Go’s native fuzz support allows an agent to write a parser, generate a fuzz target, run it, observe a panic, and fix the issue.

That is exactly the kind of automatic feedback loop agents need.

`gopls` and modernizers make refactoring more deterministic

Large language models are often asked to perform repository-wide refactors.

Instead of letting the model rewrite every occurrence manually, deterministic tools can perform mechanical work.

gopls provides references, renames, diagnostics, and code actions.

Modernized go fix capabilities can upgrade older patterns using deterministic transformations.

A safer pattern is:

agent chooses intent
→ deterministic tool performs mechanical change
→ agent reviews diff

“AI-friendly language” needs a new definition

Previously, AI friendliness might have meant short syntax and rapid prototyping.

A more useful definition now includes:

AI ergonomics
=
easy generation
+ easy verification
+ deterministic refactoring
+ controlled dependencies
+ consistent output

A language that is easy to generate but difficult to verify can become more expensive as agent output scales.

Does this make Go universally better than Python?

No.

Python remains exceptionally strong for:

data science;
machine learning;
research;
automation;
rapid prototyping.

Go’s advantages are strongest in long-lived production systems such as backend services, CLIs, cloud infrastructure, microservices, and agent tool servers.

The relevant variable is the software lifecycle, not a universal language ranking.

A concrete coding-agent verification loop

Instead of prompting an agent with “write high-quality code,” give it deterministic steps:

1. modify code
2. gofmt -w .
3. go test ./...
4. go vet ./...
5. govulncheck ./...
6. run targeted or fuzz tests on risky input paths
7. report unresolved issues and final diff

For dependency changes:

8. go mod tidy
9. explain why each new dependency is necessary

This is far more reliable than adding vague quality language to the system prompt.

The broader lesson is deterministic feedback

The same principle applies to other ecosystems.

Java can use compiler checks, static analysis, tests, and dependency scanning.

Rust can use rustfmt, clippy, cargo test, and cargo audit.

TypeScript can use tsc, linting, tests, and lockfile security checks.

The principle is universal:

do not ask an LLM to be the sole judge of the code it just generated.

Review capacity becomes the limiting resource

If generation becomes ten times faster while review becomes only twice as fast, repositories accumulate:

more PRs
→ reviewer fatigue
→ shallow approvals
→ more production defects

Toolchains that reduce review uncertainty therefore become increasingly valuable.

Go’s deliberate simplicity and uniformity can become an advantage precisely because AI output is abundant.

Conclusion

AI coding changes language economics.

The key question is shifting from:

How quickly can a developer write this?

to:

How quickly can the team prove that generated code is safe and correct?

Go offers many deterministic checkpoints:

gofmt
static types
fast compiler
go test
fuzzing
govulncheck
module checksums
gopls
go fix

Models will still hallucinate and make poor architectural decisions.

But when a toolchain exposes those mistakes earlier, more cheaply, and more mechanically, coding agents become easier to control.

The more interesting future question is not which language AI can generate most fluently.

It is which language makes AI mistakes easiest to detect.

For more AI coding, Go, coding-agent, and software-engineering analysis, visit Zyentor Picks: https://www.zyentorpicks.com/.

Originally published on Zyentor Picks.

In the AI Coding Era, Go’s Advantage Is Verifiability

cheng zhang — Mon, 24 Aug 2026 01:37:57 +0000

Article Summary

Programming-language debates historically focus on how pleasant or fast code is to write.

AI coding changes the economics.

The cost of producing code is falling rapidly.

The expensive part becomes proving that generated code is correct, secure, and maintainable.

Generation is no longer the bottleneck

A coding agent can quickly create:

APIs;
services;
tests;
deployment files;
documentation.

But production teams still need to answer:

Does it compile?
Are the APIs real?
Are dependencies maintained?
Are error paths correct?
Does concurrency behave safely?
Will the code remain maintainable?

The new bottleneck is increasingly:

Generation: fast
Verification: expensive

`gofmt` reduces review noise

Go intentionally removes formatting choice.

Human-written and AI-written code pass through the same formatter:

human code
AI code
→ gofmt
→ consistent output

This sounds minor until many agents contribute to one repository.

Consistency lets reviewers focus on behavior rather than stylistic drift.

The compiler is a cheap deterministic critic

LLMs frequently invent methods, properties, and interfaces that do not exist.

In Go, many of those errors immediately fail compilation.

That creates a useful loop:

generate
→ go build / go test
→ compiler error
→ agent fixes
→ compile again

The feedback is deterministic, fast, and does not require another LLM judge.

Fast compilation also affects AI cost

An agent may iterate ten or twenty times.

Each cycle includes:

edit
→ validate
→ read result
→ reason again

Slower validation increases wall-clock time and can enlarge the model context and token bill.

Build performance therefore becomes part of agent-task economics.

A strong standard library reduces dependency hallucination

Coding agents often suggest third-party packages based on training-memory patterns.

Those dependencies may be stale, renamed, abandoned, or even nonexistent.

Go’s relatively comprehensive standard library lets many common server tasks avoid external packages entirely.

Fewer dependencies mean fewer opportunities for supply-chain mistakes.

Module checksums matter more with autonomous dependency changes

Go’s module checksum database and mirror provide integrity guarantees for downloaded dependencies.

An agent may modify go.mod, but the toolchain verifies expected module content.

This does not prevent choosing a malicious dependency intentionally, but it reduces risks from silent tampering and disappearing upstream artifacts.

`govulncheck` gives actionable vulnerability feedback

Many scanners report every CVE in the dependency graph.

govulncheck goes further by considering whether the program actually calls vulnerable symbols.

That produces a more useful agent loop:

vulnerability
→ reachable symbol
→ upgrade or replace dependency
→ test again

Lower-noise feedback is easier to automate safely.

Testing has a clear default path

Go ships with:

go test ./...

The agent does not need to reason first about which default testing framework the project should use.

A consistent toolchain reduces unnecessary agent decisions.

Native fuzzing is especially useful for AI-generated code

Models are good at producing plausible happy paths.

Boundary cases remain difficult.

Go’s native fuzz support allows an agent to write a parser, generate a fuzz target, run it, observe a panic, and fix the issue.

That is exactly the kind of automatic feedback loop agents need.

`gopls` and modernizers make refactoring more deterministic

Large language models are often asked to perform repository-wide refactors.

Instead of letting the model rewrite every occurrence manually, deterministic tools can perform mechanical work.

gopls provides references, renames, diagnostics, and code actions.

Modernized go fix capabilities can upgrade older patterns using deterministic transformations.

A safer pattern is:

agent chooses intent
→ deterministic tool performs mechanical change
→ agent reviews diff

“AI-friendly language” needs a new definition

Previously, AI friendliness might have meant short syntax and rapid prototyping.

A more useful definition now includes:

AI ergonomics
=
easy generation
+ easy verification
+ deterministic refactoring
+ controlled dependencies
+ consistent output

A language that is easy to generate but difficult to verify can become more expensive as agent output scales.

Does this make Go universally better than Python?

No.

Python remains exceptionally strong for:

data science;
machine learning;
research;
automation;
rapid prototyping.

Go’s advantages are strongest in long-lived production systems such as backend services, CLIs, cloud infrastructure, microservices, and agent tool servers.

The relevant variable is the software lifecycle, not a universal language ranking.

A concrete coding-agent verification loop

Instead of prompting an agent with “write high-quality code,” give it deterministic steps:

1. modify code
2. gofmt -w .
3. go test ./...
4. go vet ./...
5. govulncheck ./...
6. run targeted or fuzz tests on risky input paths
7. report unresolved issues and final diff

For dependency changes:

8. go mod tidy
9. explain why each new dependency is necessary

This is far more reliable than adding vague quality language to the system prompt.

The broader lesson is deterministic feedback

The same principle applies to other ecosystems.

Java can use compiler checks, static analysis, tests, and dependency scanning.

Rust can use rustfmt, clippy, cargo test, and cargo audit.

TypeScript can use tsc, linting, tests, and lockfile security checks.

The principle is universal:

do not ask an LLM to be the sole judge of the code it just generated.

Review capacity becomes the limiting resource

If generation becomes ten times faster while review becomes only twice as fast, repositories accumulate:

more PRs
→ reviewer fatigue
→ shallow approvals
→ more production defects

Toolchains that reduce review uncertainty therefore become increasingly valuable.

Go’s deliberate simplicity and uniformity can become an advantage precisely because AI output is abundant.

Conclusion

AI coding changes language economics.

The key question is shifting from:

How quickly can a developer write this?

to:

How quickly can the team prove that generated code is safe and correct?

Go offers many deterministic checkpoints:

gofmt
static types
fast compiler
go test
fuzzing
govulncheck
module checksums
gopls
go fix

Models will still hallucinate and make poor architectural decisions.

But when a toolchain exposes those mistakes earlier, more cheaply, and more mechanically, coding agents become easier to control.

The more interesting future question is not which language AI can generate most fluently.

It is which language makes AI mistakes easiest to detect.

For more AI coding, Go, coding-agent, and software-engineering analysis, visit Zyentor Picks: https://www.zyentorpicks.com/.

Originally published on Zyentor Picks.

MCP 2026-07-28 Stateless Architecture: Scaling Agent Servers Without Sticky Sessions

cheng zhang — Fri, 21 Aug 2026 01:51:02 +0000

Article Summary

The 2026-07-28 Model Context Protocol release candidate introduces one of the biggest architectural changes since MCP launched: the transport core becomes stateless. Earlier HTTP MCP servers required an initialize handshake and an Mcp-Session-Id, forcing clients to remain attached to session state. At scale, that created sticky routing, Redis session stores, pod-failure problems, and poor serverless behavior. The new specification removes transport-level session management. Each request becomes self-describing and independent, allowing normal round-robin load balancing, transparent failover, serverless deployment, HTTP-header routing, cache controls, multi-round-trip interactions, and asynchronous tasks.

MCP originally fit an environment like:

one developer
→ one local MCP server

That was ideal for stdio and local integrations.

Enterprise deployment is different:

10,000 users
→ AI gateway
→ MCP cluster
→ many enterprise tools

At that scale, transport sessions become infrastructure.

Why the old session model was difficult

The older HTTP design used:

initialize
→ Mcp-Session-Id
→ subsequent requests reuse session

Imagine three Kubernetes pods:

Pod A
Pod B
Pod C

The first request reaches A.

The next round-robin request reaches C.

If session state lives only in A, C cannot continue.

The result is a session-not-found failure.

Traditional workarounds

Sticky routing

Pin a client to one pod.

This hurts load balancing and failover.

Shared Redis

Store session state outside the pods.

This adds network calls, latency, operational complexity, and another highly available dependency.

Session-aware gateway

Make the gateway understand protocol state.

That increases coupling and complexity.

What the 2026-07-28 specification changes

The new core removes:

the initialize / initialized handshake;
the logical Mcp-Session-Id.

Each request carries its own metadata.

HTTP headers include values such as:

MCP-Protocol-Version
Mcp-Method
Mcp-Name

The request body also includes client information and capabilities in _meta.

Any healthy server instance can process any request.

Architectural consequences

Standard round robin

Requests can go to any pod.

Transparent pod failure

A restarted container no longer destroys a transport session.

Serverless deployment

MCP servers can run naturally on serverless infrastructure and scale toward zero when idle.

No protocol-session Redis

Shared state may still be required for business tasks, but it is no longer required merely to maintain an MCP transport session.

Google notes that major production servers such as GitHub’s MCP server have already moved away from Redis session storage under this architecture.

HTTP headers become a governance layer

Promoting method and tool identity to HTTP headers allows gateways to perform:

routing;
rate limiting;
auditing;
policy;
metrics;

without parsing the JSON body.

For example:

Mcp-Name: delete_user
→ require approval

Mcp-Name: search
→ allow

The protocol also requires header and body values to match, reducing opportunities for policy bypass.

Caching becomes explicit

The new design introduces fields such as:

ttlMs
cacheScope

Clients can cache tool and resource lists for an explicit period rather than maintaining long-lived connections merely to detect changes.

This can reduce repeated requests significantly at enterprise scale.

What about user confirmation?

Stateless systems still need multi-step interactions.

The new Multi Round-Trip Requests pattern allows a server to return an InputRequiredResult plus serialized requestState.

The client:

asks the user;
collects the answer;
resends the request;
includes the original request state.

Any server instance can continue the workflow.

State moves into an explicit application-level object rather than transport affinity.

Long-running tasks

A database backup or refund may take many seconds.

The Tasks extension allows a tool call to return a taskId immediately and execute in the background.

The client can later use task primitives to retrieve status and final results.

The conversation does not need to hold one connection open.

Stateless transport does not mean stateless business logic

A long-running refund still needs:

taskId
status
result

stored somewhere.

The difference is that the datastore exists for business task state, not for protocol transport sessions.

That is a cleaner architecture boundary.

Security improvements

The specification also strengthens several security mechanisms.

Issuer verification

Clients validate authorization issuers.

Resource indicators

Tokens identify the intended MCP resource server.

JSON Schema 2020-12

Tool arguments can use richer schema composition and stricter validation.

These changes matter more as MCP moves from local development into remote enterprise infrastructure.

Formal deprecation

The new ecosystem also introduces a predictable lifecycle:

Active
→ Deprecated
→ Removed

Cloud observability increasingly moves toward OpenTelemetry rather than protocol-specific logging.

Again, MCP is beginning to look like cloud infrastructure rather than a local integration mechanism.

Recommended cloud architecture

Older deployment:

agent
→ MCP gateway
→ sticky load balancer
→ MCP pod
→ Redis session

New design:

agent
→ API gateway
→ round-robin load balancer
   ├── MCP pod
   ├── MCP pod
   └── MCP pod

Long-running business task state remains separate.

Migration approach

Do not migrate every server at once.

Start with:

inventory current protocol and session dependencies;
identify Redis and sticky-session assumptions;
upgrade one read-only server;
place it behind normal round-robin routing;
test pod restarts and autoscaling;
test user confirmation and long-running tasks;
migrate write operations only after idempotency and approval are verified.

Beta SDK support is already appearing across major languages, but production migration should still be staged.

What to measure

Track:

session-related errors;
pod-failure impact;
retry rate;
long-task completion;
infrastructure cost;
latency;
gateway policy behavior.

The strongest signal of success is that pod churn becomes invisible to clients.

Conclusion

The 2026-07-28 MCP architecture is important because MCP is moving from a session-oriented integration protocol toward stateless cloud infrastructure.

The practical outcomes include:

normal load balancing;
no sticky sessions;
easier serverless deployment;
transparent failover;
routable HTTP headers;
better caching;
asynchronous tasks;
multi-round-trip interactions;
clearer security boundaries.

If MCP remained optimized only for a developer laptop and local stdio server, it would stay a useful developer protocol.

A stateless, governable, horizontally scalable remote architecture gives it a realistic path toward becoming foundational infrastructure for enterprise agents.

For more practical MCP, agent architecture, AI gateway, and production engineering guidance, visit Zyentor Picks: https://www.zyentorpicks.com/.

Originally published on Zyentor Picks.

Google Cloud API Gateway Model Routing: One OpenAI-Compatible Endpoint for Multiple LLMs

cheng zhang — Wed, 19 Aug 2026 01:58:11 +0000

Article Summary

Enterprise AI applications increasingly use more than one model. Low-cost models may handle routine support, stronger models may handle reasoning or coding, and open models may serve batch workloads. If every application integrates directly with every provider, credentials, retries, rate limits, model IDs, and cost tracking spread across the codebase. Google Cloud API Gateway now offers Model Routing in Public Preview. It can expose an OpenAI-compatible endpoint and route requests, through OpenAPI 3.x configuration, to Google-hosted Gemini, Claude, or OpenAI OSS-GPT backends. The real value is not calling three models through one API. It is decoupling business code from model vendors.

A prototype often contains provider-specific branches:

if model == "gemini":
    call_gemini()
elif model == "claude":
    call_claude()

At scale, every service ends up with its own SDKs, provider keys, retry policies, token tracking, and migration logic.

An LLM gateway centralizes those concerns.

Target architecture

application
→ LLM gateway
   ├── authentication
   ├── routing
   ├── rate limits
   ├── token tracking
   ├── policy
   └── observability
→ model backends

The application talks to a stable internal endpoint while model choices remain behind the gateway.

What Google Cloud added

Google Cloud API Gateway Model Routing is currently in Public Preview. It can accept OpenAI-compatible requests and route them to Google Cloud-hosted model backends including Gemini, Claude, and OpenAI OSS-GPT examples.

The gateway can transcode requests, attach the required backend authentication, and invoke the selected model.

Decouple applications from provider endpoints

Applications can always call something like:

https://ai.company.com/v1/chat

The backend can change from Gemini to Claude or another hosted model without requiring business-code changes.

Separate client and backend authentication

Without a gateway:

application
→ provider credential
→ model

With a gateway:

application
→ company credential
→ gateway
→ backend credential
→ model

Applications authenticate to the enterprise endpoint while backend credentials are rotated centrally.

Conceptual routing configuration

A simplified structure looks like:

openapi: 3.0.4

x-google-api-management:
  backends:
    gemini-fast:
      address: https://aiplatform.googleapis.com/...

    claude-strong:
      address: https://aiplatform.googleapis.com/...

  ai:
    models:
      routing:
        routers:
          default-router:
            defaultModel:
              backend: gemini-fast
            rules:
              - model: claude-strong
                backend: claude-strong

The application can then send a familiar OpenAI-style request with a model value.

Prefer virtual models

Do not expose concrete provider model IDs throughout business code.

Use names such as:

chat-fast
chat-balanced
reasoning-high
coding-high
batch-cheap

The platform maps those names to actual backends.

This makes upgrades, fallbacks, and vendor changes much easier.

Why virtual models matter

When a model version changes, applications do not need to change. When one backend is degraded, the mapping can be updated centrally.

Applications should choose a capability tier. The platform should choose the actual model.

Routing strategies

A gateway can route by explicit application choice, plan level, task type, or availability policy. More complex dynamic routing based on cost and live quality may require an additional AI control service outside the basic API Gateway feature set.

Current host limitation

Google documents an important constraint: backends referenced by one router must share the same host, such as aiplatform.googleapis.com.

This is not an arbitrary cross-internet reverse proxy. It is best understood as routing across Google Cloud-hosted model backends.

Claude can still participate because it can be accessed through a Google Cloud-hosted model endpoint.

OpenAI-compatible clients reduce migration cost

Many frameworks already support configurable base_url and api_key values. A compatible gateway lets applications migrate without rewriting complete provider integrations.

But schema compatibility does not imply behavioral equivalence.

Models still differ in tool calling, structured output, reasoning, vision, streaming, context, safety, and token accounting.

Every substitution still needs evaluations.

Maintain a capability registry

For example:

chat-fast:
  supports:
    - text
    - streaming
    - tools

reasoning-high:
  supports:
    - text
    - tools
    - structured_output

Applications should request required capabilities rather than assuming all models behave the same way.

Rate limiting should be multi-dimensional

A useful policy hierarchy is:

user
tenant
application
virtual model

This prevents one agent or department from consuming all high-end capacity.

Cost attribution belongs at the gateway

Record:

tenant
user
application
virtual model
actual model
input tokens
output tokens
latency
cost
status

Otherwise centralized routing merely produces a centralized but opaque cloud bill.

Budget policies

Budgets can be defined by department or product. At 80% utilization, warn. At 100%, downgrade or require approval.

A mature model router optimizes business value, SLA, and cost rather than simply picking the strongest model.

Roll out model changes gradually

When changing the backend behind a virtual model, use staged rollout:

1%
→ 10%
→ 25%
→ 50%
→ 100%

Compare task success, latency, cost, safety, and user feedback.

Run evaluations before routing changes

Evaluation should cover the real workload: FAQ, code, RAG, structured JSON, long context, tool calling, Chinese, edge cases, and safety.

Do not rely only on vendor benchmark scores.

Fallback needs error classification

Rate limits, timeouts, and some transient server errors may justify fallback.

Invalid requests, schema failures, and permission errors usually do not.

Blind fallback can multiply costs without solving the issue.

Business side effects still need idempotency

A model gateway does not solve transactional semantics. If an agent creates an order, experiences a network timeout, and retries through another model, the business tool can execute twice unless it supports an idempotency key.

The gateway manages model routing. The business layer manages business correctness.

Recommended layered architecture

client
→ API Gateway
   ├── auth
   ├── rate limit
   ├── virtual routing
   └── token tracking
→ optional AI gateway service
   ├── capability checks
   ├── policy
   ├── budgets
   ├── eval flags
   ├── retries
   └── traces
→ Google-hosted models

Smaller teams can begin directly with API Gateway. Larger organizations may add a dedicated AI control layer.

When this approach fits

It is attractive when the organization already runs heavily on Google Cloud, prefers serverless infrastructure, does not want to operate an open-source gateway cluster, and can access required models through Google Cloud-hosted endpoints.

When another gateway may fit better

A self-managed gateway may be a better fit for direct multi-provider internet routing, highly customized caching and billing, non-Google-hosted models, or complete self-hosting.

Conclusion

Google Cloud API Gateway Model Routing is not mainly about calling Gemini, Claude, and OSS-GPT from one API. It is about keeping business code independent from specific model vendors.

The architecture shifts from:

app → provider API

to:

app → enterprise AI endpoint → routing → model

That creates a central place for authentication, model selection, rate limits, cost, fallback, and auditing.

For enterprises already operating on Google Cloud, the Public Preview is a practical LLM gateway option worth evaluating.

For more model-routing, LLM gateway, agent infrastructure, and production AI engineering guides, visit Zyentor Picks: https://www.zyentorpicks.com/.

Originally published on Zyentor Picks.

GitHub Copilot Agent Plugins 1.0: Build Once, Run Across Agent Clients

cheng zhang — Tue, 18 Aug 2026 03:42:01 +0000

Article Summary

On August 12, 2026, GitHub announced general support for Agent Plugins 1.0 in VS Code, Copilot CLI, the GitHub Copilot SDK, and the Copilot app. The more important point is that Agent Plugins 1.0 is not merely a GitHub-specific extension format. It is an open standard developed with participation from AWS, Anysphere, Microsoft, OpenAI, and Vercel, with Google also joining as a core maintainer. Its core idea is to package skills and MCP server configuration into one installable unit that compatible agent clients can inspect and consume. This article explains the relationship between skills, MCP, plugins, portability, governance, and why agent capability packaging may become an important layer of the AI developer stack.

1. Why do agents need another plugin standard?

Developers increasingly face a packaging problem.

Imagine a deployment capability. It contains a skill describing release procedures, rollback rules, production checks, and incident policies. It also contains an MCP server exposing tools such as deployment status, deploy, rollback, and log query.

The underlying capability exists once, but supporting several agent clients has historically required different manifests, directory layouts, installation steps, and vendor-specific wrappers.

Agent Plugins 1.0 attempts to reduce that duplication.

2. The core idea

The principle is:

build one package that compatible agent clients can understand.

A package may contain:

plugin.json
skills/
mcp.json
vendor-specific/

The standard portions can be shared, while vendor-specific features remain namespaced.

For Copilot, a package can contain:

com.github.copilot/

for features that other clients can ignore.

This creates a practical compromise: standardize portable capabilities without eliminating vendor innovation.

3. Skill, MCP, and Plugin are different layers

Skill

A skill tells the agent how a task should be performed. It can encode workflow, domain rules, runbooks, templates, examples, and best practices.

MCP server

MCP gives the agent controlled access to tools and external systems such as GitHub, Jira, databases, Kubernetes, CRM, and browsers.

Plugin

The plugin is the packaging and distribution unit.

A useful mental model is:

Skill = procedure and knowledge
MCP = tool access
Plugin = packaging and distribution

They complement rather than replace each other.

4. Why combining skills and tools matters

A deployment MCP server may expose:

deploy()
rollback()
get_status()

But tools alone do not teach the agent when deployment is allowed, which checks are mandatory, which environments require approval, or when rollback is required.

A skill can encode those rules.

Together:

skill → defines the safe procedure
MCP → provides controlled actions
plugin → distributes them together

That resembles a complete agent capability.

5. Current GitHub Copilot support

GitHub says Agent Plugins 1.0 is generally available in:

VS Code;
Copilot CLI;
GitHub Copilot SDK;
GitHub Copilot app.

One compatible package can therefore be reused across several Copilot surfaces.

6. Why enterprises benefit more than individuals

An enterprise may have hundreds of developers, dozens of internal agents, many tools, multiple IDEs, several CLIs, and more than one agent platform.

If every integration requires separate installation, authorization, upgrades, and auditing, governance becomes expensive.

A standardized plugin can support:

enterprise marketplace
→ approved plugin
→ developer installation
→ skill + MCP
→ controlled version
→ controlled permissions
→ controlled update

That is a major operational benefit.

7. Agent Plugin can become a software distribution unit

Traditional enterprise software is distributed as applications, packages, containers, and extensions.

The agent era may add another unit: the agent plugin.

An organization could maintain:

company-deploy
company-database
company-security
company-support
company-finance

Each package can contain procedures, tools, rules, agents, commands, and hooks.

Installing it gives a compatible agent a governed enterprise capability.

8. Why portability matters

The standard is intended to be independent of a single vendor.

GitHub says the specification was published with participation from AWS, Anysphere, Microsoft, OpenAI, and Vercel, while Google joined as a core maintainer.

This signals a broader direction: agent extension ecosystems are moving from isolated formats toward shared packaging.

9. Portable does not mean identical everywhere

A plugin can contain standard parts:

skills/
mcp.json

and vendor-specific parts such as:

com.github.copilot/

Other clients do not automatically understand Copilot-only agents, commands, rules, hooks, or canvases.

The common core can be portable while vendor-specific capabilities remain different.

10. A minimal conceptual package

A simple package might look like:

company-deploy/
├── plugin.json
├── skills/
│   └── deployment/
│       └── SKILL.md
└── mcp.json

A deployment skill could define pre-deployment checks, approval, database migration review, health verification, error-rate review, and rollback conditions.

The MCP server provides the actual deployment operations.

The result is a reusable enterprise capability.

11. Enterprise marketplaces need governance

A useful internal marketplace should include a registry, security scanning, risk-based approval, and controlled distribution.

Track plugin version, publisher, owner, source, license, risk, MCP servers, and permissions.

Allow installation only from approved marketplaces rather than arbitrary repositories.

12. GitHub managed settings

GitHub exposes managed settings such as:

enabledPlugins
extraKnownMarketplaces
strictKnownMarketplaces

These settings allow organizations to install or block plugins, add approved marketplaces, and restrict installation to managed marketplaces.

This matters because plugins can include MCP server configuration and therefore carry real system access.

13. MCP allowlists still matter

Approving a plugin should not automatically approve every MCP server or every runtime call.

A safer policy is:

plugin approved
AND
MCP server approved
AND
user has required role
AND
runtime tool policy passes

Only then does execution proceed.

14. Plugin supply-chain risk

A plugin can combine instructions, tools, hooks, and commands.

A malicious or compromised package could enable data exfiltration, malicious shell execution, credential theft, MCP redirection, prompt injection, persistence through hooks, or supply-chain compromise.

Agent plugins should therefore be governed like software dependencies, not decorative editor themes.

15. Recommended security pipeline

develop plugin
→ code review
→ static scan
→ skill-content scan
→ MCP allowlist check
→ permission review
→ signing
→ internal marketplace
→ staged installation
→ runtime auditing

Track plugin ID, version, publisher, hash, MCP servers, skills, permissions, installed users, and last update.

16. Good candidates for internal plugins

Plugins are especially useful for frequent, repeatable, tool-enabled work.

Examples include DevOps deployment and rollback, data queries and reports, security investigation, issue analysis, support order lookup, and ticket creation.

A useful rule is:

frequent workflow + stable procedure + tools + multiple agent clients.

17. Poor candidates

Avoid packaging every process.

Weak candidates include rare workflows, tasks requiring substantial human judgment, unstable systems without reliable APIs, and environments with immature authorization.

Govern the underlying system before exposing it to agents.

18. Likely future layering

A common stack may emerge:

Agent Client
↓
Plugin
├── Skills
├── MCP Servers
├── Rules
├── Commands
├── Hooks
└── Vendor Extensions

In that stack, MCP is the tool-connectivity layer, Skills are the procedural layer, Plugins are the packaging layer, and Agent Clients are the runtime layer.

This begins to resemble browsers and extensions or IDEs and plugin marketplaces.

19. Why standards may matter more than another model benchmark

Models change rapidly.

Enterprise assets are more durable: tools, workflows, policies, permissions, internal systems, and domain knowledge.

If those assets can move across compatible agent clients, organizations reduce vendor lock-in and avoid rebuilding every integration when the user-facing agent changes.

20. Do existing Copilot plugins need immediate migration?

No.

GitHub says existing plugins that do not target Agent Plugins 1.0 remain supported.

A sensible policy is:

keep stable production plugins unchanged initially;
use Agent Plugins 1.0 for new portable packages;
prioritize migration where multiple clients need the same capability;
migrate Copilot-only packages when the operational benefit is clear.

21. Suggested enterprise repository

agent-plugins/
├── deployment/
│   ├── plugin.json
│   ├── skills/
│   ├── mcp.json
│   └── com.github.copilot/
├── database/
├── security/
└── support/

Add CODEOWNERS, CI, security scanning, version tags, and release notes.

Do not let agent plugins become unmanaged configuration on individual laptops.

22. Launch checklist

Functionality

Does the target client load the plugin?
Are skills discovered correctly?
Does MCP connect?
Are vendor-specific extensions isolated?

Security

Are MCP servers allowlisted?
Does the package execute shell commands?
Can it access secrets?
Can it reach production?
Do high-risk tools require approval?

Governance

Who owns it?
Which version is installed?
Who may install it?
How is it revoked?
How is it updated?

Operations

Are calls traceable?
Are tool errors observable?
What happens when MCP is unavailable?
How are new versions rolled out gradually?

Conclusion

GitHub’s support for Agent Plugins 1.0 looks like an extension-system update, but it points to a larger shift:

agent capabilities are becoming portable software packages.

The stack is becoming clearer:

Skill → how to perform the task
MCP → which tools can be called
Plugin → how capability is packaged and distributed
Agent Client → where it runs

For individuals, this reduces configuration duplication. For enterprises, the bigger value is internal marketplaces, consistent versions, centralized permissions, security scanning, reuse across agent clients, and reduced vendor lock-in.

If the standard continues to gain cross-vendor support, organizations may package enterprise capabilities once and make them safely available to multiple compatible agents.

For more analysis of Agent Plugins, MCP, Skills, and enterprise AI governance, visit Zyentor Picks: https://www.zyentorpicks.com/.

Originally published on Zyentor Picks.

Gemini Live API Async Function Calling for Real-Time Voice Agents

cheng zhang — Tue, 18 Aug 2026 03:40:53 +0000

Article Summary

One of the biggest usability problems in real-time voice agents is tool latency. If the model needs to query CRM, orders, databases, search, or ticketing systems and every function call blocks the conversation, the user experiences several seconds of silence. Gemini Live API supports asynchronous function calling in its cascaded architecture. A function can be marked NON_BLOCKING, allowing the live conversation to continue while the tool executes in the background. This does not mean the model can invent tool-dependent facts. It means developers need explicit task state, timeouts, cancellation, concurrency controls, result correlation, permissions, and idempotency.

1. Why tool latency hurts voice agents

Text users may tolerate a few seconds. Voice users notice silence immediately.

A normal live interaction can include speech understanding, CRM, databases, search, model generation, and audio output. External systems have unpredictable latency.

If every tool call blocks the session, natural conversation disappears.

2. Blocking function calls

A traditional flow is:

user asks
→ model calls get_refund_status
→ conversation pauses
→ backend waits four seconds
→ result returns
→ model resumes

The user hears a long gap.

That may be acceptable for some transactional workflows, but it is poor conversational design.

3. What NON_BLOCKING changes

With asynchronous function calling:

user request
→ model starts tool call
├── tool executes in background
└── conversation continues
→ result returns
→ model incorporates result

The agent can ask a clarification or explain the process while waiting.

4. Asynchronous does not mean speculative

If the refund status has not returned, the agent must not claim that the refund is complete.

Separate state into:

known
pending
unknown

Only confirmed data should be presented as fact.

5. Good non-blocking tools

Useful candidates include customer lookup, order status, tickets, inventory, shipping, knowledge retrieval, web search, recommendations, and other independent read operations.

Several independent read calls can also execute in parallel.

6. Poor non-blocking candidates

Be cautious with payments, deletes, production changes, irreversible operations, and any eligibility decision that is required before downstream logic can continue.

These often require blocking behavior and explicit approval.

7. Recommended architecture

microphone
→ Gemini Live session
→ tool router
   ├── blocking
   ├── non-blocking
   └── approval-required
→ async task manager
   ├── timeout
   ├── retry
   ├── cancellation
   └── idempotency
→ enterprise APIs
→ tool result
→ live session

Add authorization, tracing, cost controls, and audit logs.

8. Tool policy belongs in the application

Maintain metadata such as:

name: get_refund_status
mode: non_blocking
timeout_ms: 5000
retry: 1
idempotent: true
risk: read
requires_confirmation: false

For a financial write:

name: create_refund
mode: blocking
timeout_ms: 10000
retry: 0
idempotent: true
risk: financial
requires_confirmation: true

The model should not classify business risk by itself.

9. Conceptual Gemini declaration

Exact syntax should follow the active SDK version, but conceptually:

tool = {
    "function_declarations": [
        {
            "name": "get_order_status",
            "description": "Get current order status",
            "parameters": {
                "type": "object",
                "properties": {
                    "order_id": {"type": "string"}
                },
                "required": ["order_id"]
            },
            "behavior": "NON_BLOCKING"
        }
    ]
}

The important behavior is that the live session does not have to stop while the function runs.

10. Use an async task manager

Do not block a WebSocket callback with synchronous HTTP work.

Instead:

async def execute_tool(call):
    task_id = create_task_id(call)
    task = asyncio.create_task(run_with_timeout(call))
    registry[task_id] = task

When it completes:

async def on_tool_done(task_id):
    result = await registry[task_id]
    await send_tool_result(result)

11. Every tool needs a timeout

External services may respond in 300 milliseconds, two seconds, eight seconds, or never.

A timed-out task should return structured state instead of remaining pending forever.

12. Continue the conversation during delays

Instead of dead air, the agent can say:

“The customer system is taking a little longer. I can confirm a few details while it finishes.”

This is one of the main user-experience benefits of asynchronous tools.

13. Limit parallelism

A model may want to call CRM, orders, tickets, payments, email, and analytics at once.

Set a practical limit such as:

max_parallel_tools = 3

Queue the rest.

This protects downstream systems and keeps state manageable.

14. Correlate results correctly

Store:

session_id
call_id
tool
arguments
start time
conversation turn

When a result returns, match the call ID, confirm that the session is still valid, decide whether the result remains relevant, and only then deliver it back to the live model.

15. Handle topic changes

The user may request order A and then immediately correct the request to order B.

If possible, cancel A. If cancellation is impossible, mark its result stale and do not inject it into the active conversation.

Useful states include:

PENDING
RUNNING
COMPLETED
TIMEOUT
FAILED
CANCELLED
STALE

16. Use idempotency for writes

Async systems create retry ambiguity. A remote write may succeed while the client loses the response.

Use an idempotency key for side-effecting operations such as ticket creation or order changes. The target service should guarantee one execution per key.

17. User interruptions must stop current speech

A real voice agent must support barge-in.

When a user interrupts, stop current audio, process the new input, decide whether existing tool tasks remain relevant, cancel or mark them stale, and continue with the updated goal.

18. What the agent can do while waiting

Useful actions include asking clarifying questions, collecting identity details, explaining process, and running independent read tools.

The agent should never invent pending results, promise unconfirmed outcomes, or claim that an operation completed before confirmation.

A useful instruction is:

Never state that a tool-dependent fact is confirmed
until the corresponding tool result is received.

19. Authorization remains mandatory

A model-generated tool call is not authorization.

The application still needs:

user identity
→ role
→ resource permission
→ tool permission
→ argument validation
→ execution

Otherwise the voice agent becomes an authorization bypass.

20. Minimize sensitive tool output

If a CRM result contains identity documents, bank information, contact details, and order state while the user asked only for order status, return only the necessary field to the model.

Minimizing model exposure is a basic production principle.

21. Observability

Track:

session_id
call_id
tool
mode
start
end
latency
status
retry count
cancellation
token use
business result

Important metrics include tool P50/P95 latency, timeout rate, cancellation rate, parallel call count, task success, first-audio latency, and conversation silence time.

22. Conversation silence time is a key metric

Backend tool latency alone does not describe user experience.

A tool may take four seconds, but if the agent keeps the conversation useful, the user may experience less than a second of dead air.

Measure silence, not only backend latency.

23. Customer-support example

A customer asks why a package has not arrived.

The agent starts shipping lookup while asking whether this is the same order discussed yesterday.

The user responds while the tools execute.

When the result returns, the agent provides the confirmed shipping status.

The backend was slow, but the conversation never fully stopped.

24. Blocking versus non-blocking policy

A practical classification is:

READ_FAST
READ_SLOW
WRITE_LOW_RISK
WRITE_HIGH_RISK
FINANCIAL
PRODUCTION

Possible defaults:

READ_FAST → NON_BLOCKING
READ_SLOW → NON_BLOCKING + status
WRITE_LOW_RISK → confirmation
WRITE_HIGH_RISK → BLOCKING + confirmation
FINANCIAL → BLOCKING + strong approval
PRODUCTION → BLOCKING + multi-party approval

25. Launch testing

Test slow responses, timeouts, disconnects, retries, out-of-order results, user interruptions, requirement changes, cancellations, authorization failures, prompt injection, sensitive data, and duplicate writes.

The final business state must remain correct even when the conversation is interrupted.

Conclusion

Gemini Live API asynchronous function calling addresses a fundamental real-time-agent problem:

slow tools should not automatically make the conversation slow.

With non-blocking calls, an agent can continue clarification, explanation, and independent work while external systems execute.

But the architecture needs task state, timeouts, cancellation, call IDs, concurrency controls, idempotency, authorization, sensitive-data filtering, and observability.

The central rule is simple: the agent may continue speaking while a tool runs, but it must never pretend to know a tool-dependent result before that result arrives.

For more practical Gemini API, voice-agent, function-calling, and production AI engineering guides, visit Zyentor Picks: https://www.zyentorpicks.com/.

Originally published on Zyentor Picks.

How to Generate E-commerce Product Pages in Bulk with AI

cheng zhang — Fri, 31 Jul 2026 06:35:59 +0000

Article Summary

Bulk-generating product pages with AI looks simple: send product attributes to a model and ask it to write persuasive copy. In practice, this approach often creates invented claims, mismatched specifications, repetitive content, prohibited wording, and formats that cannot be published across different sales channels. A production-ready system is not a loop that repeats one prompt. It is a content pipeline that combines product-data cleaning, factual constraints, structured generation, rule-based validation, human review, and multi-channel publishing. This guide provides a practical data model, prompt template, JSON output schema, Python batch-processing example, and quality-control checklist.

Why Direct AI Product-Copy Generation Often Fails

A common workflow is to copy a product name and a few attributes from a spreadsheet, then ask:

Write an attractive product detail page.

The model may produce fluent text, but fluent text is not necessarily accurate product content. Five problems appear repeatedly.

The source data is incomplete

Many product spreadsheets contain only:

SKU;
product name;
price;
one or two specifications.

A useful product page may also require target users, use cases, materials, dimensions, packaging, warnings, warranty terms, and verified benefits. When these facts are absent, a language model may fill the gaps with plausible but unsupported details.

Facts and marketing claims are mixed together

“Made with 304 stainless steel” is a factual attribute. “Designed for everyday durability” is a restrained interpretation. “The safest and most durable cup on the market” is an unverified claim.

If the system does not distinguish facts from acceptable marketing language, the model may present assumptions as product truth.

Every channel has different requirements

The same product may need:

an SEO title and meta description for a direct-to-consumer website;
marketplace-style feature sections;
Amazon bullet points;
a short video script;
social-media captions;
ad copy with strict character limits.

One generic paragraph creates more editing work downstream.

Batch processing amplifies errors and repetition

A single generated page may look good. A batch of 5,000 SKUs can reveal systemic failures:

repetitive title patterns;
clichés such as “crafted with care” on every page;
altered units and measurements;
mismatched model numbers and colors;
dirty data contaminating many outputs;
missing records after API timeouts;
duplicates created by retries.

There is no publishing gate

If model output is published immediately, the company is delegating content risk to the model. The model should generate a candidate. Rules, code, and human reviewers should decide whether that candidate is publishable.

The Right Architecture: A Six-Layer Content Pipeline

A production workflow can be divided into six layers:

Product master data
↓
Cleaning and normalization
↓
Fact locking and content policy
↓
Structured AI generation
↓
Validation and risk scoring
↓
Human review / automated publishing

Each layer has a different responsibility.

Layer 1: Product master data

Product facts should come from a PIM, ERP, product database, or approved spreadsheet—not from model inference.

Layer 2: Data cleaning

Normalize units, null values, enumerations, color names, dimensions, and brand terminology.

Layer 3: Fact locking

Define which fields may be copied, which may be paraphrased, and which claims are prohibited.

Layer 4: Structured generation

Require JSON rather than an uncontrolled block of prose.

Layer 5: Validation

Use code to inspect length, prohibited terms, specification consistency, duplication, and completeness.

Layer 6: Review and publishing

Allow low-risk content to pass automatically. Route high-risk categories and low-confidence outputs to a review queue.

Design the Product Data Table First

A useful source table should include at least the following fields:

FieldPurposeExamplesku_idUnique product identifierCUP-500-BLKproduct_nameCanonical product name500 ml vacuum insulated bottlebrandBrandExamplecategoryProduct categoryDrinkware / insulated bottlematerialMaterials304 stainless steel, food-contact PPsizeDimensions7.2 × 22.5 cmcapacityCapacity500 mlcolorColorObsidian blackpackage_contentsPackage contentsBottle ×1, manual ×1verified_featuresApproved featuresDouble-wall vacuum design, non-slip basetarget_usersIntended usersCommuters, studentsusage_scenariosUse casesOffice, commuting, short tripsrestrictionsProhibited claimsNo medical claims; no “permanent insulation”warrantyWarranty informationSeven-day returns; one-year quality warrantykeywordsSEO keywordsinsulated bottle, 500 ml bottle, commuter bottlechannelPublishing channelDTC site, Amazon, marketplace

Add three operational fields as well.

source_of_truth

Record where each fact came from: ERP, test report, supplier confirmation, or manual entry. This supports later audits and updates.

missing_fields

Identify missing information before generation. A SKU with critical missing fields should not enter an auto-publishing flow.

content_version

Increment the version whenever content is regenerated so that new and old copy cannot be confused.

Define Three Classes: Allowed, Inferable, and Prohibited

Before calling a model, classify information into three categories.

Allowed facts

These fields have a verified source:

material;
specifications;
dimensions;
capacity;
package contents;
model number;
color;
validated functions;
warranty terms.

Fact-based expressions

These can be derived from verified facts without exaggeration:

“Double-wall vacuum design” may become “helps reduce heat transfer.”
“Non-slip base” may become “designed to sit more securely.”
“500 ml capacity” may become “sized for everyday commuting.”

The model is improving language, not inventing product capabilities.

Prohibited content

This includes:

certifications that were not provided;
medical or therapeutic benefits;
“best,” “number one,” “absolutely,” or “100% guaranteed” without evidence;
unverified ingredients or materials;
nonexistent gifts;
fabricated sales figures or reviews;
invented warranty terms;
claims that violate platform or regulatory policies.

These rules should be fixed in the system prompt.

Require Structured JSON Output

Do not ask for “a product page.” Define an output contract.

{
"sku_id": "CUP-500-BLK",
"seo_title": "500 ml Vacuum Insulated Bottle | Obsidian Black",
"short_title": "500 ml Black Insulated Bottle",
"summary": "A lightweight insulated bottle designed for office and commuting use.",
"selling_points": [
{
"title": "Double-wall vacuum design",
"description": "Helps reduce heat transfer for everyday drinking needs."
}
],
"specifications": [
{"name": "Capacity", "value": "500 ml"}
],
"usage_scenarios": ["Office", "Commuting", "Short trips"],
"care_instructions": ["Wash before first use"],
"meta_description": "A 500 ml obsidian-black vacuum bottle with a 304 stainless-steel interior for office, commuting, and short trips.",
"risk_flags": [],
"missing_information": []
}

Structured output offers four major advantages:

It can be written directly to a CMS or product database.
Every field can be validated independently.
Channel-specific templates can reuse the same source.
Failures are easier to diagnose.

A Reusable Prompt Template

The following structure is suitable for batch generation.

You are an e-commerce product content editor. Generate content only from
the supplied product facts. Do not add materials, functions,
certifications, gifts, sales figures, reviews, warranty promises, or
health claims that are absent from the input.

Objectives:
1. Produce clear, accurate, and restrained product content.
2. Preserve the exact model number, dimensions, capacity, material,
quantity, and unit precision.
3. Do not use unverifiable claims such as "best," "number one,"
"absolute," "permanent," or "100%."
4. If critical information is missing, add it to missing_information.
Do not guess.
5. Return valid JSON only. Do not wrap it in a Markdown code block.

Content requirements:
- seo_title: no more than 60 characters;
- short_title: no more than 30 characters;
- summary: 80 to 160 characters;
- selling_points: 3 to 5 items, each traceable to verified_features;
- specifications: preserve all input values;
- meta_description: no more than 160 characters;
- risk_flags: list possible data or compliance risks.

Product input:
{{PRODUCT_JSON}}

Output JSON Schema:
{{OUTPUT_SCHEMA}}

For consistent batch jobs, keep the system prompt stable and place the product record in the user message. This improves cache efficiency, version control, and consistency.

Python Batch-Processing Example

The following example uses an OpenAI-compatible API format. Environment variables can be changed to point to different providers.

import csv
import json
import os
import time
from pathlib import Path

import requests

API_BASE = os.environ["LLM_API_BASE"].rstrip("/")
API_KEY = os.environ["LLM_API_KEY"]
MODEL = os.environ.get("LLM_MODEL", "your-model")
INPUT_FILE = Path("products.csv")
OUTPUT_FILE = Path("generated-products.jsonl")
ERROR_FILE = Path("generation-errors.jsonl")

SYSTEM_PROMPT = """
You are an e-commerce product content editor.
Use only supplied facts. Do not invent materials, functions,
certifications, gifts, sales figures, reviews, warranty promises,
or health claims. Return valid JSON only.
"""

def call_model(product: dict, retries: int = 3) -> dict:
payload = {
"model": MODEL,
"messages": [
{"role": "system", "content": SYSTEM_PROMPT},
{
"role": "user",
"content": json.dumps(product, ensure_ascii=False)
}
],
"response_format": {"type": "json_object"},
"temperature": 0.2
}

for attempt in range(retries):
try:
response = requests.post(
f"{API_BASE}/chat/completions",
headers={
"Authorization": f"Bearer {API_KEY}",
"Content-Type": "application/json"
},
json=payload,
timeout=90
)
response.raise_for_status()
content = response.json()["choices"][0]["message"]["content"]
result = json.loads(content)
result["sku_id"] = product["sku_id"]
return result
except (requests.RequestException, KeyError, json.JSONDecodeError) as exc:
if attempt == retries - 1:
raise RuntimeError(f"Generation failed: {exc}") from exc
time.sleep(2 ** attempt)

raise RuntimeError("Unexpected retry state")

def append_jsonl(path: Path, data: dict) -> None:
with path.open("a", encoding="utf-8") as file:
file.write(json.dumps(data, ensure_ascii=False) + "\n")

def main() -> None:
processed = set()
if OUTPUT_FILE.exists():
with OUTPUT_FILE.open("r", encoding="utf-8") as file:
for line in file:
if line.strip():
processed.add(json.loads(line)["sku_id"])

with INPUT_FILE.open("r", encoding="utf-8-sig", newline="") as file:
reader = csv.DictReader(file)
for product in reader:
sku_id = product.get("sku_id", "").strip()
if not sku_id or sku_id in processed:
continue

try:
result = call_model(product)
append_jsonl(OUTPUT_FILE, result)
print(f"OK: {sku_id}")
except Exception as exc:
append_jsonl(
ERROR_FILE,
{"sku_id": sku_id, "error": str(exc)}
)
print(f"FAILED: {sku_id}")

if __name__ == "__main__":
main()

This example includes three important production patterns:

Resumability: completed SKUs are not generated again.
Exponential backoff: temporary network failures do not immediately terminate the job.
Separate error logging: failed records can be reprocessed independently.

A production system should also add rate limiting, concurrency control, token logging, cost tracking, and idempotency keys.

Eight Validation Layers After Generation

JSON validity

Confirm that required fields exist, data types are correct, and arrays were not returned as strings.

Specification consistency

Dimensions, capacity, materials, model numbers, and quantities must match the product master data exactly. These values should be checked by code rather than by human reading alone.

Prohibited-term detection

Maintain separate dictionaries by country, industry, and marketplace. Food, supplements, cosmetics, medical devices, and children's products require different policies.

Hallucination detection

Extract materials, certifications, functions, and warranty statements from the output and compare them with the source record. Any fact that cannot be traced should enter review.

Similarity detection

Measure similarity across titles and selling points. If hundreds of pages differ only by color or model number, search engines and customers may see them as low-quality content.

Channel formatting

Set separate limits for titles, descriptions, bullet points, and keywords by channel. Do not apply one template to every marketplace.

Brand voice

Define prohibited expressions, preferred terminology, tone strength, and punctuation rules. The model should vary content within a brand system rather than improvise freely.

Human sampling

Use risk-based review:

ordinary low-risk categories: sample 5% to 10%;
high-value products: increase review coverage;
food, health, beauty, and children's products: require human approval;
new models, prompts, or data sources: review at least one full batch.

How to Reduce the Cost of 1,000 SKUs

The best optimization is not always switching to the cheapest model. It is reducing unnecessary tokens and rework.

Keep the system prompt stable

Use one shared policy for all SKUs. Providers that support prompt caching can charge much less for repeated input.

Use a model cascade

A practical three-stage flow is:

Low-cost model: cleaning, classification, missing-field detection
↓
Primary model: titles, benefits, and descriptions
↓
Low-cost model or code: compliance and risk checking

The strongest model does not need to perform every task.

Limit output length

Longer content is not automatically better. Define field-level maximums to prevent the model from producing text that cannot be published.

Use batch or asynchronous processing

Product content rarely needs an immediate response. Batch, Flex, or asynchronous tiers can cost less than standard real-time inference.

Regenerate only failed fields

If the title passes validation but the selling points fail, regenerate only selling_points. Do not pay to recreate the entire page.

Create a content fingerprint

Hash the product facts, prompt version, and model version. If none of them changed, do not call the model again.

Building the Same Workflow with n8n or Make

The pipeline can also be implemented without a custom Python service.

n8n workflow

Read Google Sheets / Excel
→ Split in Batches
→ Normalize data
→ Structured LLM generation
→ JSON Schema validation
→ Prohibited-term check
→ Risk branch
→ CMS or review table
→ Token and cost logging

n8n is a good choice when self-hosting, code nodes, databases, and detailed error handling are important.

Make scenario

Watch Rows
→ Iterator
→ Text / JSON Mapping
→ AI Module
→ Parse JSON
→ Router
→ Low risk: publish to CMS
→ High risk: create a review task

Make is often easier for operations teams and automation projects that benefit from a clear visual canvas.

Whichever platform you choose, “the model call succeeded” must never mean “the content is safe to publish.”

Conclusion

The real challenge in AI-generated product pages is not writing speed. It is ensuring that every statement can be traced to verified product data and remains reviewable, recoverable, and governable at scale.

A reliable system should:

separate facts from marketing language;
prevent the model from guessing missing information;
use a stable JSON Schema;
validate specifications, prohibited claims, and hallucinations with code;
support resumable batches and safe retries;
use different content templates for different channels;
route high-risk outputs to human reviewers;
record model, prompt, token, and content versions.

Once these controls are in place, AI stops being a copywriting shortcut and becomes a controllable product-content production system.

For more AI tools for content production, e-commerce operations, and workflow automation, visit Zyentor Picks at https://www.zyentorpicks.com/.

Originally published on Zyentor Picks.

How to Analyze 10,000 Customer Reviews with AI: VOC Clustering and Action Workflow

cheng zhang — Thu, 30 Jul 2026 05:44:12 +0000

Article Summary

Large review datasets are not useful merely because an AI can produce a positive-versus-negative summary. Companies need to know which issues are growing, which versions and segments are affected, what evidence supports each conclusion, and what action should follow.

This guide presents a reproducible VOC pipeline: ingestion, privacy, deduplication, structured classification, topic clustering, evidence sampling, prioritization, action management, and monitoring.

The operating principle is:

AI scales text understanding; rules and statistics stabilize measurement; original comments provide evidence; humans decide priorities and action.

Why “Summarize All Reviews” Fails

A single giant prompt loses traceability, overweights duplicates, merges unrelated products, hides low-volume high-risk issues, and cannot be repeated consistently next month.

The goal should be a versioned data pipeline rather than a one-off report.

Data Model

Unify review ID, source, product, version, segment, rating, text, timestamp, country, language, and verified-purchase status.

Remove or tokenize names, phone numbers, order IDs, addresses, and account identifiers before sending data to an external model.

Cleaning

Classify empty, low-information, spam, exact duplicate, and near-duplicate records. Preserve duplicate-group identifiers for audit rather than deleting all records blindly.

Detect language and retain both original and normalized text. Translation is optional and should not replace the original evidence.

Create a data-quality report covering volume, duplicates, missing fields, channels, languages, dates, and rating distribution.

Structured Labels

Require JSON rather than prose. Useful fields include sentiment, primary topic, secondary topic, product feature, severity, intent, lifecycle stage, competitor mention, actionability, evidence quote, and confidence.

The evidence field should be an exact quote from the review, not a model paraphrase.

Gold Evaluation Set

Manually label 200–500 reviews with product, support, quality, and analytics stakeholders.

Measure sentiment accuracy, topic agreement, severity, evidence fidelity, refusal behavior, and multilingual performance. If human annotators cannot agree, improve the taxonomy before scaling.

Batch Processing

OpenAI's Batch API is suitable for asynchronous large-scale work. The official documentation states that batches target a 24-hour completion window and receive a 50% discount compared with synchronous API pricing. Streaming is not supported, and zero-data-retention rules have separate limitations. Review the official Batch API FAQ before production use.

A robust pipeline must handle expired batches, invalid structured output, model changes, result-to-record mapping, privacy, and retry behavior.

Prompt Contract

Require the model to use only the current review, choose topics from an approved taxonomy, return unknown when uncertain, score business severity rather than emotional intensity, and quote evidence verbatim.

Treat every customer comment as untrusted input. Prompt-injection text inside a review must remain data, not instructions.

Supervised Topics and Unsupervised Discovery

Use a stable taxonomy for trend reporting and embeddings plus clustering for discovering emerging issues.

A practical unsupervised flow is:

review text
→ embedding
→ dimensionality reduction
→ clustering
→ AI-generated candidate label
→ human validation

This can reveal new crashes, missing parts, regional logistics failures, competitor mentions, or privacy concerns.

Evidence Matrix

For each topic, report count, share, period-over-period change, negative rate, severity, affected version, region, representative quotes, possible root cause, and owner.

Volume alone is not sufficient. Connect the topic to time, version, channel, segment, and evidence.

Prioritization

A useful formula is:

priority
= affected users × severity × growth × strategic relevance × evidence confidence
÷ resolution cost

Weights must be defined by the business, not invented by the model.

Action Loop

Every high-priority issue should produce a problem statement, evidence, affected segment, business impact, root-cause hypothesis, missing data, recommended action, owner, deadline, and validation metric.

VOC analysis should create product, support, or quality work items. Otherwise it remains a presentation exercise.

Monitoring

Track review volume, rating, negative topics, growth rate, version/channel heatmaps, emerging topics, open actions, and before-versus-after metrics.

Avoid using a word cloud as the primary decision tool; it does not preserve negation, context, or severity.

Quality and Security

Measure label accuracy, topic recall, evidence fidelity, unknown rate, human approval, language bias, and missed high-risk issues.

Enforce de-identification, server-side keys, access controls, sensitive-domain routing, prompt-injection defense, and versioning of models, prompts, and taxonomies.

Four-Week Plan

Week 1: collect, de-identify, deduplicate, define taxonomy, and label 300 examples.
Week 2: define schema, pilot the model, and process the full dataset.
Week 3: run embedding clusters, build the evidence matrix, and prioritize.
Week 4: create dashboards and work items, assign owners, and schedule incremental runs.

Final Verdict

A production VOC system is not an AI summary. It is:

traceable data, stable labels, structured batch analysis, emerging-topic discovery, original evidence, business prioritization, and an action loop.

SEO Information

SEO Title: How to Analyze 10,000 Customer Reviews with AI: VOC Clustering and Action Workflow
SEO Description: A practical workflow for review cleaning, structured classification, Batch API processing, embeddings, topic clustering, evidence matrices, prioritization, and action management.
URL Slug: ai-analyze-customer-reviews-voc-topic-clustering-action-guide

For more AI analytics and productivity guides, visit Zyentor Picks.

Originally published on Zyentor Picks.

How to Automate Sales Forecasting with AI: From CRM Data to Rolling Forecasts and Actions

cheng zhang — Wed, 29 Jul 2026 05:32:57 +0000

Article Summary

Many sales organizations still forecast by asking representatives to enter expected revenue, applying manager judgment, and aggregating the result in a spreadsheet. The number is difficult to explain and even harder to improve.

This guide builds a reproducible sales-forecasting workflow for a B2B SaaS company: CRM extraction, data quality, stage calibration, baseline pipeline forecasting, machine-learning win probabilities, close-date distribution, scenarios, Power BI, AI summaries, and action management.

The central rule is:

Deterministic rules own metric definitions, statistical models own probabilities, language models own explanation, and sales leaders own the final commitment.

Why Sales Forecasts Fail

Typical causes:

stale stages;
repeatedly pushed close dates;
inconsistent interpretation of probabilities;
concentration in a few large deals;
weak historical data;
treating total pipeline as revenue;
ignoring cancellations, implementation, and payment;
forecasts without actions.

A useful report separates:

MeasureMeaningBookedSigned or confirmed businessCommitManager-owned period commitmentBest CaseConditional upsideWeighted PipelineAmount multiplied by calibrated probabilityModel ForecastHistorical-data prediction

Case Study

The example company has 12 salespeople, four regions, three product lines, 24 months of opportunity history, about 180 new opportunities per month, and an average 63-day sales cycle.

Leadership wants a Monday report containing:

monthly and quarterly forecasts;
Commit versus model variance;
largest risks;
likely slippage;
regional and product differences;
weekly sales actions.

Architecture

CRM / contracts / payments
→ extraction
→ data-quality gates
→ standardized stages and fields
→ historical snapshots
→ baseline funnel model
→ machine-learning probability
→ scenarios
→ Power BI semantic model
→ AI narrative
→ manager approval
→ email, meeting, and task distribution

Data Dictionary

Minimum opportunity fields:

opportunity ID;
account and owner;
region and product;
amount and currency;
stage;
created, expected-close, and actual-close dates;
outcome;
next action and date;
last activity;
competitor;
source;
discount;
decision-maker and budget confirmation;
technical validation.

Contract and ERP systems should provide signed amount, cancellation, acceptance, payment schedule, and received cash.

A CRM Closed Won record is not automatically recognized revenue.

Data-Quality Gates

Block forecasting when:

close dates are expired or repeatedly changed;
every opportunity closes on the last day of the month;
a stage has not changed beyond its historical P75 duration;
activity is stale;
next steps are missing;
amounts are zero or suddenly increased tenfold;
currencies are inconsistent;
duplicate opportunities exist.

Prevent target leakage by excluding fields only known after the outcome, such as actual payment or final contract identifiers.

Start with a Baseline

Stage-weighted forecast

Weighted Forecast = Sum(Opportunity Amount × Historical Stage Win Rate)

Do not use the same stage probability for every representative. Calibrate by salesperson, segment, product, source, and deal size.

Close-date risk should also reflect stage age, activity, approval status, and previous date changes.

Train a Win-Probability Model

Use interpretable baselines such as logistic regression before complex models.

Salesforce's official Einstein Forecasting considerations state that organizations generally need at least 12 months of opportunity history and forecasts measured by opportunity revenue. The broader lesson is that stable machine-learning forecasting requires enough completed historical outcomes.

Official guidance:

Recommended features:

amount;
stage and stage age;
opportunity age;
days to expected close;
days since activity;
recent activity count;
decision-maker and budget confirmation;
product, region, source, and competitor;
representative historical win rate;
discount;
customer purchase history.

A time-based train/test split is mandatory. Random splitting can leak future behavior into training.

Measure both ranking and probability calibration. A model that says “80%” should win close to 80% over time.

Forecast the Timing

Win probability alone does not answer when a deal will close.

Estimate:

win probability;
slippage probability;
monthly close distribution;
expected amount;
cancellation probability.

A USD 500,000 opportunity might contribute USD 175,000 to July, USD 150,000 to August, USD 75,000 to September, and zero to the failure/later bucket.

This is more realistic than assigning the entire opportunity to one manually selected month.

Scenarios

Create conservative, base, and upside scenarios.

A management summary should look like:

Quarter forecast range: USD 8.6M–10.4M
Base forecast: USD 9.3M
Sales Commit: USD 10.1M
Commit exceeds model forecast by USD 0.8M.
The gap is concentrated in two large East-region opportunities with budget and close-date risk.

A forecast range is more honest than false precision.

Power BI Semantic Model

Recommended tables:

Opportunity Snapshot fact;
Contract fact;
Payment fact;
Date, Salesperson, Region, Product, and Stage dimensions.

Metrics:

pipeline;
weighted pipeline;
model forecast;
Commit;
Booked;
forecast gap;
win rate;
slippage rate;
cycle length;
stale-opportunity rate;
forecast accuracy.

Power BI Copilot can assist with report creation and explanation, but Microsoft currently requires a paid Fabric capacity (F2 or higher) or Power BI Premium P1 or higher. Power BI Pro, PPU, and trial capacity alone do not satisfy the Copilot capacity requirement.

Official overview:

Power BI can also include Copilot-generated report summaries in email subscriptions.

Official overview:

Use the LLM for Narrative, Not Arithmetic

Provide computed JSON and instruct the model not to recalculate.

Act as a sales operations analyst.
The metrics below were calculated by the semantic model. Do not change them.

Produce:
1. a 200-word executive summary;
2. Commit versus model variance;
3. the five largest risks;
4. regional and product anomalies;
5. weekly actions with owner, deadline, and verification criteria;
6. clear labels for fact, inference, and recommendation.

Never describe total pipeline as forecast revenue.
Do not invent customers, dates, amounts, or next steps.

Convert Forecasts into Actions

Examples:

RiskActionNo activity for 14 daysUpdate next step within 48 hoursDecision-maker unknownSchedule executive discoveryBudget unconfirmedCreate value and approval planTechnical validation failedOpen remediation planClose date repeatedly pushedReclassify quarter ownershipUnusual discountRequire manager approval

Automation:

Monday refresh
→ quality checks
→ model run
→ Power BI update
→ AI summary
→ manager approval
→ email distribution
→ CRM tasks for high-risk deals
→ Friday action review

Evaluation

Track:

WAPE or absolute forecast error;
direction accuracy;
probability calibration;
Commit variance;
slippage;
stale-opportunity rate;
next-step completeness;
risk recovery;
time saved in forecast meetings.

Do not optimize only for offline model accuracy.

Common Failure Modes
overwriting historical stages;
random train/test splitting;
asking an LLM to invent probabilities;
mixing contract, revenue, and cash;
treating Commit as objective prediction;
target leakage;
ignoring large-deal concentration;
publishing one exact number without a range;
allowing the LLM to recalculate metrics;
sending reports without manager approval.

Final Verdict

A reliable AI forecasting system is:

facts → quality gates → probabilities → scenarios → AI explanation → human commitment → actions.

Start by cleaning CRM definitions, build a simple stage-weighted baseline, add machine learning only after historical snapshots are trustworthy, use Power BI for controlled metrics, and use the language model for narrative and action generation.

AI improves discipline and speed, but the sales leader remains accountable for Commit.

SEO Information

SEO Title: How to Automate Sales Forecasting with AI: From CRM Data to Rolling Forecasts and Actions
SEO Description: A practical enterprise AI sales forecasting guide covering CRM cleaning, stage calibration, machine learning, close-date risk, Power BI Copilot, scenarios, narratives, evaluation, and action automation.
URL Slug: ai-sales-forecasting-crm-data-rolling-forecast-action-guide

For more AI tool reviews, practical guides, and industry analysis, follow Zyentor Picks.

Originally published on Zyentor Picks.

How to Automate Invoice and Expense Processing with AI: OCR, Approval, and Anomaly Detection

cheng zhang — Tue, 28 Jul 2026 02:50:24 +0000

How to Automate Invoice and Expense Processing with AI: OCR, Approval, and Anomaly Detection

Article Summary

Invoice automation is more than converting an image into text. A production workflow must ingest documents, extract fields and line items, match vendors and purchase orders, validate totals and tax, detect duplicates, check budgets and permissions, route human review, obtain approval, post to finance systems, archive evidence, and preserve an audit trail.

This guide designs a practical workflow using Power Automate, AI Builder, or Azure Document Intelligence.

The central rule is:

AI extracts and classifies, deterministic rules validate, business systems own financial state, and humans approve high-risk decisions.

Target Workflow

A company receives 3,000 supplier invoices per month. The current process is email download, manual spreadsheet entry, purchase-order checks, approval, ERP entry, and PDF archiving.

The automated target is:

more than 90% extraction on standard invoices;
confidence stored for every field;
low-confidence cases routed to humans;
automatic duplicate, budget, and PO mismatch detection;
traceable approvals;
only validated records posted to finance systems.

Reference Architecture

Email / SharePoint / Portal / Scanner
→ Power Automate trigger
→ file security and classification
→ AI Builder or Azure Document Intelligence
→ normalization
→ vendor, PO, and receipt matching
→ rules and anomaly signals
→ Power Apps human review
→ approval
→ ERP / finance system
→ archive
→ Power BI monitoring

Microsoft's official reference architecture uses Power Automate for orchestration, AI Builder for extraction, Power Apps for correction, Dataverse for queues and records, and Power BI for analysis: AI document-processing architecture.

Tool Selection

AI Builder and Power Automate

Best when the organization already uses Microsoft 365 and Power Platform and finance teams need a low-code solution.

The prebuilt invoice model extracts invoice ID, supplier, dates, totals, due dates, purchase-order numbers, and other common fields. Simplified Chinese is supported. Custom document models can be trained for additional fields, with Microsoft documentation stating that teams can start with five documents.

Official information: prebuilt invoice model and custom document processing.

Azure Document Intelligence

Best for API integration, large volume, custom applications, structured JSON, and developer-managed infrastructure.

Azure Document Intelligence v4.0 includes prebuilt invoice, layout, and custom models. Official information: overview and invoice model.

OCR plus a language model

A language model can normalize vendor names, map descriptions to expense categories, explain exceptions, and generate approval summaries. It should not independently calculate tax, approve payments, or decide fraud.

Data Model

Store three layers.

Raw

Original file, email ID, uploader, timestamp, hash, OCR text, and model version.

Extracted

Invoice number, invoice date, raw vendor name, tax ID, currency, subtotal, tax, total, PO number, due date, line items, and confidence.

Business

Canonical vendor ID, account code, cost center, project, remaining budget, match status, risk level, approval status, and ERP voucher ID.

Never overwrite raw values. Store normalized values separately with the rule or reviewer responsible for the change.

Power Automate Flow
Trigger from email, SharePoint, Power Apps, Teams, or API.
Validate type, size, page count, encryption, duplicates, malware, and multi-invoice PDFs.
Call AI Builder invoice extraction or a document model.
Normalize names, dates, currency, decimal values, and invoice IDs.
Perform three-way matching across invoice, purchase order, and receipt/service acceptance.
Execute anomaly rules.
Route to review and approval.
Post validated data to ERP.

Microsoft recommends specifying pages when a large file contains only one invoice, reducing prediction cost and improving performance. Official guide: invoice processing in Power Automate.

Deterministic Checks

Examples:

subtotal plus tax does not equal total;
supplier tax ID and invoice number already exist;
duplicate file hash;
future invoice date;
due date earlier than invoice date;
insufficient PO amount;
bank account differs from master data;
submitter is also approver;
budget exceeded;
extraction confidence below threshold.

AI can flag semantic anomalies such as an invoice description that conflicts with the purchase purpose or a note requesting payment to a personal account. These are risk signals, not proof of fraud.

Human Review

A useful review screen shows:

original PDF with highlighted evidence;
extracted fields and confidence;
purchase order and receipt data;
anomaly reasons;
change history;
approve, return, and escalate actions.

Low-risk matched invoices can move directly to approval. Low-confidence or medium-risk invoices require finance review. Duplicate, bank-account-change, or high-risk cases require senior finance or audit review.

Security and Audit

Implement minimum privilege, service-account separation, encryption, retention policies, tamper-resistant logs, masking, model and prompt versioning, reviewer history, deprovisioning, and environment separation.

Do not upload real supplier invoices to personal free AI accounts. Bank accounts, tax IDs, and employee information are sensitive business data.

Pricing

Microsoft's Singapore pricing page currently lists:

Power Automate Premium: USD 15/user/month, paid yearly;
Power Automate Process: USD 150/bot/month, paid yearly;
Hosted Process: USD 215/bot/month, paid yearly.

Prices exclude GST and can vary by contract. Official page: Power Automate pricing.

AI Builder consumes AI credits. Azure Document Intelligence uses pay-as-you-go or commitment pricing rather than a fixed monthly subscription: Azure pricing.

Evaluate total cost per invoice, including licenses, recognition, storage, ERP integration, human review, and maintenance.

Evaluation

Build an anonymous test set of at least 200 documents covering clean PDFs, phone photos, multilingual invoices, multiple currencies, complex line items, low-quality scans, duplicates, and anomalies.

Measure field accuracy, line-item accuracy, straight-through rate, incorrect auto-approval rate, duplicate recall, average review time, failure rate, cycle time, and cost per invoice.

The most important metric is not OCR accuracy. It is the rate at which high-risk errors are incorrectly allowed through.

Four-Week MVP

Week 1

Collect samples, define fields, rules, approval matrix, and ERP interfaces.

Week 2

Build ingestion, extraction, raw/structured storage, and the review interface.

Week 3

Add vendor/PO/receipt matching, duplicate rules, approval routing, retries, and alerts.

Week 4

Pilot with one department and a limited supplier group while running the manual process in parallel.

Final Verdict

AI reads documents, rules validate facts, systems execute the process, and humans remain accountable.

Power Automate plus AI Builder is the fastest low-code route for Microsoft organizations. Azure Document Intelligence is more flexible for API-driven and high-volume systems. Both require human review, deterministic validation, authorization, and auditability.

SEO Information

SEO Title: How to Automate Invoice and Expense Processing with AI
SEO Description: A practical guide to invoice OCR, AI extraction, three-way matching, duplicate detection, approvals, human review, Power Automate, AI Builder, Azure Document Intelligence, pricing, and security.
URL Slug: ai-invoice-expense-automation-ocr-approval-anomaly-detection-guide

For more enterprise AI automation guides, visit Zyentor.

Originally published on Zyentor Picks.

DEV Community: cheng zhang

Qwen3 Embedding on Cloud TPU: Production Long-Context Retrieval with vLLM

Article Summary

Two different workloads

Why long-context embeddings matter

Why native vLLM TPU support matters

Heterogeneous elasticity with GKE

Embedding correctness is stricter than generation correctness

Golden-reference testing

Do not benchmark only QPS

Why chunked prefill is difficult for embeddings

StepPool and cached state

Tensor alignment

JAX/XLA warm-up

Published throughput result

Why requests/s may look modest

Multimodal serving is harder

Recommended enterprise architecture

Separate batch and online capacity

The embedding gateway should standardize

Use dual indexes for model upgrades

Is TPU always better?

Metrics that matter

Conclusion

Qwen3 Embedding on Cloud TPU: Production Long-Context Retrieval with vLLM

Article Summary

Two different workloads

Why long-context embeddings matter

Why native vLLM TPU support matters

Heterogeneous elasticity with GKE

Embedding correctness is stricter than generation correctness

Golden-reference testing

Do not benchmark only QPS

Why chunked prefill is difficult for embeddings

StepPool and cached state

Tensor alignment

JAX/XLA warm-up

Published throughput result

Why requests/s may look modest

Multimodal serving is harder

Recommended enterprise architecture

Separate batch and online capacity

The embedding gateway should standardize

Use dual indexes for model upgrades

Is TPU always better?

Metrics that matter

Conclusion

In the AI Coding Era, Go’s Advantage Is Verifiability

Article Summary

Generation is no longer the bottleneck

gofmt reduces review noise

The compiler is a cheap deterministic critic

Fast compilation also affects AI cost

A strong standard library reduces dependency hallucination

Module checksums matter more with autonomous dependency changes

govulncheck gives actionable vulnerability feedback

Testing has a clear default path

Native fuzzing is especially useful for AI-generated code

gopls and modernizers make refactoring more deterministic

“AI-friendly language” needs a new definition

Does this make Go universally better than Python?

A concrete coding-agent verification loop

The broader lesson is deterministic feedback

Review capacity becomes the limiting resource

Conclusion

In the AI Coding Era, Go’s Advantage Is Verifiability

Article Summary

Generation is no longer the bottleneck

gofmt reduces review noise

The compiler is a cheap deterministic critic

Fast compilation also affects AI cost

A strong standard library reduces dependency hallucination

Module checksums matter more with autonomous dependency changes

govulncheck gives actionable vulnerability feedback

Testing has a clear default path

Native fuzzing is especially useful for AI-generated code

gopls and modernizers make refactoring more deterministic

“AI-friendly language” needs a new definition

Does this make Go universally better than Python?

A concrete coding-agent verification loop

`gofmt` reduces review noise

`govulncheck` gives actionable vulnerability feedback

`gopls` and modernizers make refactoring more deterministic

`gofmt` reduces review noise

`govulncheck` gives actionable vulnerability feedback

`gopls` and modernizers make refactoring more deterministic