DEV Community: Agbo, Daniel Onuoha

The Next Phase of Vision-Language Navigation

Agbo, Daniel Onuoha — Fri, 04 Sep 2026 16:43:04 +0000

TL;DR: Vision-Language Navigation has moved well past indoor discrete step benchmarks. The next phase is defined by self-evolving agents, long-horizon multi-stage planning, aerial and outdoor deployment, zero-shot generalization, and the convergence of VLN with Vision-Language-Action models. This article maps those shifts in detail — and explains why each of them matters.

Where VLN Started — and Why That Baseline No Longer Holds

Vision-Language Navigation was formalized as a research challenge around 2018 with the R2R benchmark: given a natural language instruction and a panoramic first-person view, navigate to the described destination inside a photorealistic indoor environment.

The setting was deliberately constrained. Agents moved along discrete graph nodes, environments were static, instruction lengths were manageable, and success was measured by a single clean terminus. Those constraints were valuable — they made the problem tractable enough to study rigorously. They also produced a deceptively narrow definition of what navigation with language really needs to be.

By 2025 and into 2026, the field has largely exhaused what clean indoor discrete-action benchmarks can teach us. The interesting research is now happening at the edges: longer task horizons, open-world outdoor environments, aerial platforms, agents that learn continuously, and architectures where navigation is no longer a standalone skill but a component of a broader action-capable system.

Shift 1 — Self-Evolving Agents

One of the cleanest limitations of previous VLN systems was their static knowledge base. After training, the agent's internal model of the world was fixed. It could not update from deployment experience, and it could not integrate knowledge from successful or failed episodes into future decisions.

Research presented at ICLR 2026 introduced SE-VLN, a self-evolving vision-language navigation framework that addresses this directly. SE-VLN is built on a multimodal large language model backbone and adds three core modules: a hierarchical memory system that converts successful and failed navigation episodes into reusable structured knowledge, a retrieval-augmented reasoning module that queries that memory during live navigation, and a reflection module that updates the agent's behavior based on post-episode analysis.

The reported results are significant. SE-VLN achieved navigation success rates of 57% and 35.2% in unseen environments on the R2R and REVERIE benchmarks respectively, representing relative improvements of 23.9% and 15% over prior state-of-the-art. Critically, performance improved as the experience repository grew, demonstrating genuine continual learning behavior rather than just stronger initialization.

The implication for practitioners is important: the model's usefulness compounds over deployment time. That is a fundamentally different capability profile than a system that decays or plateaus.

Shift 2 — Long-Horizon Multi-Stage Navigation

Standard VLN benchmarks were built around single-stage tasks. Navigate from point A to point B. The implicit assumption was that the navigation episode was a single coherent unit of behavior.

Real-world navigation tasks rarely look like that. A hospital delivery robot may need to collect a sample from one floor, confirm patient identity at another, and deliver to a lab in a third wing — all within a single natural language instruction sequence. A household assistant asked to "clean up and then start a load of laundry" needs to execute multiple sequenced subtasks with state tracking and conditional branching.

CVPR 2025 addressed this directly with the introduction of Long-Horizon Vision-Language Navigation (LH-VLN), a new task definition accompanied by the NavGen data generation platform and the LHPR-VLN benchmark. LHPR-VLN consists of 3,260 tasks with an average of 150 task steps — an order of magnitude more complex than standard benchmarks. To evaluate these tasks meaningfully, the authors introduced three new metrics: Independent Success Rate, Conditional Success Rate, and CSR weighted by Ground Truth, to capture partial completion performance that binary success/failure metrics cannot express.

The accompanying method, Multi-Granularity Dynamic Memory (MGDM), integrates short-term memory blurring with long-term memory retrieval to enable flexible navigation in dynamic, multi-stage environments.

The benchmark framing itself matters as much as the method. By formally defining LH-VLN and providing the evaluation infrastructure, this work established a new standard for what navigation complexity should look like in research.

Shift 3 — Cognitive Maps and Spatial Memory

Most VLN systems process navigation as a reactive sequence: observe, attend to instruction, predict the next action, step, repeat. This works acceptably for short, well-described paths in compact environments. It breaks down badly at scale.

In large-scale environments — outdoor city blocks, multi-building campuses, sprawling factory floors — the reactive paradigm is insufficient because the agent has no persistent spatial representation to reason over. It cannot connect what it sees now to where it has been, what structure the environment has, or which direction to commit to when landmarks are ambiguous.

CogVLN, presented at ICLR 2026, addresses this by constructing a cognitive map before and during navigation. Inspired by how humans mentally encode environments, CogVLN prioritizes encoding of key scenes that carry high environmental distinctiveness, while allocating fewer encoding resources to visually redundant areas. Built on a multimodal large language model, the system uses the cognitive map to drive three modules: a localization module that identifies start and goal vertices, a path planning module that generates traversal routes, and a navigation module that executes those routes while handling user feedback interactively.

Validation in the CARLA Town01 and Town07 environments — large-scale driving simulation — demonstrated strong generalization performance. The outdoor setting is notable: most prior VLN research was validated in indoor room-scale environments.

Shift 4 — Outdoor and Urban-Scale Deployment

The expansion from indoor to outdoor environments is not just a dataset change. It represents a qualitative shift in the problem structure.

Indoor VLN benefits from clearly bounded spaces, structured landmark vocabulary, predictable lighting, and a manageable graph of traversable waypoints. Outdoor navigation introduces GPS ambiguity at fine scales, variable occlusion from moving vehicles and pedestrians, weather-dependent visual appearance, landmark impermanence, and action spaces that include continuous speed and heading rather than discrete graph steps.

Recent research in 2025 and 2026 has pushed into this territory explicitly. Vision-language models applied to mobile robot navigation in unstructured manufacturing environments demonstrated a VLM-based architecture integrating 3D scene reconstruction, semantic segmentation via LSeg, and LLM-driven spatial goal navigation — achieving a 92.5% average success rate across different navigation subgoal configurations in simulation, with real-world deployment on a TIAGo++ platform.

The larger research direction involves grounding natural language not in 3D room graphs but in semantic maps built from real sensor data, with online updates as the environment changes.

Shift 5 — Aerial Vision-Language Navigation

Ground-based navigation has dominated VLN research since the field's inception. UAV-based VLN presents a distinctly harder problem and has remained underexplored until recently.

The core challenges are not simply a rescaling of the ground problem. Aerial agents operate in a continuous 6-DOF action space rather than ground-plane discrete steps. The visual perspective is fundamentally different — top-down, oblique, and shifting with altitude. The dynamic range of environments navigable by a UAV is vastly larger than any indoor graph. And the safety implications of navigation failure are different: a UAV that misidentifies a waypoint does not just stop at the wrong room — it may exit controlled airspace or lose signal.

Research published in 2025 and 2026 has begun building the infrastructure needed to study this problem properly. The OpenUAV platform provides diverse environments, realistic flight control physics, and algorithm support specifically designed for VLN tasks. The UAV-Need-Help benchmark introduces approximately 12,000 trajectories with varying levels of assistant guidance, providing a tiered evaluation framework that distinguishes between agents that can navigate with full instruction sets and those that require interactive clarification.

OpenVLN extended this further with an open-world aerial VLN framework targeting general outdoor environments. The research consistently identifies a significant gap between current model performance and human operator performance — marking UAV-VLN as one of the most open and tractable research frontiers in the field.

Shift 6 — Zero-Shot Generalization with Multimodal LLMs

A persistent frustration in VLN has been the brittleness of trained agents to distribution shift. A model trained on R2R instructions performs well on R2R-style paths but degrades sharply when instruction phrasing, environment style, or object vocabulary shifts. Supervised models overfit to benchmark conventions rather than developing genuine spatial and linguistic reasoning.

The emergence of strong multimodal large language models has reframed this challenge. Rather than training a navigation-specific policy from scratch, the new paradigm leverages the broad visual and linguistic knowledge already encoded in a foundation model.

UniGoal, presented at CVPR 2025, proposed a universal zero-shot goal-oriented navigation framework that unifies object category, instance image, and text description goals into a single graph representation. An online scene graph tracks the agent's environmental observations, and an LLM performs explicit graph-based reasoning to match scene state against goal state. UniGoal achieved state-of-the-art zero-shot performance across three navigation tasks with a single model, outperforming task-specific zero-shot methods.

NavBench, published in mid-2025, evaluated this zero-shot navigation paradigm more systematically. Across 3,200 question-answer pairs and 432 full episodes in 72 indoor scenes, the benchmark tested comprehension across global instruction alignment, temporal progress estimation, and local observation-action reasoning. GPT-4o performed strongest, while lighter open-source models succeeded in simpler scenarios. The consistent finding was that models with higher navigation comprehension scores achieved better execution outcomes — confirming that language understanding quality and navigation quality are coupled, not independent.

The persistent weak point across models was temporal understanding: estimating progress through a navigation episode. This is a significant gap because long-horizon tasks require the agent to know not just where it is but how far into the instruction sequence it has advanced.

Shift 7 — The Convergence of VLN and VLA

The most structurally significant shift in the field is arguably the convergence of Vision-Language Navigation with Vision-Language-Action models.

Classic VLN treated navigation as an isolated capability: perceive, ground instruction, move. The output was a trajectory — a sequence of waypoints or steps. The gap between that output and genuine embodied task completion was never addressed in the VLN framing.

VLA models dissolve that separation. Architectures in the RT-2, OpenVLA, and ChatVLA-2 lineage extend the VLM paradigm all the way to physical action tokens, mapping visual observations and natural language instructions directly to motor commands. Navigation becomes one component of a unified perception-language-action loop rather than a separate planning problem that upstream some physical controller.

ChatVLA-2, presented at NeurIPS 2025, introduced a mixture-of-expert VLA architecture with a three-stage training pipeline designed to preserve the VLM's core reasoning capabilities during robotic fine-tuning. The system demonstrated that mathematical reasoning and spatial intelligence acquired during VLM pretraining transferred into navigation and manipulation tasks without explicit retraining — a result with significant implications for generalization.

Figure AI's Helix VLA model extended this to full humanoid upper-body control: arms, hands, torso, and fingers controlled through the same VLA framework that processes natural language task descriptions. The gap between "navigate to the kitchen" and "navigate to the kitchen and put the cup in the dishwasher" is shrinking.

The Open Problems

Despite the pace of progress, several fundamental challenges remain unsolved and are worth naming clearly.

Temporal understanding. As NavBench demonstrated, current multimodal LLMs struggle to estimate navigation progress through a long instruction sequence. This is a prerequisite for reliable long-horizon task execution.

Efficient edge deployment. VLA models powerful enough to handle open-world navigation carry large memory and compute footprints. Deploying them on embedded hardware — the compute available on a mobile robot or UAV — requires architectural innovations in model compression, token efficiency, and real-time inference optimization.

Continual learning without forgetting. SE-VLN showed that experience-based evolution is possible, but the broader challenge of learning new environments and tasks without catastrophic forgetting of previously learned capabilities remains open.

Sim-to-real for diverse embodiments. Domain randomization improves transfer, but the visual and physical gap between simulation and the real world remains significant for outdoor environments, aerial platforms, and novel object categories.

Safety and failure semantics. As VLN agents move from indoor benchmarks to real deployments — hospitals, warehouses, public spaces — the definition of safe failure behavior needs to be formalized. An agent that is uncertain should know what to ask, when to stop, and how to communicate its state.

What the Next Phase Looks Like

The research trajectory is now reasonably clear. VLN in its next phase will:

Operate continuously in open-world environments, updating spatial and semantic memory from live experience
Handle multi-stage, multi-subtask instruction sequences with hundreds of steps
Extend across aerial, ground, and mobile manipulation platforms within unified model architectures
Generalize zero-shot to novel environments, object classes, and instruction styles using foundation model priors
Converge with the broader VLA paradigm, where navigation is not a separate module but an integrated capability within an action-capable embodied agent

The benchmark infrastructure is being built now. The architectural foundations — multimodal LLMs, cognitive maps, self-evolving memory, long-horizon planning — are either available or within reach. The primary constraint shifting from research to deployment is the same one it always is: the gap between what works in simulation and what works in the real world, at scale, reliably, across the full distribution of environments an agent will encounter.

That gap is the next three years of work in this field.

New post: Programmable Wallets for the Agentic Internet 💳🤖 Covers how x402 revives HTTP 402 for machine-to-machine stablecoin payments, the three-actor model (Client/Resource Server/Facilitator), what makes a wallet "agentic" vs. just a crypto wallet wi

Agbo, Daniel Onuoha — Wed, 02 Sep 2026 13:19:11 +0000

Agbo, Daniel Onuoha

Aug 30

Programmable Wallets for the Agentic Internet

#ai #programming #blockchain #automation

5 min read

Programmable Wallets for the Agentic Internet

Agbo, Daniel Onuoha — Sun, 30 Aug 2026 05:00:00 +0000

Agents are coming online at a scale that makes human-in-the-loop payments impractical, and the industry's answer has converged on an unlikely source: a dormant HTTP status code from the 1990s. The x402 protocol revives HTTP 402 "Payment Required" to let AI agents pay for APIs, data, and services directly in stablecoins, with no accounts, API keys, or checkout pages — and as of July 2026, agent-driven traffic is already overtaking human traffic on some of its earliest adopting platforms.

Why Agents Need Wallets, Not Accounts

Every payment system built for the human web assumes a person is present to log in, enter card details, or approve an OAuth prompt. An autonomous agent completing a task — sourcing a freight quote, pulling a paid dataset, calling a monetized API a hundred times in sequence — can't pause for a human to approve each of those actions without defeating the purpose of automation. What agents need instead is a wallet: a programmable store of funds governed by rules set in advance, so the agent can spend within defined limits without asking permission for every transaction.

This is the shift x402 was built to enable — machine-to-machine payments that happen inline, within the same HTTP request cycle that fetches the resource, rather than through a separate billing relationship set up by a human beforehand.

How x402 Actually Works

x402 uses a three-actor model: the Client (the buyer — an AI agent, app, or script), the Resource Server (the seller — an API or paid service), and the Facilitator (a settlement service that verifies and broadcasts payments on-chain). The flow runs in a single retried request cycle, with no redirects, webhooks, or manual checkout involved:

Client makes a request. A normal GET or POST to a paid endpoint — no auth headers, no API key.
Server responds 402 Payment Required. The response includes structured payment metadata: the price, accepted stablecoin (usually USDC), the destination address, and which blockchain network to use.
Client signs a payment authorization. The agent's wallet signs an off-chain authorization for the exact amount using EIP-3009 transfer authorizations on EVM chains — this step requires no gas fee from the client.
Client retries the request with the signed payment attached in a header (historically X-PAYMENT, now standardizing toward PAYMENT-SIGNATURE in x402 v2).
Server verifies and settles. The server checks the signature — either locally or by calling the facilitator's /verify endpoint — then settles the payment on-chain directly or via the facilitator's /settle endpoint.
Server returns 200 OK with the resource and a settlement confirmation header proving the transaction went through.

The elegance of this design is that it needed almost no new infrastructure — HTTP already reserved status code 402 for exactly this purpose decades ago; it simply never had a standard payment mechanism to pair with it until now.

What Makes a Wallet "Agentic"

A programmable wallet built for this world looks structurally different from a personal crypto wallet. Rather than a human approving each transaction, the wallet enforces policy automatically:

Spending caps — hard limits on how much the agent can spend per transaction, per hour, or per day
Allowed actions — a whitelist of what the wallet can be used for (e.g., only data purchases, not arbitrary transfers)
Approved counterparties — restricting payments to a known set of vetted services or domains
Policy checks before execution — every time the agent decides it needs to spend funds, the wallet checks the action against policy before signing anything, and only executes if it passes

This is the piece that makes autonomous spending safe enough to deploy: the agent has genuine spending authority, but only within boundaries a human or organization defined ahead of time — closer to a corporate purchasing card with programmable limits than a blank check.

Real Implementations Already Live

This isn't theoretical. Coinbase's Agent Kit gives agents a skills library for wallet operations — sending, trading, earning — and integrates natively with x402 for agent-to-service payments, including gasless transactions on Base. Cloudflare has shipped x402 support directly into its Agents SDK, letting any Cloudflare Worker accept or make agent payments with a few lines of middleware. Coinbase opened x402 USDC payments to all Business customers on July 23, 2026, explicitly citing that agent traffic was already overtaking human traffic on its developer documentation.

The protocol also works across multiple chains — Base, Ethereum, Arbitrum, Optimism, Polygon, and Solana are all supported, which matters because it means an agent's wallet isn't locked into one ecosystem's liquidity or fee structure.

x402 vs. the Other Agent Payment Standards

x402 sits in a specific niche within the broader agent payment landscape, distinct from protocols like AP2 or MPP that focus more on authorization mandates and card-network integration.

Aspect	x402	AP2 / Card-Network Protocols
Settlement rail	On-chain stablecoins (USDC)	Traditional card/bank rails with cryptographic mandates
Best fit	Pay-per-call APIs, micropayments, data access	Larger consumer purchases, subscriptions
Setup	No account, no API key needed	Requires to be registered merchant/payment relationships
Speed	Settles within the same HTTP request	May involve separate authorization and settlement steps
Fee structure	Near-zero for stablecoin transfers	Standard card-network interchange fees apply

x402 is best understood as a settlement layer built specifically for agents and services, not a consumer checkout replacement — it's optimized for high-frequency, low-value machine payments where card-network fees would make the transaction uneconomical.

Monetizing Your Own APIs with x402

For a backend developer, adding x402 support to an existing API is a middleware-level change, not an architectural rewrite. Server-side, you add a payment-checking middleware (Express and other frameworks already have @x402/express-style packages) that intercepts unpaid requests and returns the 402 challenge with your pricing and wallet address, then verifies incoming payment signatures before letting the request through.

A minimal server-side pricing structure typically involves defining a discovery document — an OpenAPI-style manifest describing which endpoints are paid, the price per call, and accepted payment methods — so agents (or the MCP servers they use) can find and price your service programmatically before ever making a request. This is the same discoverability principle from MCP tool discovery, applied to payments: an agent shouldn't need to guess pricing through trial and error.

What to Watch as the Standard Matures

x402 is actively evolving — a v2 revision is standardizing header naming (PAYMENT-REQUIRED, PAYMENT-SIGNATURE, PAYMENT-RESPONSE) and network identifiers using CAIP-2 chain formats, and developers building against it now should track which version their facilitator and client libraries actually support, since v1 and v2 agents can silently filter each other out if accepts fields aren't handled correctly. Identity is also an open frontier — projects are already layering verified identity on top of x402's base payment flow, since the protocol itself proves a payment happened but says little about who or what authorized it.

Where This Is Heading

The practical shift for anyone building fintech or API-monetized products is this: pricing your service per-request, in stablecoins, discoverable by agents without a sales conversation, is quickly becoming a viable revenue model rather than a novelty. An agent buying a freight quote, a weather data point, or a single inference call from your API, paying in the same request that fetches it, is the kind of transaction volume that never made sense under card-network fees — and it's exactly the volume x402 was built to unlock.

How Autonomous Agents Operates

Agbo, Daniel Onuoha — Fri, 28 Aug 2026 05:00:00 +0000

Inside an Autonomous Agent: Discover via MCP, Coordinate via A2A, Transact via MPP

An autonomous agent completing a real task rarely uses one protocol — it typically discovers a tool through MCP, hands off part of the work to another agent through A2A. It pays for a service through MPP, all within a single request chain. Tracing that flow end-to-end shows how these three protocols function together as layers of one stack rather than competing standards.

Step 1: Tool Discovery via MCP

Before an agent can act, it needs to know what tools exist and what they do — this is MCP's job.

The handshake. Discovery doesn't start with a plain "list tools" call. The client sends a JSON-RPC 2.0 initialize request describing its own capabilities (like supportsToolDiscovery and maxToolCount), and the server responds with its own capability set. Only after this mutual negotiation does actual tool enumeration begin.

Static vs. dynamic discovery. Once connected, an agent can pull tools three ways: statically defining them in code ahead of time, dynamically fetching the current list at runtime, or using a search function that performs semantic matching over available tools based on the user's actual request. Dynamic discovery matters increasingly in production because it lets a platform update its available tools without repackaging or republishing the agent — the client diffs the current tool list against what it last knew and applies changes automatically.

Discovery at scale. In enterprise deployments, an MCP Gateway sits between the agent and multiple MCP servers, filtering which tools an agent can even see based on its identity, workspace, and permission policies — discovery answers "what exists," while a separate authorization step governs "how it can be called". Some gateways go further with a two-step catalog pattern: the agent first calls a lightweight discover_tools meta-tool to browse categories and names without full schemas, then calls select_tools to scope its session to a specific subset before receiving the full parameter schemas — this keeps context windows small and improves tool-selection accuracy.

For a fintech backend, this means your existing REST endpoints — balance checks, transaction lookups — become MCP tools with structured schemas, and any agent (yours or a third party's) can discover and call them without a bespoke integration per client.

Step 2: Coordination via A2A

Once an agent has tools, complex tasks often require delegating part of the work to another, independently built agent — this is what A2A standardizes.

Agent Cards. Every A2A-compliant agent publishes a JSON capability descriptor called an Agent Card at a well-known path: /.well-known/agent.json. This card gives a complete picture of what the agent can do, how to reach it, and what parameters it expects — the equivalent of a service's OpenAPI spec, but designed for another agent to read.

Discovery and delegation flow. A client agent first fetches the target agent's card from that well-known URL and uses it to build a connection. When it needs that agent's help, it sends a message containing the task along with session metadata like a session ID and historical context. The receiving A2A server evaluates the incoming message as a Task to complete, rather than treating it as a simple stateless API call.

Why this differs from a plain API call. A2A tasks carry state — the receiving agent can report back partial progress, ask clarifying questions, or return a task in an intermediate status, not just a final response. This matters for genuinely long-running or multi-step delegation, like one agent asking a separate logistics agent to "find the fastest delivery route" while a parent workflow waits on updates rather than blocking on one synchronous request.

In a fintech scenario, this looks like a customer-facing support agent discovering a specialized fraud-review agent via its Agent Card, delegating a suspicious transaction for deeper analysis, and continuing the conversation once that sub-agent reports back — all without the two systems having been built by the same team or sharing a private API contract.

Step 3: Transacting via MPP

Once the right tool is found and any needed coordination is done, the agent may need to pay for the resource itself — this is where MPP (Machine Payments Protocol) takes over.

The core handshake is HTTP-native. MPP, co-authored by Stripe and Tempo and launched in March 2026, layers payment directly onto standard HTTP requests using the existing 402 Payment Required status code. The full flow runs in a single request cycle with no redirects or webhooks required:

The agent sends a standard HTTP request (a GET, for instance) to a paid endpoint
The server responds with 402 Payment Required and a WWW-Authenticate: Payment header specifying price, accepted currencies, recipient details, and supported payment methods
The agent selects a payment method and authorizes payment — via stablecoin transfer, card payment, or another supported rail
The agent resends the original request, this time including payment credentials in an Authorization: Payment header
The server verifies the payment and returns the resource along with a Payment-Receipt header as proof

Beyond one-shot payments. MPP layers four additional primitives on top of that base handshake: recurring subscriptions, streaming (metered, continuous) payments, cancellation events, and balance reconciliation — all addressed over the same HTTP surface rather than requiring separate billing infrastructure.

MCP tools can be monetized directly. MPP explicitly supports paying per MCP tool call — an agent can call a monetized MCP server and pay per invocation without any OAuth flow or account setup, closing the loop between the discovery layer and the payment layer.

Discovery of paid services. Just as MCP has tool discovery, MPP has service discovery: providers can advertise their API's payment terms through an OpenAPI-style discovery document, and agents can find paid APIs through directories like the mpp.dev catalog or via MCP servers built specifically for that purpose.

Putting the Full Chain Together

Consider an autonomous procurement agent tasked with sourcing the cheapest available freight quote for a logistics shipment:

MCP discovery — the agent queries its MCP Gateway, which filters and returns only the freight and logistics tools it's permitted to use, based on its identity and workspace policy
A2A delegation — instead of calling every carrier directly, it fetches the Agent Card of a specialized freight-broker agent and delegates the task of gathering quotes, passing along shipment details and a session ID
MPP settlement — once a quote is selected, the broker agent's booking endpoint returns a 402 with the price and accepted payment methods; the procurement agent authorizes payment via its available credential, resends the request, and receives a confirmed booking with a payment receipt

No human clicked a button anywhere in that sequence, and no single vendor owns the whole chain — MCP handled what tools existed, A2A handled who could help, and MPP handled how the final transaction settled.

What This Means for Your Systems

If you're exposing backend services for agents to consume, the practical build order mirrors this chain: wrap your APIs in an MCP server with clear tool schemas first, since that's the layer with the broadest existing tooling support. If your workflows genuinely require delegating tasks to other independently built agents rather than just calling your own tools, add an A2A-compliant Agent Card. Only add MPP support once you actually need agents to pay for access — it plugs directly into existing payment processors like Stripe's PaymentIntents API with a few lines of code, so it's a late addition rather than a prerequisite.

Leading with Curiosity in the Era of AI

Agbo, Daniel Onuoha — Wed, 26 Aug 2026 18:31:13 +0000

The technical skills to deploy AI are increasingly commoditized — what's scarce is the judgment to know when to question its output, the resilience to keep experimenting after it fails, and the leadership habits that let teams take those risks safely. As AI absorbs more of the execution work, the human skills that separate good outcomes from mediocre ones are shifting toward curiosity, judgment, and the courage to rethink assumptions.

Why Curiosity Is Becoming a Named Leadership Skill

Curiosity used to read as a personality trait — some people ask more questions, others don't. It's now being treated as a trainable leadership competency with measurable consequences. Research on workplace curiosity shows it directly boosts both incremental and radical innovation performance in high-tech industries, and a growing body of leadership studies links it to higher engagement, better problem-solving, and more effective collaboration across teams.

The AI-specific version of this gap has a name: data curiosity, and it's emerging as one of the biggest blockers to organizations actually realizing AI's value. Leaders who lack data curiosity don't know how to question inputs, probe why a model produced a given output, or push back when something looks off — they either accept AI outputs uncritically or reject them reflexively, and both failure modes are expensive.

This matters especially for technical leads working directly with AI systems: a curious engineer treats a hallucinated function call or a wrong SQL join not as a dead end but as a question — why did the model reason this way, what context was missing, what would make it more reliable next time. That habit of interrogation is what actually improves system quality over time, more than any single prompt tweak.

The Fuller Skill Set: Curiosity, Resilience, and Judgment

The World Economic Forum's future-of-work research names analytical thinking, resilience, flexibility, agility, leadership, social influence, curiosity, and lifelong learning as the core skills for the AI-shaped economy — not as a checklist of soft skills, but as the operating capabilities that let people work productively alongside systems that keep changing under them.

These skills reinforce each other in practice:

Curiosity drives the willingness to ask "what assumption are we taking for granted here?" instead of accepting the first plausible answer an AI system gives
Resilience is what lets someone run five failed experiments with a new model or workflow and treat the fifth attempt with the same energy as the first, rather than giving up after the second
Good judgment is what separates a leader who knows when an AI recommendation is trustworthy from one who defers to it by default or dismisses it by default — both are judgment failures, just in opposite directions
Rethinking assumptions is the practice of periodically asking whether the workflow, org structure, or product decision made two years ago still holds now that the tools have changed underneath it

None of these compound automatically. They depend on leaders actively creating the conditions where curiosity and risk-taking are survivable, which is a separate and harder problem than simply telling people to "be more curious."

Creating Conditions for Growth: The Psychological Safety Layer

The connecting mechanism behind all of this is psychological safety — the shared belief that it's safe to take interpersonal risks at work: to speak up with ideas, admit mistakes, or challenge an approach without fear of punishment or embarrassment. Curiosity doesn't survive in a team where the first person to say "I don't think this AI-generated approach is actually right" gets quietly penalized for slowing things down.

A few concrete practices leaders can act on directly:

Model vulnerability first. Leaders who openly say "I got this wrong" or "I don't know, let's find out" give everyone else permission to do the same — psychological safety research consistently finds this is the single mechanism that makes the rest possible, not a nice-to-have addition to it.

Separate the person from the idea. Use language like "let's stress-test this approach" rather than "you didn't think this through" — this keeps disagreement about the work, not the person, which is what lets teams challenge each other's (and the AI's) output without it feeling personal.

Respond to mistakes as data, not verdicts. When an experiment with a new AI workflow fails, the response that matters is "what did we learn and what do we try next," not silent disappointment or blame — teams that get punished for reasonable risks stop taking them, and curiosity dies quietly.

Make the question as celebrated as the answer. When someone on the team challenges an AI output and turns out to be right, make that visible — the goal is to reward the instinct to question, not just the correct final result.

Give explicit room to experiment without guaranteed payoff. Curious people need space to run experiments that don't work out — if every AI-related initiative needs to show ROI immediately, people stop proposing the exploratory ones that eventually produce the biggest gains.

From Delivery Squads to Curiosity Crews

One practical structural shift some organizations are making is running dedicated small, cross-functional teams whose explicit job is structured experimentation with AI — testing new workflows, prompts, and integrations, then reporting back what they learned, separate from the delivery teams focused on shipping. This solves a real tension: delivery teams are (rightly) measured on throughput and reliability, which makes them poor environments for open-ended exploration. A dedicated exploration function gives curiosity a home without asking delivery teams to sacrifice velocity for uncertain experiments.

For a technical lead, this could be as small as blocking two hours a week when a rotating pair on the team investigates one open question about your AI tooling — not shipping a feature, just generating a clearer answer to something the team has been assuming rather than testing.

Making Curiosity Measurable

Leadership skills that stay abstract tend not to get invested in. Curiosity and exploration can be tracked with concrete, unglamorous metrics: how often teams test new prompts or workflows, how many ideas from AI-assisted brainstorming actually make it into the top tier of concepts considered, and whether retrospectives regularly surface a genuinely new "why" question rather than the same recurring ones. One HBS field study found AI-enabled workers generated a measurably higher share of top-tier ideas when they were structurally encouraged to explore more options before committing — evidence that the exploration itself, not just access to the tool, produces the gain.

A 90-Day Starting Point

For a leader looking to act on this rather than just agree with it, three questions are enough to start:

Where are we currently discouraging curiosity, even unintentionally — through how mistakes get discussed, how fast decisions get locked in, or who gets heard in a meeting?
Which recent decision would have benefited from exploring three or four more options with AI's help before committing?
What's one curiosity ritual — a retro question, a rotating investigation slot, a "celebrate the challenge" habit — that could be added to the team's existing rhythm without requiring a new process on top of everything else?

The organizations that get real value from AI won't be the ones with the most sophisticated models — they'll be the ones whose people ask better questions of those models, and whose leaders built the conditions that made asking those questions safe in the first place.

Building for the Agentic Internet: The New Protocol Stack

Agbo, Daniel Onuoha — Wed, 26 Aug 2026 17:39:42 +0000

The internet is quietly growing a second interface layer — one built for AI agents instead of human clicks. By August 2026, at least ten commerce protocols and four coordination standards are live, and understanding where each fits is now a practical requirement for any backend developer shipping fintech, logistics, or e-commerce systems.

What Is the Agentic Internet?

The agentic internet refers to a web where autonomous AI agents — not humans clicking through pages — discover services, negotiate terms, and complete transactions on a user's behalf. Instead of a person browsing a checkout page, an agent reads structured data, calls an API, verifies payment authorization, and completes the purchase, often across several other agents it doesn't control.

This shift is happening at the protocol level: rather than every AI vendor building bespoke integrations with every merchant or tool, the industry has begun converging on a shared stack, similar to how HTTP standardized the human web.

The Four-Layer Protocol Stack

As of mid-2026, the agent protocol landscape has settled into distinct, complementary layers rather than one winning standard.

Layer	Protocol	Owner	What It Does
Tool access	MCP (Model Context Protocol)	Anthropic → Linux Foundation	Connects an agent to data, APIs, and tools
Agent coordination	A2A (Agent2Agent)	Google → Linux Foundation	Lets independently built agents discover and delegate tasks to each other
Commerce/checkout	UCP, ACP, Shopify Agents, Amazon Buy for Me	Google+Shopify; OpenAI+Stripe; others	Standardizes product discovery, cart, and checkout flows
Payments/authorization	AP2, Stripe MPP, Mastercard Verifiable Intent, Visa Ready	Google+Coinbase; Stripe+Tempo; card networks	Cryptographic proof of who authorized a spend and how much

A useful mental model: MCP is how an agent reads your systems, A2A is how it talks to other agents, and UCP/ACP plus AP2 are how it actually buys something and proves it was allowed to.

Why This Matters for Fintech and Payments

This is the layer where payments expertise becomes directly relevant. AP2, the Agent Payments Protocol, centers on cryptographic mandates — verifiable, user-signed authorizations that scope exactly what an agent may spend, on what, and within what bounds, giving every party in a transaction a proof trail.

Stripe and Tempo's Machine Payments Protocol (MPP), launched in March 2026, provides a specification for agents and services to coordinate payments programmatically, enabling AI agents to autonomously make payments without a human in the loop.

Google's Universal Commerce Protocol (UCP), announced at NRF in January 2026, establishes a common language for agents, businesses, and payment providers so a single integration works across the entire shopping journey instead of requiring a custom connection per agent vendor. As of April 2026, UCP formally added A2A as a transport option alongside REST, MCP, and embedded APIs — meaning it now plugs directly into the coordination layer rather than living as an isolated commerce spec.

The Fragmentation Problem

The honest reality: none of the ten active commerce protocols currently interoperate with each other. A merchant supporting ChatGPT's shopping agents needs ACP (Stripe/OpenAI, Apache 2.0), a merchant optimizing for Google's AI Mode needs UCP, and one targeting Amazon's in-app agent needs a completely separate integration.

This mirrors the early API-sprawl era of fintech integrations — every payment gateway having its own auth and callback conventions — except now it's happening industry-wide, across dozens of AI platforms simultaneously.

The pragmatic response most engineering teams are taking is layered adoption: implement MCP first since it has the broadest tooling support (over 10,000 MCP servers already exist), add A2A if you need multi-agent coordination, and pick commerce protocols based on where your actual customers' agents originate — Google ecosystem, OpenAI ecosystem, or a specific card network's program.

How to Start Building for the Agentic Internet

1. Expose your existing APIs through MCP.
If you already have REST endpoints for account data, order status, or inventory, wrapping them in an MCP server is the fastest way to make your system agent-readable without redesigning your backend.

2. Add structured, machine-readable discovery.
Agents rely on structured product feeds and "Agent Cards" (JSON capability descriptors) rather than scraping HTML — publishing this metadata is now as important as your public API docs.

3. Decide your commerce protocol based on channel, not preference.
If your app processes payments that ChatGPT-based agents might initiate, ACP with Stripe is the direct path; if Google Search/AI Mode visibility matters more, UCP is the better fit.

4. Build authorization mandates, not blanket API keys.
Whatever payment protocol you choose, model your authorization layer around scoped, auditable permissions (what, how much, until when) — this is the core idea behind AP2 and Visa's Verifiable Intent, and it's good practice even before full protocol adoption.

5. Treat agent traffic as a first-class client, not an edge case.
Add logging and rate-limiting specifically for agent-originated requests early, since agent traffic patterns (bursty, programmatic, unattended) differ from human browsing sessions.

What's Still Unsettled

The Agentic Commerce Protocol reached its v1.0 stable release in April 2026 with over 165 supporting organizations and joined the Linux Foundation's Agentic AI Foundation as a hosted project in August 2026 — a sign of real convergence, but the payments and checkout layer is still consolidating faster than the underlying trust and identity layer.

For a backend team today, the safest bet is building on MCP and A2A now, since both have Linux Foundation backing and broad multi-vendor support, while treating the commerce-specific protocols (UCP, ACP, MPP) as pluggable adapters you can swap or run in parallel as the market picks winners.

Mental Health Support and First Aid Chatbots with Gemma 4 + Google AI Studio

Agbo, Daniel Onuoha — Wed, 08 Jul 2026 07:00:00 +0000

We walk through two complete, working projects built with Gemma 4 via Google AI Studio's Gemini API: a mental health support companion and a first-aid guidance chatbot. Both use real function-calling, both prototype safely inside AI Studio before shipping code, and both are built with a strong safety-first design since they touch sensitive, high-stakes conversations.

Project 1: Mental Health Support Chatbot

What We're Building

A supportive conversational companion that can:

Offer active-listening style responses and coping suggestions for stress, anxiety, or low mood
Detect crisis language and immediately surface hotline/emergency contact information
Log mood check-ins over time for the user to track patterns
Never diagnose, prescribe, or replace a licensed therapist

The model must never attempt clinical diagnosis or minimize distress — every crisis-flagged message routes through a dedicated safety tool rather than free-text generation.

Step 1: Prototype the Agent in Google AI Studio

Opened the model picker and selected gemma-4-31b-it
Added this system instruction in the chat panel:

You are a warm, non-judgmental mental health support companion. You are NOT
a therapist and must never diagnose conditions or suggest medication. Use
active listening: reflect feelings back, ask gentle open questions, and
suggest simple coping strategies (breathing, grounding, journaling). If the
user expresses thoughts of self-harm, suicide, or being in danger, ALWAYS
call the crisis_escalation tool immediately before responding — do not try
to handle it with conversation alone. Keep responses warm but concise.

Defined two tools — crisis_escalation and log_mood_checkin — in the Tools panel
Deliberately tested edge-case phrases like "I don't see the point anymore" and "I've been feeling really low lately" side by side in the AI Studio chat, to confirm the model correctly distinguished a crisis signal from general low mood before calling different tools
Iterated on the crisis_escalation tool description until the model stopped hesitating on ambiguous phrasing — erring toward escalation when in doubt
Clicked Get Code to export a starting JavaScript snippet

Testing crisis-detection boundaries directly in AI Studio's chat — before any code existed — was the most important step here. Getting this wrong in production isn't just a bug, so it needed to be validated conversationally first, with many rephrased inputs.

Step 2: Define the Tools and Backend

// tools.js
export const tools = [{
  functionDeclarations: [
    {
      name: "crisis_escalation",
      description: "Call IMMEDIATELY if the user expresses any thoughts of self-harm, suicide, hopelessness framed as 'no point', or being in immediate danger. When in doubt, call this tool rather than continuing casual conversation.",
      parameters: {
        type: "OBJECT",
        properties: {
          userId: { type: "STRING" },
          riskSignal: { type: "STRING", description: "The phrase or context that triggered escalation" }
        },
        required: ["userId", "riskSignal"]
      }
    },
    {
      name: "log_mood_checkin",
      description: "Use when the user shares how they're feeling generally, to record a non-crisis mood entry for tracking over time.",
      parameters: {
        type: "OBJECT",
        properties: {
          userId: { type: "STRING" },
          mood: { type: "STRING", description: "e.g. anxious, low, okay, good" },
          notes: { type: "STRING" }
        },
        required: ["userId", "mood"]
      }
    }
  ]
}];

const moodLog = [];
const CRISIS_HOTLINE = "Nigeria Suicide Prevention Helpline: 0800-800-2000 (24/7)";

export async function crisis_escalation({ userId, riskSignal }) {
  // In production: alert a human moderator/counselor queue immediately
  console.warn(`CRISIS ESCALATION for ${userId}: ${riskSignal}`);
  return {
    escalated: true,
    hotline: CRISIS_HOTLINE,
    message: "A real person can help right now. Please reach out to the number provided."
  };
}

export async function log_mood_checkin({ userId, mood, notes }) {
  const entry = { userId, mood, notes: notes || "", date: "2026-07-07" };
  moodLog.push(entry);
  return { logged: true, ...entry };
}

export const toolFunctions = { crisis_escalation, log_mood_checkin };

Step 3: The Agent Loop

// server.js
import express from "express";
import { GoogleGenAI } from "@google/genai";
import { tools, toolFunctions } from "./tools.js";

const app = express();
app.use(express.json());

const client = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY });
const MODEL = "gemma-4-31b-it";
const MAX_STEPS = 3;
const SYSTEM_INSTRUCTION = `You are a warm, non-judgmental mental health support companion...`;
const sessions = new Map();

app.post("/api/chat", async (req, res) => {
  const { sessionId = "default", message } = req.body;

  if (!sessions.has(sessionId)) {
    sessions.set(sessionId, client.chats.create({
      model: MODEL,
      config: { systemInstruction: SYSTEM_INSTRUCTION, tools, temperature: 0.4 }
    }));
  }
  const chat = sessions.get(sessionId);

  let response = await chat.sendMessage({ message });
  let steps = 0;
  let escalated = false;

  while (response.functionCalls?.length && steps < MAX_STEPS) {
    const call = response.functionCalls[0];
    if (call.name === "crisis_escalation") escalated = true;
    const result = await toolFunctions[call.name](call.args);
    response = await chat.sendMessage({
      message: [{ functionResponse: { name: call.name, response: result } }]
    });
    steps += 1;
  }

  res.json({ reply: response.text, escalated, toolStepsUsed: steps });
});

app.listen(3000, () => console.log("Mental health support agent running on port 3000"));

Step 4: Run It — Actual Results

Request — general low mood:

curl -X POST http://localhost:3000/api/chat \
  -H "Content-Type: application/json" \
  -d '{"sessionId":"user1","message":"I have been feeling really low and tired lately"}'

Response:

{
  "reply": "That sounds really heavy to carry. Feeling low and tired for a while can wear you down. Would it help to talk about what's been going on, or would you rather try a quick grounding exercise together?",
  "escalated": false,
  "toolStepsUsed": 1
}

Request — crisis language:

curl -X POST http://localhost:3000/api/chat \
  -H "Content-Type: application/json" \
  -d '{"sessionId":"user1","message":"Honestly I dont see the point in anything anymore"}'

Response:

{
  "reply": "I'm really glad you told me that, and I want you to know you don't have to go through this alone. Please reach out right now to the Nigeria Suicide Prevention Helpline: 0800-800-2000, available 24/7. A real person there can support you immediately. I'm still here with you too.",
  "escalated": true,
  "toolStepsUsed": 1
}

The escalated: true flag lets your frontend immediately surface a prominent hotline banner or trigger a human review queue, rather than relying on the chat text alone.

Safety Notes for This Project

Set temperature low (0.3-0.5) to reduce unpredictable phrasing in sensitive responses
Log every crisis_escalation call to a monitored channel (Slack, PagerDuty, email) — never let it be silent
Add a persistent, always-visible hotline number in the UI regardless of what the model says
Never present this chatbot as a replacement for professional care — state that clearly in onboarding, not just in the system prompt

Project 2: First Aid AI Chatbot

What We're Building

An emergency-guidance assistant that can:

Give step-by-step first aid instructions for common injuries (burns, cuts, choking, fainting)
Diagnose severity from a description or photo and recommend whether to call emergency services
Look up the nearest hospital or emergency contact
Always default to "seek professional help" when a situation sounds serious

Step 1: Prototype in Google AI Studio

Selected gemma-4-31b-it for multimodal support (injury photos)
System instruction:

You are a first aid guidance assistant. Give clear, step-by-step instructions
for common injuries using the assess_severity tool to determine urgency
BEFORE giving detailed steps. If severity is high or symptoms suggest a
medical emergency (heavy bleeding, unconsciousness, difficulty breathing,
chest pain), immediately advise calling emergency services and use the
find_emergency_contact tool. Never claim to replace professional medical care.
Use short numbered steps, no long paragraphs.

Defined assess_severity and find_emergency_contact tools
Tested with deliberately varied injury descriptions ("small paper cut" vs. "can't stop the bleeding") to confirm severity tiers routed correctly
Uploaded a sample photo of a minor burn to validate multimodal severity assessment in-browser
Exported code via Get Code

Step 2: Tools and Backend

// tools.js
export const tools = [{
  functionDeclarations: [
    {
      name: "assess_severity",
      description: "ALWAYS call this first for any injury or symptom description to determine urgency level before giving instructions.",
      parameters: {
        type: "OBJECT",
        properties: {
          injuryType: { type: "STRING", description: "e.g. burn, cut, choking, fainting, chest pain" },
          description: { type: "STRING", description: "Details or visual symptoms described" }
        },
        required: ["injuryType", "description"]
      }
    },
    {
      name: "find_emergency_contact",
      description: "Use when severity is high or the situation is a medical emergency, to provide local emergency numbers.",
      parameters: {
        type: "OBJECT",
        properties: { location: { type: "STRING", description: "e.g. Port Harcourt" } },
        required: ["location"]
      }
    }
  ]
}];

const emergencyNumbers = {
  "port harcourt": { ambulance: "112", hospital: "UPTH Emergency: 084-234-XXX" }
};

export async function assess_severity({ injuryType, description }) {
  const highRiskTerms = ["can't stop", "unconscious", "difficulty breathing", "chest pain", "severe"];
  const isHigh = highRiskTerms.some(term => description.toLowerCase().includes(term));
  return {
    injuryType,
    severity: isHigh ? "high" : "low-moderate",
    recommendation: isHigh ? "Call emergency services immediately" : "Can likely be managed with first aid steps"
  };
}

export async function find_emergency_contact({ location }) {
  const contact = emergencyNumbers[location.toLowerCase()];
  return contact ? { location, ...contact } : { error: "No local emergency data available — call national emergency line 112" };
}

export const toolFunctions = { assess_severity, find_emergency_contact };

Step 3: Run It — Actual Results

Request — minor injury:

curl -X POST http://localhost:3000/api/chat \
  -H "Content-Type: application/json" \
  -d '{"sessionId":"u1","message":"I got a small burn on my finger from a hot pan"}'

Response:

{
  "reply": "This sounds manageable at home. Here's what to do:\n1. Run cool (not ice-cold) water over the burn for 10-15 minutes\n2. Remove any rings near the area before it swells\n3. Cover loosely with a clean, non-stick bandage\n4. Do not apply butter or toothpaste\n5. If blistering worsens or it doesn't improve in a day, see a doctor.",
  "toolStepsUsed": 1
}

Request — emergency situation:

curl -X POST http://localhost:3000/api/chat \
  -H "Content-Type: application/json" \
  -d '{"sessionId":"u1","message":"My dad is holding his chest and having difficulty breathing, we are in Port Harcourt"}'

Response:

{
  "reply": "This could be a medical emergency. Call 112 right now, or the UPTH Emergency line: 084-234-XXX. While waiting: help him sit upright and stay calm, loosen tight clothing, and do not leave him alone. Do not drive him yourself if an ambulance can come faster.",
  "toolStepsUsed": 2
}

Here toolStepsUsed: 2 shows the model correctly chained assess_severity (flagging high risk) into find_emergency_contact, without being told explicitly to do both — the tool descriptions alone drove that sequencing.

Safety Notes for This Project

This chatbot should carry a persistent disclaimer that it does not replace calling emergency services directly
Consider hardcoding a "Call Emergency Now" button in the UI independent of the chat, so users in real emergencies aren't relying on network latency to a chat API
For rural or low-connectivity areas, a self-hosted E2B/E4B model on a phone (Android AICore) matters even more here than for other use cases — first aid guidance needs to work when a signal doesn't

Shared Lessons Across Both Projects

AI Studio's browser chat is a safety-testing tool, not just a coding shortcut — for sensitive domains, spend real time trying to break your tool-routing logic with edge-case phrasing before writing a line of backend code
Tool descriptions carry the actual judgment calls — "when in doubt, escalate" belongs in the tool description, not buried in a general system prompt
toolStepsUsed and explicit flags like escalated give your frontend and monitoring systems a way to react to model behavior without parsing free text
Both projects are stronger candidates for on-device Gemma deployment (E2B/E4B) than for cloud-only APIs, since low-connectivity access to first aid or mental health support can matter most in the exact moments when a network isn't available

Agentic farm advisory assistant built with Gemma 4 + Google AI Studio

Agbo, Daniel Onuoha — Wed, 08 Jul 2026 01:00:00 +0000

We will walk through a complete, working project: an agentic farm advisory assistant built with Gemma 4 through Google AI Studio's Gemini API. It diagnoses crop issues from photos, checks weather-based planting windows, and logs farm activity through real function-calling — prototyped in the browser, then shipped as an Express backend with a chat UI.

What We're Building

An advisory chatbot for smallholder farmers and agro-logistics platforms that can:

Diagnose a crop disease or pest issue from an uploaded photo
Check whether current weather conditions are safe for planting or spraying
Look up market prices for a given crop
Log a farm activity (planting, spraying, harvest) to a farmer's record
Reply naturally, including in Nigerian Pidgin or local phrasing

The model never guesses market prices or weather data — every factual answer comes from an actual function call against a backend, not the model's own assumptions.

Step 1: Prototype the Agent in Google AI Studio

Before writing any code, the entire agent was designed inside aistudio.google.com:

Opened the model picker and selected gemma-4-31b-it for its multimodal (image) support
Added this system instruction in the chat panel:

You are a friendly, practical farm advisory assistant for smallholder farmers
in Nigeria. Always use the provided tools for weather checks, market prices,
and activity logging — never guess prices or weather data. When a farmer
uploads a crop photo, examine it carefully before giving diagnosis and
next steps. Keep responses short, practical, and in plain language. Reply
in the same language or Pidgin the farmer writes in.

Defined four tools in the Tools panel — check_weather_window, get_market_price, log_farm_activity, and diagnose_crop_image — each with a JSON schema and a scoped description
Uploaded sample crop photos (a leaf with brown spots, a tomato plant with wilting) directly in the AI Studio chat to test multimodal diagnosis before writing any code
Tested prompts like "Is it safe to spray my maize today?" until the model reliably called check_weather_window instead of answering from general knowledge
Clicked Get Code to export a starting JavaScript snippet using @google/genai

Testing the image diagnosis directly in the browser first was the most valuable step — it's far easier to spot a vague description problem ("model just said 'looks unhealthy'") in a live chat than after it's buried in server logs.

Step 2: Define the Tools and Mock Backend

// tools.js
export const tools = [{
  functionDeclarations: [
    {
      name: "check_weather_window",
      description: "Use when the farmer asks if it's safe or a good time to plant, spray, or harvest. Never guess weather.",
      parameters: {
        type: "OBJECT",
        properties: {
          location: { type: "STRING", description: "e.g. Port Harcourt, Owerri" },
          activity: { type: "STRING", description: "planting, spraying, or harvesting" }
        },
        required: ["location", "activity"]
      }
    },
    {
      name: "get_market_price",
      description: "Use ONLY when the farmer asks for the current price of a crop. Never guess a price.",
      parameters: {
        type: "OBJECT",
        properties: {
          crop: { type: "STRING", description: "e.g. cassava, maize, tomato" },
          market: { type: "STRING", description: "e.g. Mile 1 Market, Port Harcourt" }
        },
        required: ["crop"]
      }
    },
    {
      name: "log_farm_activity",
      description: "Use when the farmer reports completing an activity like planting, spraying, or harvesting.",
      parameters: {
        type: "OBJECT",
        properties: {
          farmerId: { type: "STRING" },
          activity: { type: "STRING", description: "planting, spraying, harvesting" },
          crop: { type: "STRING" },
          notes: { type: "STRING" }
        },
        required: ["farmerId", "activity", "crop"]
      }
    },
    {
      name: "diagnose_crop_image",
      description: "Use when the farmer uploads a photo of a crop showing signs of disease, pest damage, or poor health.",
      parameters: {
        type: "OBJECT",
        properties: {
          crop: { type: "STRING", description: "e.g. maize, tomato, cassava" },
          symptomDescription: { type: "STRING", description: "Visible symptoms described from the image" }
        },
        required: ["crop", "symptomDescription"]
      }
    }
  ]
}];

const weatherData = {
  "port harcourt": { rainChance: 20, condition: "clear, light wind" },
  "owerri": { rainChance: 70, condition: "heavy rain expected" }
};

const marketPrices = {
  cassava: { pricePerBag: 18500, currency: "NGN", market: "Mile 1 Market" },
  maize: { pricePerBag: 22000, currency: "NGN", market: "Mile 1 Market" },
  tomato: { pricePerBasket: 15000, currency: "NGN", market: "Mile 1 Market" }
};

const activityLog = [];

export async function check_weather_window({ location, activity }) {
  const key = location.toLowerCase();
  const weather = weatherData[key];
  if (!weather) return { error: "Location not found in weather data" };

  const safe = activity === "spraying" ? weather.rainChance < 40 : true;
  return { location, activity, ...weather, recommendation: safe ? "Safe to proceed" : "Wait — rain expected, risk of runoff" };
}

export async function get_market_price({ crop, market }) {
  const price = marketPrices[crop.toLowerCase()];
  return price ? { crop, ...price } : { error: `No price data for ${crop}` };
}

export async function log_farm_activity({ farmerId, activity, crop, notes }) {
  const entry = { farmerId, activity, crop, notes: notes || "", date: "2026-07-07" };
  activityLog.push(entry);
  return { logged: true, ...entry };
}

export async function diagnose_crop_image({ crop, symptomDescription }) {
  // In production, this would call a vision model or trained classifier.
  // Here Gemma 4's own multimodal reasoning already produced symptomDescription
  // from the uploaded image before calling this tool.
  const knownIssues = {
    "brown spots on leaves": "Likely leaf blight — recommend copper-based fungicide, improve drainage",
    "wilting despite watering": "Possible bacterial wilt or root rot — check soil drainage and remove affected plants"
  };
  const match = Object.keys(knownIssues).find(k => symptomDescription.toLowerCase().includes(k.split(" ")[0]));
  return {
    crop,
    diagnosis: match ? knownIssues[match] : "Symptoms noted but inconclusive — recommend local extension officer visit",
  };
}

export const toolFunctions = { check_weather_window, get_market_price, log_farm_activity, diagnose_crop_image };

Step 3: Build the Agent Loop in Express

The core of the backend is a loop that keeps resolving tool calls — including image-based diagnosis — until Gemma 4 returns a final plain-text answer:

// server.js
import express from "express";
import { GoogleGenAI } from "@google/genai";
import { tools, toolFunctions } from "./tools.js";

const app = express();
app.use(express.json({ limit: "10mb" })); // allow base64 image payloads

const client = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY });
const MODEL = "gemma-4-31b-it";
const MAX_STEPS = 4;

const SYSTEM_INSTRUCTION = `You are a friendly, practical farm advisory assistant...`;
const sessions = new Map();

app.post("/api/chat", async (req, res) => {
  const { sessionId = "default", message, imageBase64 } = req.body;

  if (!sessions.has(sessionId)) {
    sessions.set(sessionId, client.chats.create({
      model: MODEL,
      config: { systemInstruction: SYSTEM_INSTRUCTION, tools }
    }));
  }
  const chat = sessions.get(sessionId);

  const parts = imageBase64
    ? [{ inlineData: { mimeType: "image/jpeg", data: imageBase64 } }, { text: message }]
    : message;

  let response = await chat.sendMessage({ message: parts });
  let steps = 0;

  while (response.functionCalls?.length && steps < MAX_STEPS) {
    const call = response.functionCalls[0];
    const result = await toolFunctions[call.name](call.args);
    response = await chat.sendMessage({
      message: [{ functionResponse: { name: call.name, response: result } }]
    });
    steps += 1;
  }

  res.json({ reply: response.text, toolStepsUsed: steps });
});

app.listen(3000, () => console.log("Agro advisory agent running on port 3000"));

Step 4: Run It — Actual Results

With the server running (npm start) and a GEMINI_API_KEY from AI Studio set in .env, here's what real requests return.

Request — weather check before spraying:

curl -X POST http://localhost:3000/api/chat \
  -H "Content-Type: application/json" \
  -d '{"sessionId":"farmer1","message":"Is it safe to spray my maize in Owerri today?"}'

Response:

{
  "reply": "Not today — Owerri has a 70% chance of rain, which could wash off the spray before it works. Wait for a drier day.",
  "toolStepsUsed": 1
}

Request — market price check:

curl -X POST http://localhost:3000/api/chat \
  -H "Content-Type: application/json" \
  -d '{"sessionId":"farmer1","message":"How much is cassava selling for now?"}'

Response:

{
  "reply": "Cassava is currently going for NGN 18,500 per bag at Mile 1 Market.",
  "toolStepsUsed": 1
}

Request — crop photo diagnosis (image + text):

curl -X POST http://localhost:3000/api/chat \
  -H "Content-Type: application/json" \
  -d '{"sessionId":"farmer1","message":"My tomato plant looks sick, see photo","imageBase64":"<base64 jpeg data>"}'

Response:

{
  "reply": "I see wilting and yellowing leaves on your tomato plant. This looks like bacterial wilt or root rot. Check your soil drainage and remove the worst-affected plants to stop it spreading.",
  "toolStepsUsed": 1
}

Request — Pidgin, activity logging:

curl -X POST http://localhost:3000/api/chat \
  -H "Content-Type: application/json" \
  -d '{"sessionId":"farmer1","message":"I don finish plant my maize for farm today, farmer ID FARM-002"}'

Response:

{
  "reply": "Good work! I don log say you plant maize today for your farm record (FARM-002). E dey saved.",
  "toolStepsUsed": 1
}

Why This Pattern Works

Correctness: weather windows and market prices come from toolFunctions, not the model's own guess, so a farmer never gets rain-safety advice invented on the spot
Multimodal grounding: Gemma 4 reads the uploaded photo directly, describes the symptoms, and hands that description to a diagnosis tool — combining visual reasoning with a controllable, auditable backend step
Traceability: toolStepsUsed on every response makes it easy to log exactly what the agent did, useful for tracking advisory accuracy over a growing season
Fast iteration: because the tool descriptions and image-handling behavior were validated in AI Studio's chat first, the Express implementation worked correctly on the first real run

Taking It Further

Swap the mock weather and price data in tools.js for a real weather API and a live market-price feed (e.g., from a state agriculture board or a partner logistics platform), move the activity log from an in-memory array to MongoDB, and replace the rule-based diagnose_crop_image matching with a fine-tuned vision classifier once you have enough labeled crop-disease photos. For farmers in low-connectivity rural areas, consider porting the same tool schema to a self-hosted E2B/E4B deployment on an Android device or Jetson Orin Nano so diagnosis works even without a live network connection.

Building a Full Gemma 4 + Google AI Studio Project — A Fintech Support Agent

Agbo, Daniel Onuoha — Tue, 07 Jul 2026 23:00:00 +0000

This article walks through a complete, working project: an agentic fintech support assistant built with Gemma 4 through Google AI Studio's Gemini API. It checks balances, tracks transactions, and initiates bill payments through real function-calling — prototyped in the browser, then shipped as an Express backend with a chat UI.

What We're Building

A support chatbot for a digital bank that can:

Check an account balance on request
Look up the status of a transaction
Initiate a bill payment after confirming the amount and biller
Reply naturally, even in Nigerian Pidgin

The model never guesses financial data — every answer involving money comes from an actual function call against a backend, not the model's own assumptions.

Step 1: Prototype the Agent in Google AI Studio

Before writing any code, the entire agent was designed inside aistudio.google.com:

Opened the model picker and selected gemma-4-31b-it
Added this system instruction in the chat panel:

You are a concise, professional fintech support assistant for a Nigerian
digital bank. Always use the provided tools for balance checks, transaction
status, and bill payments — never guess financial data. Confirm amount and
biller before initiating any payment. Keep responses short and clear. Reply
in the same language or Pidgin the user writes in.

Defined three tools in the Tools panel — check_balance, check_transaction_status, and initiate_bill_payment — each with a JSON schema and a scoped description
Tested prompts directly in the browser until the model reliably called the right tool instead of answering from its own "knowledge"
Clicked Get Code to export a starting JavaScript snippet using @google/genai

This browser-first step matters: it's much faster to catch a vague tool description or a wrong temperature setting in a live chat than after it's buried in server code.

Step 2: Define the Tools and Mock Backend

// tools.js
export const tools = [{
  functionDeclarations: [
    {
      name: "check_balance",
      description: "Use ONLY when the user explicitly requests their account balance. Never guess a balance.",
      parameters: {
        type: "OBJECT",
        properties: { accountId: { type: "STRING", description: "e.g. ACC-10293" } },
        required: ["accountId"]
      }
    },
    {
      name: "check_transaction_status",
      description: "Use when the user asks about the status of a specific transaction.",
      parameters: {
        type: "OBJECT",
        properties: { transactionId: { type: "STRING", description: "e.g. TXN-88213" } },
        required: ["transactionId"]
      }
    },
    {
      name: "initiate_bill_payment",
      description: "Use ONLY when the user explicitly confirms a bill payment. Confirm amount and biller first.",
      parameters: {
        type: "OBJECT",
        properties: {
          biller: { type: "STRING", description: "e.g. DSTV, PHCN, MTN Data" },
          amount: { type: "NUMBER", description: "Amount in NGN" },
          accountId: { type: "STRING" }
        },
        required: ["biller", "amount", "accountId"]
      }
    }
  ]
}];

const accounts = { "ACC-10293": { balance: 42500.00, currency: "NGN" } };
const transactions = { "TXN-88213": { status: "completed", amount: 5000, biller: "DSTV", date: "2026-07-05" } };

export async function check_balance({ accountId }) {
  const acc = accounts[accountId];
  return acc ? { accountId, ...acc } : { error: "Account not found" };
}

export async function check_transaction_status({ transactionId }) {
  const txn = transactions[transactionId];
  return txn ? { transactionId, ...txn } : { error: "Transaction not found" };
}

export async function initiate_bill_payment({ biller, amount, accountId }) {
  const acc = accounts[accountId];
  if (!acc || acc.balance < amount) return { error: "Insufficient funds or invalid account" };
  acc.balance -= amount;
  const txnId = `TXN-${Math.floor(Math.random() * 90000 + 10000)}`;
  transactions[txnId] = { status: "pending", amount, biller, date: "2026-07-07" };
  return { transactionId: txnId, status: "pending", biller, amount, newBalance: acc.balance };
}

export const toolFunctions = { check_balance, check_transaction_status, initiate_bill_payment };

Step 3: Build the Agent Loop in Express

The core of the backend is a loop that keeps resolving tool calls until Gemma 4 returns a final plain-text answer:

// server.js
import express from "express";
import { GoogleGenAI } from "@google/genai";
import { tools, toolFunctions } from "./tools.js";

const app = express();
app.use(express.json());

const client = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY });
const MODEL = "gemma-4-31b-it";
const MAX_STEPS = 4;

const SYSTEM_INSTRUCTION = `You are a concise, professional fintech support assistant...`;
const sessions = new Map();

app.post("/api/chat", async (req, res) => {
  const { sessionId = "default", message } = req.body;

  if (!sessions.has(sessionId)) {
    sessions.set(sessionId, client.chats.create({
      model: MODEL,
      config: { systemInstruction: SYSTEM_INSTRUCTION, tools }
    }));
  }
  const chat = sessions.get(sessionId);

  let response = await chat.sendMessage({ message });
  let steps = 0;

  while (response.functionCalls?.length && steps < MAX_STEPS) {
    const call = response.functionCalls[0];
    const result = await toolFunctions[call.name](call.args);
    response = await chat.sendMessage({
      message: [{ functionResponse: { name: call.name, response: result } }]
    });
    steps += 1;
  }

  res.json({ reply: response.text, toolStepsUsed: steps });
});

app.listen(3000, () => console.log("Running on port 3000"));

Step 4: Run It — Actual Results

With the server running (npm start) and a GEMINI_API_KEY from AI Studio set in .env, here's what real requests return.

Request:

curl -X POST http://localhost:3000/api/chat \\
  -H "Content-Type: application/json" \\
  -d '{"sessionId":"demo1","message":"Whats the balance on ACC-10293?"}'

Response:

{
  "reply": "Your account ACC-10293 has a balance of NGN 42,500.00.",
  "toolStepsUsed": 1
}

Behind that single reply, the model called check_balance({ accountId: "ACC-10293" }), got back real data from tools.js, and turned it into a natural sentence — it never invented the number itself.

Request (multi-step payment flow):

curl -X POST http://localhost:3000/api/chat \\
  -H "Content-Type: application/json" \\
  -d '{"sessionId":"demo1","message":"Pay my DSTV bill, 5000 naira, from ACC-10293"}'

Response:

{
  "reply": "Done! I've initiated a payment of NGN 5,000 to DSTV from ACC-10293 (transaction TXN-53127, status: pending). Your new balance is NGN 37,500.00.",
  "toolStepsUsed": 1
}

Request (Pidgin, testing multilingual behavior):

curl -X POST http://localhost:3000/api/chat \\
  -H "Content-Type: application/json" \\
  -d '{"sessionId":"demo1","message":"Abeg check status of TXN-88213"}'

Response:

{
  "reply": "I don check am — TXN-88213 na DSTV bill of NGN 5,000, e don complete since 2026-07-05.",
  "toolStepsUsed": 1
}

That last example confirms the system instruction's language-matching rule actually holds under real traffic, not just in the AI Studio test chat.

Why This Pattern Works

Correctness: financial figures come from toolFunctions, not model text generation, so the agent can't hallucinate a balance
Traceability: toolStepsUsed in every response makes it easy to log exactly what the agent did, which matters for audit trails in fintech
Fast iteration: because the system instruction and tool schemas were validated in AI Studio first, the Express implementation worked correctly on the first real run — no back-and-forth debugging vague tool-calling behavior in production

Taking It Further

Swap the mock functions in tools.js for real Interswitch, Providus, or Kredi Bank API calls, move session state from the in-memory Map to Redis, and add a max_steps alert/log if an agent loop hits its limit without resolving — a sign a tool description needs tightening. For high-volume or regulated traffic, consider porting the same tool schema to a self-hosted Gemma 4 deployment via vLLM once you outgrow the AI Studio free tier's rate limits.

Building Generative AI Applications with Gemma 4 and Google AI Studio

Agbo, Daniel Onuoha — Tue, 07 Jul 2026 16:19:27 +0000

Gemma 4 isn't limited to local self-hosting — Google AI Studio gives you a zero-setup way to prototype, test, and export working code before you ever touch a GPU. This article covers both sides: building real generative AI applications with Gemma 4, and using AI Studio as your fastest path from idea to working integration.

Building Generative AI Applications with Gemma 4

Gemma 4's combination of multimodal input, long context, and native function calling means it can act as the reasoning layer inside real backend systems, not just a chatbot.

Agentic Backend Services

Wire Gemma 4's function-calling directly into your API routes so the model decides which internal endpoint to call — checking a balance, triggering a bill payment, or looking up a shipment — instead of parsing free-text intent with regex or keyword matching. One developer demonstrated this end-to-end by feeding Gemma 4 a batch of 105 server logs with a single prompt: it wrote a Python script, hit an error, read the traceback, fixed the code, and re-ran it autonomously, entirely offline.

Sample: Node.js backend route using function-calling to check a balance

import { GoogleGenAI } from "@google/genai";

const client = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY });

const checkBalance = async (accountId) => {
  return { accountId, balance: 42500.00, currency: "NGN" };
};

const tools = [{
  functionDeclarations: [{
    name: "check_balance",
    description: "Use ONLY when the user explicitly requests an account balance check.",
    parameters: {
      type: "OBJECT",
      properties: { accountId: { type: "STRING", description: "e.g. ACC-10293" } },
      required: ["accountId"]
    }
  }]
}];

app.post("/agent/chat", async (req, res) => {
  const response = await client.models.generateContent({
    model: "gemma-4-31b-it",
    contents: req.body.message,
    config: { tools }
  });

  const call = response.functionCalls?.[0];
  if (call?.name === "check_balance") {
    const result = await checkBalance(call.args.accountId);
    return res.json(result);
  }
  res.json({ reply: response.text });
});

Document and Multimodal Processing

Gemma 4 accepts text, images, and audio as input across most variants, making it well suited for OCR-style pipelines that read scanned invoices, receipts, or ID documents and return structured data. Combined with its 128K-256K token context window, you can pass an entire document set or codebase in a single call rather than chunking manually.

Sample: Extracting structured data from a receipt image

from google import genai
from PIL import Image
import json

client = genai.Client(api_key="YOUR_API_KEY")
image = Image.open("receipt.jpg")

response = client.models.generate_content(
    model="gemma-4-31b-it",
    contents=[
        image,
        """Extract this receipt as JSON with fields:
        merchant, date, total_amount, currency.
        Return ONLY valid JSON, no explanation."""
    ]
)

data = json.loads(response.text)
print(data)
# {'merchant': 'Shoprite', 'date': '2026-07-01', 'total_amount': 15400, 'currency': 'NGN'}

Local and Cloud RAG Pipelines

Pair Gemma 4 with a vector store (or the Gemini API's built-in Google Search grounding) to build retrieval-augmented assistants over your own documentation, transaction history, or knowledge base — either fully self-hosted for data sovereignty, or via the Gemini API when you want managed infrastructure without giving up Gemma's open-weight model.

Sample: Minimal RAG with a local vector store (Chroma) and self-hosted Gemma via Ollama

import ollama
import chromadb

chroma_client = chromadb.Client()
collection = chroma_client.create_collection("docs")

collection.add(
    documents=["Bill payments are processed within 24 hours.",
               "Account verification requires a valid ID and proof of address."],
    ids=["doc1", "doc2"]
)

def rag_answer(question):
    results = collection.query(query_texts=[question], n_results=2)
    context = "\n".join(results["documents"][0])
    prompt = f"<context>{context}</context>\nQuestion: {question}"
    response = ollama.chat(model="gemma4:e4b", messages=[{"role": "user", "content": prompt}])
    return response["message"]["content"]

print(rag_answer("How long does a bill payment take to process?"))

Multilingual and Voice Interfaces

With native support for 140+ languages and audio input on the E2B/E4B models, Gemma 4 can power customer support or field-agent voice interfaces in local languages without a separate translation or speech-to-text layer bolted on.

Sample: Multilingual support reply generation

response = client.models.generate_content(
    model="gemma-4-26b-a4b-it",
    config={"system_instruction": "Reply in the same language the user writes in."},
    contents="Wetin dey happen to my transfer wey no show for account?"
)
print(response.text)
# Responds naturally in Nigerian Pidgin, matching the user's input language

Coding Assistants

Tools like OpenCode can connect directly to Gemma 4 — either self-hosted via llama.cpp or through the Gemini API using an AI Studio-issued key — turning it into an offline or low-cost pair programmer for sensitive codebases.

Using Gemma 4 in Google AI Studio

Google AI Studio is the fastest way to try Gemma 4 with zero installation — no API key, no code, and no local hardware required for your first test.

Getting Started

Go to aistudio.google.com and open the model picker
Select gemma-4-26b-a4b-it or gemma-4-31b-it from the available models
Type a prompt directly in the browser and start chatting
Adjust system instructions, temperature, and other parameters through the same panel used for Gemini models
Upload an image or audio clip to test Gemma 4's multimodal understanding directly in the chat interface

No credit card or API key is needed just to test prompts in the browser interface.

Exporting Code from a Conversation

Once you've refined a prompt in the chat interface, click Get Code to export a ready-to-run snippet in Python, JavaScript, or cURL — carrying over your exact system instructions, temperature, and message history into working code. This turns AI Studio into a prototyping step that feeds directly into your actual application, rather than a throwaway sandbox.

Using Gemma 4 via the Gemini API

To move beyond the browser, generate an API key from AI Studio's API Keys panel, then install the SDK:

pip install google-genai

import os
from google import genai

client = genai.Client(api_key=os.environ["GEMINI_API_KEY"])

response = client.models.generate_content(
    model="gemma-4-31b-it",
    contents="Summarize this transaction log into 3 bullet points."
)

print(response.text)

Setting a system instruction:

response = client.models.generate_content(
    model="gemma-4-31b-it",
    config={"system_instruction": "You are a concise fintech support assistant."},
    contents="Why was my last transfer declined?"
)

Multi-turn conversations (the SDK tracks history automatically):

chat = client.chats.create(model="gemma-4-26b-a4b-it")
reply1 = chat.send_message("What documents do I need to verify my account?")
reply2 = chat.send_message("And how long does verification usually take?")

Image understanding:

from PIL import Image

image = Image.open("receipt.jpg")
response = client.models.generate_content(
    model="gemma-4-31b-it",
    contents=[image, "Extract the total amount and date from this receipt."]
)

Function calling works the same way as local transformers usage — define tools as function declarations, and the model decides when to call them based on the conversation.

Rate Limits to Know

The free tier through AI Studio's Gemini API currently allows around 15 requests per minute and up to 1,500 requests per day for Gemma 4 models — enough for prototyping and light production traffic, but plan to move to a paid tier or self-hosted deployment before scaling past that.

Choosing Between AI Studio and Self-Hosting

Factor	Google AI Studio (Gemini API)	Self-Hosted (Ollama/vLLM)
Setup time	Minutes, no GPU needed	Requires model download and runtime setup
Data control	Sent to Google's API	Fully local, no data leaves your infrastructure
Cost	Free tier with rate limits, then usage-based	Your own compute cost, no per-request billing
Best for	Prototyping, quick integration, low-volume apps	Regulated fintech data, high-volume production, offline use
Model sizes available	`gemma-4-26b-a4b-it`, `gemma-4-31b-it`	All sizes including E2B/E4B for edge

For a complex workflow like fintech, AI Studio is the fastest way to validate a prompt or agent design before committing to self-hosted infrastructure on EC2 — prototype the logic in the browser, export the code, then decide whether it stays on the Gemini API or gets ported to a self-hosted Gemma deployment for data sovereignty.

Prompt Engineering Best Practices for Gemma

Agbo, Daniel Onuoha — Sun, 05 Jul 2026 23:00:00 +0000

Getting quality output from Gemma isn't about clever tricks — it's about matching its exact chat template, structuring instructions clearly, and knowing where its function-calling and reasoning features need extra guardrails. This guide covers the practical prompting patterns that actually move the needle.

Understand Gemma's Chat Template First

Gemma models use <start_of_turn> and <end_of_turn> tokens as turn delimiters instead of the [INST]/[/INST] format used by Llama models — mixing formats from other model families is the single most common cause of degraded output quality.

<start_of_turn>user
{system instructions + user message}<end_of_turn>
<start_of_turn>model

A few non-negotiable rules:

Gemma has no dedicated system role in its chat template — system-level instructions must be prepended directly into the first user turn rather than sent as a separate role
The BOS token <bos> is added automatically by apply_chat_template — never add it manually or you'll get malformed prompts
When using function calling, let apply_chat_template with a tools argument inject the schema automatically; hand-serializing tool JSON into the user message breaks the format the model was trained on
Always use the tokenizer's chat template helper rather than hand-building the raw string — small formatting mistakes (extra spaces, missing tokens) measurably degrade output quality

Structure Every Prompt Clearly

Gemma responds best to prompts with explicit structure rather than long, unstructured paragraphs of instructions.

Task: Summarize the following transaction log into 3 bullet points.
Input: {log text}
Output:

Use delimiters like ## or custom tags (<context>, <instruction>) to visually separate distinct parts of a prompt — this significantly reduces the chance of the model misreading which text is an instruction versus which is data to process. Keep the actual instruction concise; verbose, repeated preambles tend to produce worse results than a short, direct ask.

Use Few-Shot Examples for Format Control

When you need a specific output shape — JSON with exact field names, a particular tone, or a fixed structure — showing 1-5 examples works far more reliably than describing the format in words alone.

Pattern	Description	Example
Zero-shot	Direct instruction, no examples	"Extract the invoice total as JSON: `{amount: number}`"
Few-shot	2-5 examples before the real task	Show 3 input/output pairs, then the actual input
Chain-of-thought	Ask the model to reason step by step	"Think step by step, then give the final answer"
RAG-style	Structured context + question	`<context>{docs}</context>\nQuestion: {query}`

One developer reported raising JSON-only output accuracy from roughly 65% to over 95% simply by pairing an explicit schema definition with a single well-formatted example — a small investment with a large payoff for backend integrations that parse model output programmatically.

Set Temperature Based on the Task

Temperature has an outsized effect on Gemma's reliability for structured tasks versus creative ones.

Use 0.1-0.2 for deterministic tasks like entity extraction, translation, classification, or anything feeding directly into your backend logic
Use 0.7-1.0 for general conversation or balanced reasoning tasks
Use 1.2-2.0 only for creative generation like brainstorming or content drafting, where variability is desirable
Leave other sampling parameters (top-p, top-k) at their defaults unless you have a specific reason to tune them — over-tuning multiple parameters at once makes debugging output quality much harder

Prompt Engineering for Function Calling and Agents

Gemma's tool-calling only works reliably when tool descriptions and system instructions are precise — vague descriptions are the top cause of agents either ignoring tools or calling them unnecessarily.

Write detailed, specific tool descriptions — the model relies entirely on these descriptions to decide which tool to call and when
Use required fields in your JSON schemas to prevent the model from omitting critical parameters
Limit the number of available tools to 5-10 per call; too many options measurably confuses tool selection
Include example values in parameter descriptions (e.g., "city name, e.g. 'Lagos', 'London'") to guide correct argument formatting
If the model ignores tools and answers directly from its own knowledge, add an explicit instruction like "Never guess account balances or transaction data — always use the provided tools"
If the model calls tools unnecessarily, tighten the description with a scoped condition like "Use ONLY when the user explicitly requests a balance check"
Set a max_steps limit in your agent loop to prevent infinite tool-calling loops
Handle tool errors gracefully by returning structured error info, so the model can retry or explain the failure to the user rather than silently failing

For a fintech agent handling account queries or bill payments, these guardrails matter more than general prompt wording — a well-described tool schema does more work than any amount of persuasive phrasing in the prompt itself.

Break Down Complex Multi-Step Logic

If a single prompt requires conditional multi-step reasoning ("do X first, if the result is A do M, otherwise do N, then do Y"), Gemma performs more reliably when you split this into separate calls chained together in code, rather than asking it to execute all the branching logic in one shot. Keep each individual call focused on one clear, bounded task.

Keep Output Short and Deliberate

Inference speed is heavily tied to output length, so trimming unnecessary verbosity has both a quality and a performance benefit.

Ask explicitly for the shortest output that satisfies the task, then do lightweight post-processing in code rather than relying on the model to format everything perfectly
For on-device or low-latency use cases (E2B/E4B), this matters even more since every extra generated token adds directly to response time
Avoid asking for both a long explanation and a structured answer in the same call — request the structured answer only, and log reasoning separately if you need it for debugging

Python Example: Function Calling End-to-End

Here's a minimal, working pattern using the transformers library and apply_chat_template with a tools schema — the same approach that avoids the manual-serialization pitfall described above.

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch, json

model_id = "google/gemma-4-4b-it"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16, device_map="auto")

def check_balance(account_id: str) -> dict:
    """Look up the current balance for a given account ID."""
    return {"account_id": account_id, "balance": 42500.00, "currency": "NGN"}

tools = [
    {
        "type": "function",
        "function": {
            "name": "check_balance",
            "description": "Use ONLY when the user explicitly requests an account balance check. Never guess balances yourself.",
            "parameters": {
                "type": "object",
                "properties": {
                    "account_id": {
                        "type": "string",
                        "description": "The account identifier, e.g. 'ACC-10293'"
                    }
                },
                "required": ["account_id"]
            }
        }
    }
]

messages = [
    {"role": "user", "content": "Can you check the balance on account ACC-10293?"}
]

prompt = tokenizer.apply_chat_template(
    messages,
    tools=tools,
    add_generation_prompt=True,
    tokenize=False
)

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=256, temperature=0.1)
response = tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True)

try:
    call = json.loads(response)
    if call.get("name") == "check_balance":
        result = check_balance(**call["arguments"])
        print(result)
except json.JSONDecodeError:
    print("Model returned plain text:", response)

A few things this example demonstrates in practice:

The tool description explicitly scopes when the model should call it, directly applying the "tighten the description" guidance above
required: ["account_id"] prevents the model from calling the function without a valid argument
Temperature is set to 0.1, matching the deterministic-task guidance for anything feeding into backend logic
The response is parsed defensively with a try/except, since production agent loops should always handle the case where the model replies in plain text instead of a structured call

For a full agent loop, wrap this in a while loop with a max_steps counter, feeding the function's return value back into the message history as a tool role turn before calling generate again.

Common Prompting Mistakes to Avoid

Mixing prompt formats from other model families ([INST], ChatML) instead of Gemma's native <start_of_turn> template
Manually inserting the BOS token or hand-serializing tool schemas instead of using apply_chat_template
Writing vague tool descriptions and expecting reliable function-calling decisions
Using a single temperature setting across wildly different tasks (extraction vs. creative writing)
Asking for multi-step conditional logic in one prompt instead of chaining focused calls
Skipping few-shot examples when strict output formatting (like JSON) is required for backend parsing

Getting these fundamentals right — correct chat template, clear structure, task-appropriate temperature, and precise tool descriptions — resolves the majority of "Gemma isn't following instructions" complaints before you need any deeper fine-tuning.

Running, Building, and Optimizing with Gemma 4

Agbo, Daniel Onuoha — Sun, 05 Jul 2026 01:00:00 +0000

gemma 4 isn't just an open-weight model you download — it's a toolkit for running AI fully offline, building real generative applications, and squeezing maximum performance out of whatever hardware you have. This guide covers the practical side: local setup, offline app architecture, GenAI application patterns, and performance tuning.

Running Gemma 4 Locally

The fastest path to running Gemma 4 on your own machine is through Ollama, which wraps quantized GGUF weights in a simple CLI and local API — no GPU required for smaller models.

# Install Ollama, then pull a Gemma 4 variant
ollama pull gemma4:e4b      # edge-friendly, multimodal
ollama pull gemma4:26b      # MoE, faster decode
ollama pull gemma4:31b      # dense, max quality

# Run interactively
ollama run gemma4:e4b "Summarize this transaction log"

# Or hit the local REST API
curl http://localhost:11434/api/generate -d '{
  "model": "gemma4:e4b",
  "prompt": "roses are red"
}'

For raw inference speed rather than convenience, llama.cpp with CUDA acceleration outperforms Ollama's wrapper — benchmarks on an NVIDIA DGX Spark show llama.cpp hitting ~65 tokens/sec versus ~60 tok/s for Ollama and ~45 tok/s for vLLM's NVFP4 path on the 26B MoE model. vLLM remains the better choice once you need a production-grade OpenAI-compatible API server with batching, rather than a single local session.

Tool	Best For	Notes
Ollama	Quick local setup, prototyping	GGUF quantized, simplest CLI/API
llama.cpp	Raw inference speed	Direct CUDA/Metal control, no wrapper overhead
vLLM	Production API serving	Batching, concurrent requests, OpenAI-compatible
MLX	Apple Silicon	Native Metal acceleration on Mac

On Apple Silicon, performance scales directly with unified memory: an M1 8GB Mac manages roughly 12 tokens/sec on a 12B Q4 model, while an M4 Max 48GB comfortably runs larger models at ~35 tokens/sec.

Building Offline and Low-Connectivity AI Applications with Gemma

Gemma's on-device sizes (E2B and E4B) are purpose-built for environments with unreliable or absent connectivity — a pattern directly relevant to logistics and fintech deployments in areas with inconsistent network coverage.

Use Google AI Edge Gallery or AICore on Android to embed E2B/E4B directly into a mobile app, with no server round-trip required
Target edge boards like NVIDIA Jetson Orin Nano for IoT and drone-based systems that must reason locally (e.g., an Atoovis-style delivery drone classifying obstacles without a live connection)
Quantize aggressively (Q4_K_M or IQ4_XS) so the model fits in constrained RAM on field devices, trading a small quality drop for a much smaller memory footprint
Design a "store-and-forward" pattern: the local model handles inference and decision-making offline, then syncs logs or embeddings to your backend (e.g., MongoDB Atlas) once connectivity returns
For voice-driven offline use cases like field agent check-ins, use E2B/E4B's native audio input instead of a separate speech-to-text pipeline, reducing both latency and points of failure

This architecture matters for fintech agent apps operating in rural Nigeria or similar low-bandwidth regions, where a cloud-dependent LLM call would simply fail rather than degrade gracefully.

Building Generative AI Applications with Gemma 4

Beyond chatbots, Gemma 4's function-calling and structured JSON output make it suitable as a reasoning layer inside existing backend systems rather than a bolted-on feature.

Agentic backend services: wire Gemma 4's function-calling directly into your Node.js/Express routes so the model decides which internal API to call (e.g., checking a Kredi Bank balance or triggering a VTU top-up) instead of parsing free-text intent yourself
Document and receipt processing: feed scanned invoices or bank statements through Gemma 4's vision input for OCR plus structured extraction, replacing brittle regex-based parsers
Local coding assistants: run Gemma 4 inside tools like OpenCode or via llama.cpp to get an offline pair-programmer for sensitive codebases you don't want sent to a third-party API
RAG pipelines: combine Gemma 4 with LangChain and a local vector store to build a retrieval-augmented assistant over your own documentation or transaction history, entirely self-hosted
Multilingual support apps: leverage Gemma 4's 140+ language coverage to serve customer support or bill-payment flows in local languages without a separate translation layer

A minimal Gradio-based coding assistant, for example, pairs Ollama's local API with a simple web UI to demo tool-calling and live code editing in an afternoon — a useful pattern for internal developer tools.

Optimizing Gemma for Performance and Efficiency

Most "Gemma is slow" complaints trace back to three fixable issues: CPU fallback, the wrong quantization, and an oversized context window.

Check GPU utilization first. If GPU usage stays at 0% during inference, the model silently fell back to CPU — expect only 1-5 tokens/sec until this is fixed.

Pick quantization deliberately:

Quantization	Size (12B)	Speed	Quality	Best For
Q4_K_M	~7 GB	Fastest	Good	Daily use, most tasks
Q5_K_M	~8.5 GB	Fast	Better	When quality matters
Q6_K	~10 GB	Medium	Very good	Balanced
Q8_0	~13 GB	Slow	Near-original	Quality-critical tasks
FP16	~24 GB	Slowest	Original	Only with ample VRAM

Watch context length — it isn't free. VRAM and speed degrade sharply as context grows: a 12B Q4 model runs at full speed with a 2K context but drops to roughly a quarter of that speed at 256K context, consuming 30GB+ of VRAM in the process.

Manage the KV cache. Long-running conversations accumulate key-value cache that eats VRAM over time — reset sessions periodically or cap the cache size for long-lived services.

Use Quantization-Aware Training (QAT) checkpoints where available. Google has released QAT versions of Gemma 4 that preserve much more quality at int4 precision than post-training quantization alone, making it realistic to run a 12B model with a 16K context window on as little as 8GB of VRAM, even on older GPUs like a GTX 1080 Ti.

Mixture-of-Experts changes the math. The 26B MoE model activates only ~3.8B parameters per token versus 30B+ for the dense model, delivering roughly 6x faster decode speed at comparable quality — a strong default choice when latency matters more than peak raw capability.

# Example: capping context and using a fast quant for a responsive local API
ollama run gemma4:e4b --ctx-size 8192