DEV Community: Yaniv Shenhav

Your First AI Agent Task Needs a Contract, Not More Permissions

Yaniv Shenhav — Mon, 12 Oct 2026 08:06:05 +0000

Adapted from the Hebrew guide "AI Agent: How to Choose a Task, Set Boundaries and Check the Result" on AI NEWS IL (published 17 September 2026, updated 24 September 2026).

An agent's first production task should be small enough to test and safe enough to stop. "Manage our support inbox" is neither. "Classify one type of support request, find the relevant policy and prepare a draft for a representative" is a better starting point.

This guide turns that narrower task into a testable contract. The support example is hypothetical, not a case study or a claim of measured results.

1. Decide whether you need an agent

Use an agent when the goal is clear but the next step depends on what it finds. A support request may require a policy lookup, another document or a handoff to a person.

If every step is fixed, ordinary automation is usually easier to predict and test. Copying a field, sending a scheduled reminder or generating a standard report does not automatically need an agent.

Before connecting anything, write one sentence:

The agent receives X, returns Y, and Z decides what happens next.

If you cannot name the output or the person responsible, the task is not ready. Start with a reversible task. Do not make payments, deletion, publication or permission changes part of the first pilot.

2. Use a narrow support-triage example

A customer writes: "My monthly charge has increased. Why?"

The agent can:

Classify the request as billing.
Retrieve the approved policy section about price changes.
Draft an explanation with a link to that section.
Flag that it has not checked the customer's invoice and cannot confirm the amount.

A representative opens the invoice, checks the policy, corrects the draft and decides whether to send it. The agent does not send the reply or update the customer record.

The point is to reduce search work without inventing service terms or exposing information. Adapt the example to your systems, privacy rules and business risks before using it.

3. Define an input and output contract

Allow only the information needed: the request text, a minimal customer reference, the response language and links to approved policies. Keep passwords, payment details and unrelated files out of the model input.

Use a fixed output shape:

Field	What the reviewer checks
Category	Is it one of the allowed categories?
Confidence or uncertainty	What remains unresolved?
Source section	Does the cited policy support the answer?
Draft reply	Has the agent added a promise or changed the policy's meaning?
Items to verify	What must the representative check before sending?

Define invalid outputs too. A missing source link, an unknown category or an unsupported refund promise should produce "review required" with a reason, rather than a ready-to-send draft.

A missing or conflicting document is a reason to stop and ask for review. It is not a blank the model should fill.

4. Separate reading, drafting and acting

For the first test, grant read access only to the approved policy folder and test requests. Do not give the agent the entire inbox, full CRM, invoices or contact directory because they might be useful later.

Keep credentials outside prompts and user-visible logs. Limit access to the environment and operations required, and use short-lived credentials where supported.

Treat these as separate permission stages:

Read approved material.
Produce a draft.
Take an external action.

Passing the first ten cases does not authorize sending messages. That is a new capability with a new risk and needs its own decision and tests.

Ask the organization's access owner to approve the permission list. Also ask what an attacker could reach if the credential leaked. Narrow the scope before the pilot if the answer is unacceptable.

5. Test failure cases before real use

Write the expected outcome for each case before running it: draft, clarification or stop.

Include routine requests alongside:

Missing information or conflicting policies.
An outdated document or the wrong customer reference.
An unavailable external service.
Sensitive information or a request requiring a financial decision.
Instructions embedded in a customer message or file that conflict with the workflow.

For example, a customer message might contain a demand to ignore the policy and approve a full refund. The agent should apply the approved policy and flag the request for review. The message is task data, not permission to change the workflow.

Measure category accuracy, source fit, changes in meaning, reviewer corrections, time to approval and cost. There is no universal number of test cases that proves readiness. Expand the test set according to task variety, error severity and how easily harm can be detected.

6. Make stopping and rollback concrete

Stop when a required source is missing, information is sensitive, uncertainty is unresolved, a financial decision is needed or a tool attempts an operation outside the allowed scope.

Name one process owner who can stop the pilot. Record the limited inputs, sources, tool calls, output and human decision under your privacy policy. Changes to the model, prompt, data or permissions should trigger regression tests.

Before the pilot, rehearse rollback: revoke the credential, return requests to representatives and verify that the manual process works. A rollback plan that has never been tried is still an assumption.

7. Measure accepted work, not generated drafts

Run a short, defined draft-only pilot after the test set passes. Two weeks is an example, not a universal requirement.

For each draft, record whether it was:

Used with little change.
Used after corrections.
Rejected.

Track review time and whether the source was correct. Compare similar request types with and without the agent. A fast draft is not a saving if checking it takes longer than writing the reply manually.

Set the continuation rule before the pilot. For example, continue only if corrections are small, no draft invents service terms and handling time falls. Stop for a data leak, skipped human review or costs that exceed the benefit.

The first milestone is a useful draft under controlled permissions. More access is a separate decision, not a reward for a convincing demo.

AI Knowledge Base Without Stale Answers: A Practical Guide to Freshness, Permissions and Sources

Yaniv Shenhav — Mon, 05 Oct 2026 08:12:01 +0000

Adapted from the Hebrew guide on AI NEWS IL.

A knowledge base for an AI assistant does not start by uploading every file into one folder. It starts with a source register: an owner, a validity window and an access rule for each source. When retrieval, permissions and the update path work together, you connect it to real users.

The goal is not a model that never errs. The goal is an error you can detect: which source was returned, when it was approved, who may see it, what was missing and who fixes it. A knowledge base that cannot retire an old document or honor a permission change will give fluent, wrong answers even if the model never changed.

1. Build a source register before you index

Create one row per source, not per folder. Each row answers: what is the document, who owns it, who is it for, when is it valid, what replaces it, and how is it removed from the index.

Do not assume the newest file wins. Upload date can be later than content date, and a copy moved between folders can look new while being stale. Define a precedence order per information type. A signed policy may beat a training deck, and a numbered correction notice may beat the policy until a new version ships. If there is no rule, the right state is conflict. Do not ask the model to pick the document that sounds more convincing.

2. Separate active, historical and draft content

A historical document can matter for audit, but it must not silently compete with the active one. Keep it in an archive behind access control and remove it from the normal answer path with a status field and an expiry date. A draft stays out of the active store until the owner approves it. A filename that says "final" is not approval.

Useful metadata at minimum:

source_id: expenses-policy-2026
owner: ops-team
audience: all-employees
status: active        # draft | active | superseded | expired
effective_from: 2026-01-01
reviewed_at: 2026-09-01
version: 3
supersedes: expenses-policy-2025

Some search products can filter on file attributes. That capability does not set your policy. Define what each field means and who may change it first, then implement filters in the product you chose.

3. Keep permissions end to end

Access to the knowledge base is not permission to see everything in it. At ingest, store the audience of the source. At query time, filter results by the user's identity. In the answer, never show, quote or link a document the user may not open. A broad service account is not a reason to widen what the user sees.

Check the source permissions too. A product that respects user permissions can still surface content that was shared too widely in the past. Microsoft's documentation, for example, explains that SharePoint and OneDrive controls affect discovery without changing user permissions, and that lifecycle and sharing policies help reduce oversharing. That is a statement about Microsoft 365 Copilot, not a guarantee for every RAG tool.

Start read-only. The knowledge base admin can import, disable and rebuild. A normal user can ask. The model gets no write access to source documents. Deleting, changing an audience or replacing a version are logged admin actions, not the result of a chat.

4. Make ingestion verifiable

A typical pipeline is ingestion, text extraction, chunking, embeddings, indexing and retrieval. Google's RAG Engine documentation shows a similar chain. It describes the product. It does not prove your documents were chunked well or that the output is faithful to the source.

Log every run: run ID, source list, versions, successes and failures. A file is not searchable just because you sent it to an API. OpenAI's File Search documentation, for example, says to check that a file reached the completed state, and it describes file citations and metadata filtering. If ingestion is partial, keep the previous version active or block the topic. Do not mix half an update without a marker.

The tooling can be simple: a sheet or database for the register, an exporter or crawler, a parser, a search engine or vector store, a link checker, a set of reference questions, and a log that ties each question to the retrieved results and the answer.

5. Require an openable, versioned citation

The answer should show a clear source name and a link that opens with the user's permissions. For testing, also keep source_id, version and chunk ID. A filename alone can change or repeat across folders. If the source cannot be opened, the citation does not let the user verify the claim.

A citation shows the system linked a claim to a file. It does not prove the claim follows from it. When testing, compare each material claim to the source sentence, check that the context was not cut off, and check that the source was valid at answer time. Also sample answers where a similar paragraph was found but belongs to another audience, product or period.

6. Build an acceptance test for retrieval

Do not only check whether the wording sounds good. For each question, store the expected result at three levels: which sources should be retrieved, what may be concluded from them, and what the correct decision state is. That lets you tell a retrieval failure from a wording failure. If the right document never reached the context, a better prompt is not the fix.

Include:

a normal question and a rephrased one
a technical term in another language
a superseded document
two conflicting versions
a user without permission
a file that failed ingestion
a question with no answer
a deleted link

There is no universal number of questions that guarantees quality. Pick coverage by content variety and the harm a wrong answer can cause. Useful metrics: right-source retrieval rate, share of answers with a valid source, conflicts detected, permission leaks, unanswered questions that correctly stopped, and time from approval to active. An average cannot make up for a permission leak or use of a revoked document.

7. Update and remove without a gray zone

Define a change path: the owner approves a version, the admin ingests it, regression tests run, and only then does an alias switch to the new active index. Keep a way back to the previous index, but never restore a document that was revoked for safety or privacy reasons. A technical rollback is subject to the business status of the source.

When a permission is removed or a document is revoked, set a response time and check every copy: source, cache, index, saved results and the test environment. Do not promise instant deletion if the product describes another retention period. Record what was deleted, what remains under policy, and when you checked.

Example (hypothetical): expense policy

This is a proposed design that was not tested in production. An ops team wants the assistant to answer travel expense questions. There is an active policy, an old training deck and a draft for next year.

Input: the active policy and an approved correction notice. The deck is marked superseded. The draft stays out of the active store.
Output: a short answer, an amount or rule only if it appears in a valid source, a link to the section, the version, and a state of answered or needs-clarification.
Permissions: all employees can read the policy. Only ops manages versions. The model cannot write or approve expenses.
Tests: a regular employee, a unit with an exception, Hebrew and English phrasing, a question about next year, a broken link, and a conflict between the notice and the policy.
Stop conditions: no active source found, two equal-rank sources conflict, the user is not authorized, a file did not finish ingestion, or the answer adds a condition the source does not contain.

"Contact ops" is an acceptable result when there is no valid source. It is expected behavior that keeps the model from inventing policy.

Failure cases

Two active versions: stop with conflict, show both sources to the owner, do not pick automatically.
Partial ingestion: do not turn on the new index. Log the file and the error, retry without lowering the threshold.
Source too widely shared: fix the sharing or remove the source until a decision. An app-level filter does not replace source access control.
Valid document not retrieved: mark a retrieval failure. Check extraction, chunking, metadata and the query before you rewrite the answer.
Citation does not support the claim: block the claim, fix the context check, add the case to regression.
No answer in the base: return no-valid-source with an escalation path. Do not fill in general knowledge as if it were company policy.
Broken link: show that the source cannot be verified and route it to the owner.
Permission changed after indexing: run removal or rebuild, invalidate caches, and test with the user's identity that access is gone.

Limits

RAG or file search does not make a source correct, current or permitted. Chunking can separate a rule from its exception, metadata can be wrong, retrieval can miss a term, and a model can state a conclusion the source does not support even with the right context. Permissions in one product do not prove permissions in another.

This guide does not replace data classification, a privacy assessment, legal advice, records management or a vendor contract review. Retention, deletion, processing region and sub-processors depend on the product, the plan and the real settings. Before using personal, confidential or regulated material, get approval from the responsible people in your organization.

Sources

NIST AI 600-1, Generative AI Profile (published July 26, 2024). A voluntary cross-cutting framework for AI risk management. The source register and worksheet here are a proposed implementation, not a NIST requirement.
OpenAI API, File Search documentation. Describes ingestion status, file citations, search result inclusion and metadata filtering for that product.
Microsoft Learn, Microsoft 365 Copilot data protection architecture. Explains how permissions, sensitivity labels, sharing and lifecycle policies affect Copilot. Not a deployment guide for every RAG system.
Google Cloud, Vertex AI RAG Engine overview. Lists the ingestion, transformation, embeddings, indexing, retrieval and generation chain.

Originally published in Hebrew on AI NEWS IL (ainewsil.co.il), with a printable worksheet. This is an English adaptation.

The full Hebrew guide, with the printable worksheet, is on AI NEWS IL: read the original article. The Israeli Institute for AI runs AI training and implementation programs for organizations worldwide.

Seven Controls to Put in Place Before an AI Agent Gets Production Access

Yaniv Shenhav — Sun, 20 Sep 2026 19:26:15 +0000

AI agents cross a line that ordinary chat systems do not: they connect probabilistic decisions to real actions.

An agent that can read mail, update a record, run a tool, or spend money is part of your production control plane. Prompt quality matters, but prompts are not a security boundary. The controls below should exist before an agent receives production access.

1. Separate read, recommend, and execute

Do not treat "access" as a single permission.

Use three distinct stages:

Read - retrieve the minimum data needed for the task.
Recommend - prepare a proposed action without changing external state.
Execute - make the approved change through a narrowly scoped tool.

This split gives you useful checkpoints. A support agent may read a ticket and draft an answer automatically, while sending remains gated. A finance agent may analyze invoices but require an approval before changing payment details.

2. Give every tool its own narrow permission

A single broad credential turns every mistake into a high-impact mistake. Prefer separate credentials and scopes for separate tools.

Ask four questions for each tool:

Which resources can it access?
Which operations can it perform?
How long does the permission last?
Which environment does it apply to?

Short-lived credentials and environment-specific roles reduce the damage from a bad decision, a leaked token, or a malicious instruction hidden in external content.

3. Add an action budget

Agents need more than an API rate limit. They need a business-action budget.

Useful limits include:

maximum records changed per run;
maximum messages sent per hour;
maximum cost per transaction and per day;
maximum number of retries;
maximum number of external systems touched in one workflow.

The budget should fail closed. Crossing it should stop the run and request review, not trigger an improvised workaround.

4. Put approval at the point of commitment

Approval works only when the reviewer can see the exact action that will occur.

For a message, show the final recipient and wording together. For a deployment, show the target environment and revision. For a purchase, show the item, delivery address, and full total.

Avoid vague approvals such as "continue" when several actions are still possible. The approval object should be concrete enough to log and later audit.

5. Treat external content as untrusted input

Email, webpages, documents, issue comments, and retrieved knowledge can all contain instructions that conflict with the agent's actual task.

Do not let retrieved content redefine the agent's goals, choose new tools, expand permissions, or select a new disclosure destination. Parse it as data. Keep authority in a separate, trusted control path.

Test whole workflows rather than individual prompts. A sequence of individually allowed actions can still produce a harmful result.

6. Log decisions and effects separately

A useful audit trail answers two different questions:

Why did the agent decide to act?
What changed in the external system?

Record the task identifier, tool, target, approval reference, before/after state, and provider response. Keep logs outside the agent's own write permissions when possible.

Observability should also include alerts for unusual tool combinations, repeated failures, sudden increases in volume, and actions outside normal hours.

7. Build a real stop mechanism

A stop button is not a prompt that says "please stop." It is an independent control that can revoke credentials, disable tool execution, or block the workflow runner.

Define who can activate it, what it disables, and how recovery works. Practice using it before launch.

A small pre-production failure drill

Before giving an agent live access, run at least these tests:

A document contains an instruction that conflicts with the assigned task.
The agent is given the wrong recipient or resource identifier.
A required external service fails halfway through the workflow.
The same event is delivered twice.
An approval expires before execution.
The agent tries to exceed its action budget.
The stop mechanism is activated during a run.

The goal is not to prove that the agent never fails. It is to prove that failures are bounded, visible, and recoverable.

The minimum launch checklist

Before production access, confirm that you have:

a tool allowlist;
least-privilege, short-lived credentials;
separate read and write paths;
explicit approval for high-impact actions;
action and cost limits;
idempotency protection;
tamper-resistant logs;
monitoring and alerts;
a credential-level stop mechanism;
tested failure and recovery paths.

An AI agent should earn more authority only after its behavior is observable under real conditions. Start with read-only access, measure what happens, and expand one permission at a time.

This article is an English technical adaptation of the original Hebrew guide published by AI NEWS Israel: לפני שנותנים לסוכן הרשאה: שבע בקרות שחייבות להגיע לפני הייצור.