DEV Community: Guido Tapia

A research assistant grounded in your own precedents and knowledge

Guido Tapia — Wed, 30 Sep 2026 04:42:16 +0000

A senior associate wants to know how the firm usually drafts a limitation of liability clause for a software supply agreement. The answer is in the firm somewhere, either in the precedent bank or in a file note a partner wrote three years ago. A general-purpose chatbot will write a confident clause from its training data. The associate needs the firm's own clause and a link to the document it came from.

This post is part of our Practical AI for Law Firms series. It covers how we would build that assistant. The design uses retrieval-augmented generation (RAG) over the firm's own material. Answers cite the source document, the assistant refuses when the corpus has nothing, and access control works at matter level so information cannot cross an ethical wall.

What grounding means in practice

A grounded assistant answers only from the documents it retrieved for the current question. Every claim in the answer points to a passage in a specific document, and the reader can open that document and check it.

The open-source ai-advisory-board project has a clean version of this pattern. A recent change added a Materials sidebar with versioned source citations, which ties each answer to the exact version of the document it used. Versioning matters for a precedent bank because precedents get revised after a judgment or a change in legislation. A citation to "the indemnity precedent" tells the reader much less than a citation to version 4, approved in March, which they can compare with the current version.

Refusal is the other half. If retrieval returns nothing relevant, the assistant says so and stops. It doesn't fall back on general knowledge, because that is where invented authorities come from.

The failure mode grounding prevents

Courts in Australia and overseas have dealt with submissions that cited cases that don't exist. General-purpose AI tools produced the citations and nobody checked them before filing. Several Australian courts have since issued practice notes or guidance on using generative AI in proceedings. Any firm planning an assistant should read the one for its jurisdiction before it designs anything.

These incidents have one cause. Someone asked a model for authority, and it produced text that looked like authority with no document behind it. A grounded design closes that path. The model sees only the retrieved passages. The system attaches citations from retrieval metadata, and a post-check drops any reference that doesn't match a passage in the retrieved set.

The lawyer still reviews the answer before relying on it. That review is quick, because each citation opens the paragraph it came from.

An architecture sketch

The components are ordinary. The design work is in deciding where each control sits.

Ingestion: connectors pull from the document management system (DMS), the precedent bank and the knowledge wiki. The connectors are read-only. The same PR adds an optional read-only local MCP reader that works the same way: the assistant can read documents but has no way to change them.
Chunking and metadata: each document is split into passages. Every passage carries its document ID, version, practice group, matter number where there is one, and the access groups copied from the DMS.
Index: a vector index combined with keyword search, hosted in an Australian cloud region. Legal text needs the keyword side because embeddings handle defined terms and clause numbers poorly.
Retrieval: every query runs under the user's own identity, and the index applies the access filter inside the search call.
Generation: the model receives the retrieved passages. Its instructions are to answer only from them and to say plainly when they don't cover the question.
Citation check: a post-processing step confirms that every cited passage ID was in the retrieved set. It removes or flags anything that wasn't.
Audit log: every query is logged with the user and the passages returned, so the firm can reconstruct any answer later.

The agentic-graphrag project made a similar choice when it grounded its query generation in the engine's resolved schema in place of a hardcoded description, with one bounded retry. The same change treats the caller's base scope as an authorisation boundary. Filters on an individual call can narrow that scope but never widen it, and the system refuses requests outside it. Matter access needs exactly that rule.

In outline, the request path looks like this:

def answer(question, user):
    scope = access_scope_for(user)    # DMS groups plus the wall register
    passages = index.search(
        question,
        filter=scope,                  # enforced by the index at query time
        top_k=12,
    )
    relevant = [p for p in passages if p.score >= MIN_RELEVANCE]
    if not relevant:
        return Refusal("No matching precedent or know-how in the firm's library.")

    draft = llm.generate(question, context=relevant, rules=GROUNDED_ONLY)
    allowed = {p.id for p in relevant}
    cited = [c for c in draft.citations if c.passage_id in allowed]
    log_query(user, question, relevant)
    return Answer(draft.text, citations=cited)

Ethical walls belong in the retrieval layer

The firm's DMS already records who can open which matter, and the assistant has to inherit those permissions exactly. Some information barriers are recorded outside the DMS, such as a conflicts register entry that walls a team off from a former client's files. The assistant has to honour those as well.

The filter has to run inside the index query. Suppose walled documents reach the model's context and you try to strip them from the answer afterwards. The model can still paraphrase what it read. When the index applies the filter, walled material is never retrieved, so it has no route into an answer.

Some practical points from designing this:

Sync permissions often. A wall put up this morning must apply to this afternoon's queries. Permission changes should reach the index as events, with a scheduled full reconciliation as a backstop.
Decide carefully what goes into the shared corpus. A precedent that has been cleaned of client detail and approved for firm-wide use belongs there. A raw advice on a matter file stays in that matter's scope, even if it is the best example of the point.
Keep refusals identical. The user should get the same message whether nothing exists or they lack access, so a refusal doesn't reveal that a walled document exists.
Test the walls. Write a set of questions that must return nothing for a walled user, and run it on every index rebuild.

Refusal is a written policy

Where you draw the refusal line is a choice, and teams do change it. The argus project, a financial research assistant, recorded a decision to reverse its earlier refusal of forward-looking questions. It now answers them as grounded scenarios built from cited inputs. For a law firm we would start stricter and answer only from the corpus. Practice groups can then loosen the rule for particular question types, with each change written down and approved by a named partner.

Where the documents live

Precedents and know-how are among a firm's most sensitive assets. In September a widely discussed report alleged that OpenAI took mathematicians' private research from their Codex chats. The allegation is unverified. It still shows why firms are wary of pasting confidential material into hosted AI tools.

For this assistant we would keep the index and document store in the firm's own cloud tenancy, in an Australian region. The model endpoint should be under enterprise terms that exclude training on inputs, and those terms should be confirmed in writing before any client material is indexed. The firm owes clients the same confidentiality in the index as it does in the DMS.

Limitations and costs

Corpus quality sets the ceiling. If the precedent bank holds conflicting versions of a clause with no status marked, the assistant will retrieve all of them. Most of the effort in a first project goes into marking which precedent is current and which are superseded.
The assistant doesn't research external law. Case law and legislation still come from the firm's subscription services. The assistant reports what the firm has written and done before.
Retrieval sometimes misses. If a question is phrased differently from the precedent, the search can fail to find it, and the assistant refuses even though the answer exists. Reviewing refusals regularly shows where the corpus or the search needs work.
Model usage is usually the smaller cost. Ingestion, permission sync and evaluation take most of the effort, and that work continues after launch because the corpus keeps changing.

A sensible first project covers one practice group's precedent bank under its existing permissions. That group's lawyers write the test questions. The assistant then runs under supervision for a few weeks before anyone else gets access.

PicNet builds production AI systems for Australian organisations. If a research assistant grounded in your own precedents is on your list, talk to us about what a first project could look like.

Originally published at picnet.com.au.

Code-first integration with Centazio: why we built an open-source alternative to drag-and-drop iPaaS

Guido Tapia — Mon, 28 Sep 2026 04:38:23 +0000

Integration projects often go the same way. A team buys an iPaaS subscription, drags a CRM connector onto a canvas and wires it to the finance system. By Friday, records are flowing. Six months later the canvas has forty boxes. A dozen of them hold expressions that only one person understands, and nobody wants to touch any of it before end of month.

We built Centazio, PicNet's open-source data integration platform, to do integration a different way. In Centazio an integration is plain .NET code, split into small isolated functions, and developers review and unit test it the same way they handle the rest of their code.

This post is part of our AI-Ready Data and Integration series. An AI system that reads your operational data picks up every duplicate and stale record your integrations produce, so AI readiness has to start at the integration layer.

Where drag-and-drop tools fall short

Visual integration tools demo well because a demo only runs the happy path: a record is created in one system and shows up in another. Most of the work in a production integration is the other cases. The target API goes down for an hour. Someone edits a record in both systems between syncs. A vendor renames a field, or a rate limit kicks in halfway through a batch.

Every one of those cases needs logic. On a canvas, that logic ends up in condition blocks and expression fields spread across the flow, where it is hard to find and harder to change safely. These are the problems we run into most:

Version control is weak. Flows are usually stored as large generated JSON or XML files, so a diff tells a reviewer very little about what changed.
Testing is manual. You can usually run a flow against a sandbox. What you can't do is write an automated test that says "this CRM contact becomes this finance contact" and have it run on every change.
Pricing grows with usage. Licences are often metered per connector or per task run, and many are billed in US dollars, so an Australian budget moves with the exchange rate.
The logic is locked in. A flow built in one vendor's designer won't open in another vendor's designer, so if you leave, you rebuild.

How Centazio structures an integration

Centazio splits every integration into three kinds of step. Each step runs as its own serverless function:

A read function for each source system pulls changed records and stages them in that system's own format.
A promote function takes the staged records, validates them and maps them into a shared core model that your organisation defines.
A write function for each target system takes core records and pushes them out in the shape that system expects.

No system knows about any other. The CRM read function has no idea the finance system exists, and the finance write function only knows the core model. To add a third system, you write its functions against the core model and leave the existing ones alone.

Fault tolerance comes from the structure

Each step reads from storage and writes to storage, so a failure stays in the step where it happened. Say the finance system's API is down. Its write function fails and tries again on the next run. Meanwhile the CRM read and the promote step keep running, so the core model stays current. When the API comes back, the write function processes everything that queued up in the meantime. Nobody has to replay a flow by hand or work out which half of a batch went through.

The split also helps when a mapping turns out to be wrong. The raw data is still in staging, so once the promote code is fixed you re-promote what was already read. You don't need to pull it from the source system again, which matters when that source has tight API limits.

Every function is small and does one job, so when a log entry or alert fires, it points to one step for one system.

Tests come with the design

A promote step takes a staged record and returns either a core record or a reason to skip it. That is ordinary code with no network calls, so you unit test it like any other code. The sketch below shows the pattern. The type names are made up for illustration and are not Centazio's exact API.

public class ContactPromoter {
  public PromoteResult Promote(CrmContact c) {
    if (String.IsNullOrWhiteSpace(c.Email)) return PromoteResult.Ignore("no email");
    var name = $"{c.First} {c.Last}".Trim();
    return PromoteResult.Ok(new Customer(name, c.Email.ToLowerInvariant()));
  }
}

[Test] public void Promote_ignores_contacts_without_email() {
  var res = new ContactPromoter().Promote(new CrmContact("Jo", "Smith", ""));
  Assert.That(res.Ignored, Is.True);
}

Tests like this run in the build pipeline on every change. When someone changes a mapping rule, the pull request shows exactly which line changed and which test covers it. An IT manager can approve that after reading the diff, without sitting through a demo in a sandbox.

What code-first costs

Code-first has real costs, and they should be on the table before anyone picks it:

You need .NET developers, either in-house or through a partner. A business analyst can't build a new flow in an afternoon.
You host it and you run it. The functions run in your own cloud tenancy. That makes it easier to keep data in an Australian region when your privacy obligations or client contracts require it, but the monitoring, patching and cloud bills are also yours.
Connectors are code too. An iPaaS may ship a ready-made connector for a popular SaaS product. With Centazio, someone writes and maintains the read and write functions for that product's API.
Open source doesn't come with a vendor support desk. The code is yours to read and change, and support comes from your own team or from us.

When an iPaaS is still the right call

A commercial iPaaS is often the better fit in these situations:

The systems are mainstream SaaS products with maintained prebuilt connectors, and the flows are simple one-way pushes.
Volumes are low, and a missed or duplicated record costs little.
There is no development capacity, none is planned, and the people who own the process need to change the flows themselves.
The integration won't be around long, for example a bridge during a system migration.

Centazio suits the other end of the scale. That means several systems sharing the same entities, two-way sync, business rules that keep changing, and data that other systems will depend on. We use one test to decide: would you accept this logic untested if it lived in your main application? If the answer is no, it belongs in code.

Where it runs today

Centazio runs production integrations at CYCA and at Guide Dogs NSW/ACT, built with the read, promote and write structure described above. Both are plain .NET code, so changes go through code review and automated tests before they deploy, and neither organisation pays a per-task integration licence.

Why this matters for AI work

The core model is what pays off later. Once records from every system have been promoted into one validated shape, an AI project has one tested source to read from. If an agent needs to write back to a system, it can use the same write functions and the tests that cover them. AI tooling is moving towards code over configuration as well: OpenMCP, posted to Hacker News on 22 September, calls itself an open, code-first fork of the Model Context Protocol.

PicNet builds production AI systems for Australian organisations. If your data sits in systems that don't talk to each other, talk to us about what a first project could look like.

Originally published at picnet.com.au.

Watching the watchers: monitoring your CI/CD pipelines, branch protection and repo hygiene

Guido Tapia — Fri, 25 Sep 2026 04:38:23 +0000

Most organisations monitor production closely and barely monitor their delivery tooling. Someone sets up a nightly build or a branch protection rule and confirms it works that day. After that, nobody looks at it again until a release breaks or an auditor asks how changes get onto main.

This post is part of our Practical AI in DevOps series. It covers the continuous checks we run over GitHub and Azure DevOps, and a trap that produces false findings if you build those checks the obvious way. AI comes in at the end, where it turns a week of findings into a short digest so engineers only read what changed.

What the checks cover

We run five checks on a daily schedule:

Scheduled workflows and pipelines whose recent scheduled runs failed, or that have stopped running.
Default and release branches with no branch protection or ruleset, or with rules weaker than the organisation's baseline, such as no required review.
Open Dependabot alerts, grouped by severity and by how long they have been open.
Open secret-scanning alerts.
Stale branches with no commits past a threshold you choose (90 days is a reasonable start) and no open pull request.

Scheduled jobs are on the list because nobody is waiting for them. When a pull request build fails, the person who opened the pull request sees it. When a dependency refresh fails at 2am on a Sunday, the notice goes to whatever notification setting was configured, which may belong to someone who left last year.

How it fits together

We keep the architecture small on purpose:

A timer-triggered job runs once a day. A GitHub Actions workflow in a separate admin repository will do the job, and so will an Azure Function.
It reads GitHub through a GitHub App with read-only permissions for Actions, Administration, Contents, Dependabot alerts and Secret scanning alerts. For Azure DevOps it uses a service principal or a PAT scoped to Build (read) and Code (read).
Each check writes its findings to a store in one shared format. A JSON file per day in blob storage is enough.
Once a week, a diff step compares the latest findings with the previous week's.
A language model gets only the diff and writes the digest, which goes to a Teams channel or an email list.

A single finding looks like this:

{
  "source": "github",
  "repo": "example-org/payments-api",
  "check": "branch-protection",
  "subject": "main",
  "severity": "high",
  "detail": "Default branch has no protection rule or ruleset",
  "first_seen": "2026-09-11"
}

Each finding is identified by its source, repo, check and subject. The weekly diff only works if that identity stays stable from one run to the next.

GitHub branch protection has a trap of its own. The classic protection endpoint (GET /repos/{owner}/{repo}/branches/{branch}/protection) doesn't report repository rulesets, so a branch protected only by a ruleset looks unprotected. Query GET /repos/{owner}/{repo}/rules/branches/{branch} as well, and count either one as protection.

Dormant YAML files versus registered pipelines

The obvious first version of a pipeline check walks the repository tree, looking for files under .github/workflows or an azure-pipelines.yml. That approach is wrong in both directions. A file in a repository only records what someone once declared, and the CI platform decides what runs.

In Azure DevOps, a YAML file does nothing until someone creates a pipeline that points at it, and that pipeline can point at any path. Repositories collect dormant pipeline files from project templates and abandoned experiments. If you probe for azure-pipelines.yml, a dormant file makes it look as if CI exists when it doesn't. A repository whose real pipeline uses build/ci.yml gets reported as having no CI. A registered pipeline can also be paused or disabled, and a file probe can't see either state.

GitHub registers workflow files automatically, but the file still leaves things out. Scheduled triggers only run from the default branch, so a schedule in a file that exists only on a feature branch never fires. GitHub can also disable a workflow, and the API reports this as a state of disabled_manually or disabled_inactivity.

Ask the platform instead:

# GitHub: registered workflows and their state
gh api repos/ORG/REPO/actions/workflows \
  --jq '.workflows[] | {name, path, state}'

# GitHub: recent scheduled runs for one workflow
gh api "repos/ORG/REPO/actions/workflows/WORKFLOW_ID/runs?event=schedule&per_page=5" \
  --jq '.workflow_runs[] | {created_at, conclusion}'

# Azure DevOps: registered pipelines, their YAML path and queue status
curl -s -u ":$AZDO_PAT" \
  "https://dev.azure.com/ORG/PROJECT/_apis/build/definitions?includeAllProperties=true&api-version=7.1" \
  | jq '.value[] | {name, yaml: .process.yamlFilename, status: .queueStatus}'

Then join the file view to the platform view. A pipeline file with no registered definition gets an informational note saying the file is dormant. A registered definition whose YAML path no longer exists, or whose recent scheduled runs failed, is a real finding.

The same failure turned up recently in another tool. In a fix to mise, mise doctor reported a background watcher as "declared but not running" while the service manager was running it, and the command it recommended did nothing. The maintainers changed the health checks to use the same test as the command that acts on that state, so the checks no longer recommend a command that won't do anything. A monitoring check should read the same state the system acts on, and that is rarely a file path.

Keeping the hygiene reports readable

A stale-branch report that lists 300 branches will get ignored. Report a count per repository, list only stale branches that have no open pull request, and leave out release branch patterns you keep on purpose. Don't delete branches automatically. An old branch can hold the only copy of a hotfix that was never merged.

Any open secret-scanning alert should get a person's attention that week. The check records the secret type and the repository, with a link to the alert. It never stores the secret value, so secrets can't end up in anything sent to a model later.

For Dependabot, the useful signal is age and severity together. A critical alert that has been open for a day is normal triage. If the same alert has been open for six weeks, nobody is looking after that repository.

The weekly digest

Engineers stop reading a report that says the same thing every week. The diff step lists what is new and what was resolved since last week. It also carries forward anything still open past an age threshold. Only those items go to the model.

The model handles wording and grouping, and deterministic code decides what counts as a finding. The prompt asks for one line per item, grouped by team, with each line ending in the link supplied in the input. A cheap post-check compares every repository named in the digest against the input and rejects the output if the model names one that wasn't there.

An example digest:

New this week: the payments-api main branch lost its ruleset (link). The nightly dependency refresh in customer-portal has failed its last three scheduled runs (link). Resolved: two high-severity Dependabot alerts in reporting-service.

The model only ever sees metadata such as repository names, check names, severities and dates. Some organisations treat repository names as sensitive too. In that case, use a model hosted in an Australian region, such as Azure OpenAI in Australia East, and keep the digest inside your own tenant.

Costs and limits

GitHub and Azure DevOps both rate-limit their APIs. Large organisations need pagination and caching, and a daily run is plenty.
Dependabot alerts come with every GitHub plan. Secret scanning on private repositories needs GitHub's paid Advanced Security features. The Azure DevOps equivalent, GitHub Advanced Security for Azure DevOps, is also a paid add-on. Check what you are licensed for before promising coverage.
The model input is a short diff, so the model cost is small.
The platforms change under you. GitHub added rulesets alongside classic branch protection, and a check written before rulesets misses any rule configured that way. Budget time for maintenance.
The monitor is a scheduled job too. Give it a heartbeat, so that if a daily findings file doesn't arrive, something outside the monitor raises an alert.
The checks report problems and fix nothing. Every finding needs an owner, and the digest only helps if someone is expected to act on it.

PicNet builds production AI systems for Australian organisations. If a weekly digest over your pipelines sounds like a sensible place to start, talk to us about what a first project could look like.

Originally published at picnet.com.au.

Contract review and clause extraction: what AI can and can't do

Guido Tapia — Wed, 23 Sep 2026 04:38:26 +0000

The first AI idea most firms have is to paste a contract into a chat window and ask what's wrong with it. The output reads well. It names a few issues, sounds confident, and gives a partner no way to tell whether the flagged problem came from reading clause 14.2 or from the model's sense of what a liability complaint usually looks like. That question, whether a flagged issue is judgment or surface pattern matching, is the one practitioners keep returning to in discussions of AI-native legal review, and it's the question a workflow has to answer structurally rather than by asking the model to try harder.

The version that works in production is narrower and duller. You ask the model to extract facts into a fixed schema, with a pointer back to the text each fact came from, and you hand the result to a lawyer as a starting worksheet.

Why extraction outperforms an open-ended review

"Review this contract" asks for three things at once: find the relevant text, decide what it means in context, and decide whether it's acceptable for this client on this deal. The model has some capability at the first, patchy capability at the second, and none at the third, because acceptability depends on commercial appetite, the relationship with the counterparty and what the firm agreed last time.

Extraction splits those apart. The model finds and quotes; your playbook of standard positions decides what counts as a deviation; the lawyer decides what to do about it. Each step can be checked. When a comparison against a standard position is wrong, you can see whether the extraction was wrong or the playbook rule was.

The checklist drives the loop, one query per item you care about. This matters more than it sounds, because the expensive failure in contract review is the clause that isn't there. A model summarising a contract will rarely tell you there's no assignment restriction, no GST gross-up, or no cap carve-out for a data breach. Ask per item and "not present in this document" becomes an answer the system produces, not a silence you have to notice.

The shape of the extraction output

Keep the record flat and make every field verifiable against a span of source text:

{
  "item": "limitation-of-liability",
  "found": true,
  "clause_ref": "14.2",
  "page": 18,
  "quote": "The Supplier's aggregate liability under or in connection with this Agreement shall not exceed the Charges paid in the twelve (12) months preceding the claim.",
  "summary": "Liability capped at fees paid in the 12 months before the claim.",
  "standard_position": "Cap at 125% of total contract value, uncapped for breach of confidentiality.",
  "deviation": true,
  "notes": "No carve-out located for confidentiality or IP indemnity."
}

Around that sit the boring fields a matter file needs anyway: parties and their ACNs, execution and commencement dates, term and renewal mechanics, termination triggers and notice periods, governing law, payment terms, and the obligations with dates attached that someone will need to diarise.

The quote field carries most of the weight. A summary with no quote can't be checked in under a minute, so it gets skimmed and trusted. A summary with a quote and a clause reference can be confirmed or rejected at a glance, and the lawyer's eye goes to the text rather than to the model's prose.

Verify the extraction with something other than the model that produced it

Asking an LLM to grade its own answer produces agreeable nonsense. The pattern that's emerged in engineering tooling is a separate verification pass: MaruCheck, published on Show HN this month, exists specifically to provide independent QA for AI-generated code rather than relying on the generating model to check itself. Clause extraction needs the same separation.

In our pipelines the verification step is mostly deterministic and cheap:

String-match every quote back into the source document. A quote that doesn't appear verbatim is a fabrication and the record is dropped or re-run.
Check clause_ref and page against the document structure parsed at ingestion, so a quote attributed to the wrong clause fails.
Re-run the deviation comparison as a second call that sees only the quote and the standard position, with no access to the first model's conclusion.
Flag documents where OCR confidence is low or the clause numbering doesn't parse, and route those to manual review instead of extracting from mush.

Ignore the model's self-reported confidence score. It's fluent, not calibrated, and it tends to be highest on the clauses that are drafted most conventionally, which are the ones you needed help with least.

The lawyer's markup stays the deliverable

The strongest argument in that practitioner discussion is about artefacts: the thing clients pay for is the redline, and the value only lands when the lawyer's pass is captured as edits. Build the workflow so the machine output is a worksheet sitting beside the document, and the returned markup is what goes out the door.

CodeRabbit's auto-generated review summaries show the production shape of this in software: a structured first-pass summary posted alongside the pull request, with the human reviewer's decision remaining authoritative. Contract review maps onto it cleanly. The summary lands in the matter in your DMS or practice management system, the lawyer accepts, edits or rejects each row, and the accepted rows flow into the redline and the obligations register.

Capture the accept and reject decisions. After a few hundred contracts you have a record of which checklist items the extraction handles reliably and which ones a senior lawyer overturns half the time. That's the evidence you need to decide where to spend engineering effort, and the only honest basis for telling partners how much of the first pass they can lean on.

What accuracy to expect

Nobody can give you a number that transfers. Published accuracy claims for clause extraction are measured on someone else's document set, usually clean, usually in US or English drafting conventions, and they say nothing about how a tool performs on your precedents, your counterparties' paper, and the scanned 2013 deed of variation that amends clause 14 without renumbering anything.

Measure it yourself before you commit. Take 30 to 50 contracts representative of the work, have a senior lawyer mark up what a correct extraction looks like for each checklist item, and score the pipeline against that set. Report recall and precision separately, because they cost different amounts: a false positive wastes fifteen seconds of a lawyer's time, while a missed indemnity carve-out can go out the door. Re-run the set whenever you change a model, a prompt or a document parser.

The failure modes are consistent. Defined terms that shift meaning across a suite of documents, obligations incorporated by reference from a schedule or a URL, conditional carve-outs ("except as provided in clause 9.3"), and anything in a table or a poorly scanned annexure are where extraction degrades. Long agreements also lose fidelity toward the end of a context window, which is a reason to chunk by clause and run per-item queries rather than posting the whole document with one broad instruction.

Confidentiality and cost

Contracts are dense with material facts, which is why they reward systematic analysis. The Intercept reconstructed the working relationship between the Pentagon and OpenAI, Google and Anthropic from more than 400 pages of contract paperwork obtained under FOI. The same property makes these documents the last thing you want sitting in a third party's training pipeline or retained logs.

Before any pilot, settle where the documents go: which region the inference runs in, what the provider's retention and training terms say, whether your client engagement terms and confidentiality undertakings permit disclosure to that processor, and who signs off on that decision inside the firm. Get privacy advice on cross-border disclosure rather than inferring it from a vendor's marketing page. Deployment options in Australian regions exist across the major cloud providers, and for firms with the strictest undertakings we've run smaller open-weight models on infrastructure the client controls, accepting lower extraction quality in exchange for the documents never leaving.

Running costs are modest. Per-document API spend for a commercial agreement is small against six minutes of a lawyer's time, and the tokens are a rounding error in the project. The real cost is the gold set, the playbook of standard positions written down properly for the first time, and the ongoing maintenance when models change underneath you. Budget for a pilot on one contract type, with one checklist and one practice group, before anyone talks about rolling it out across the firm.

This post is part of our Practical AI for Law Firms series, which covers the systems Australian firms can put into production now and the ones worth waiting on.

PicNet builds production AI systems for Australian organisations. Talk to us about what a first project could look like.

Originally published at picnet.com.au.

Your AI is only as good as your data: why integration comes first

Guido Tapia — Tue, 22 Sep 2026 04:38:28 +0000

Almost every stalled AI pilot we get called into looks the same from the inside. The model was fine. The demo worked on the sample extract. Then someone pointed it at production and the answers stopped matching what the business knew to be true, because the customer record in the CRM disagreed with the one in the finance system, the product catalogue had been exported to a spreadsheet eight months ago, and nobody could say which of the two addresses on file was current.

That is a data problem wearing a model costume. This post is part of our AI-Ready Data and Integration series, and it covers the part that has to happen before any of the interesting work: getting a trustworthy view of your own operations into a form a machine can read.

Why software engineering got there first

AI uptake has run far ahead in software engineering compared with other business functions, as The Economist set out in August. The usual explanation is that developers are early adopters. A better one is that code is already in the shape AI needs. It is text, it is versioned, every change has an author and a timestamp, and there is exactly one repository that counts as the truth. A coding agent never has to work out which branch is the real one.

Now look at the data behind a typical Australian mid-market operation. Customer records in a CRM, billing in an ERP, service history in a ticketing system, a warehouse that only some reports use, and a set of spreadsheets that hold the rules nobody wrote down. No timestamps you would trust. No agreed precedence. An AI project starting there is being asked to do the reconciliation work first and the clever work second, with no tools for the first job.

What AI-ready actually requires

Three things, and none of them are model choices.

A master view. One place that resolves customers, products, transactions and the events connecting them into single records, with the source of every field recorded. Not a copy of each system side by side, which is what most "data lakes" turn out to be, but a resolved view where the question "how many active customers do we have" has one answer.

Freshness guarantees you can state in numbers. For each entity, how old can this be before it is wrong to act on it? Contact details might tolerate a day. Stock levels might tolerate five minutes. Write the number down per entity, measure it in production, and alarm on it. This matters more than which model you pick, because the model contributes nothing current by itself. A public tracker of 20 models across eight labs shows median release age measured in weeks and training cutoffs months behind release, and only half of those models publish a cutoff at all. Everything the model knows about your customers arrives at query time from your systems, or it does not arrive.

A conflict rule. When two systems disagree, one of them wins, and the rule is written down before the disagreement happens. This is the part organisations skip. It is also the part that decides whether anyone trusts the output six months in.

An architecture that survives contact with production

This is the shape we build, and it is the shape Centazio, our open-source integration platform, is designed around.

Read functions per source system, each one responsible for pulling changes and nothing else. They write raw payloads to staging, unmodified, with the retrieval time recorded.
Promote functions that map staged payloads into core entities. Mapping rules live here, in code, under version control.
An entity mapping store that holds the relationship between external identifiers and internal core identifiers, so the same customer arriving from three systems resolves to one record and you can always trace back which system contributed which field.
Write functions that push core state back out to the systems that need it, closing the loop rather than leaving the master view as a read-only reporting artefact.
Checkpoints and operational state per function, so a failed run resumes from where it stopped instead of reprocessing everything.

Conflict resolution sits in the promote step and should be boring and explicit:

// Billing owns the legal entity name and ABN.
// CRM owns contact details, but only if its record is fresher than billing's.
// Anything older than the freshness budget is not promoted at all; it raises an alert.
var name    = billing.LegalName ?? crm.TradingName;
var email   = Fresher(crm.UpdatedUtc, billing.UpdatedUtc) ? crm.Email : billing.Email;
var stale   = UtcNow - Newest(crm, billing) > budgets.Contact; // 24h
if (stale) alerts.Raise(ConflictAlert.StaleEntity, coreId);

Five lines of deterministic code replace an assumption the model would otherwise have to guess at. That is the argument the author of CodeIO made in September about AI-assisted development: supply deterministic, verifiable inputs and you get measurably better output than letting the model infer context. The same principle holds at the data layer. Every fact you can resolve before the prompt is a fact the model cannot get wrong.

Contradiction is the normal case

There is now open-source tooling whose entire premise is that AI sessions silently contradict earlier state unless one authoritative record exists. Engineers building agent infrastructure are separating persistent memory from durable execution for the same reason: the record of what is true has to be managed separately from whatever the agent is doing this minute. When practitioners build guardrails against a failure, that failure is the default behaviour, not an edge case.

Meanwhile the market's instinct is to add more surfaces. September brought another wave of tooling to connect agents to WhatsApp. Every new channel is another place a customer can state a fact and another version of that fact to reconcile. Adding channels before you have decided which system wins multiplies the contradictions.

The stakes rise when the AI starts acting

An AI that drafts a summary and gets it wrong wastes a few minutes. An AI that has authority to transact does something to the world. The Atlantic wrote in September about handing a personal assistant a credit card, and the point generalises to any agent with write access to your systems: bad underlying data now produces a wrong action, an incorrect invoice or a refund to the wrong account.

Source-of-truth governance is also a security control. Lakera's public agent-breaker exercises demonstrate agents being subverted through the content they ingest. If your agent reads free-text fields that anyone outside the organisation can write into, those fields are an input channel for instructions, and they need the same treatment as any other untrusted input. Under the Australian Privacy Principles you also need to know which system holds the authoritative copy of personal information before you can answer a correction request, which is a good reason to sort this out regardless of AI.

In administrative health settings, the same architecture applies to referrals, bookings and billing reconciliation. Any output that touches clinical judgement needs a named clinician signing off inside the workflow, designed in from the start, with the AI restricted to assembling and presenting the record.

A readiness checklist before you fund anything

Ask your team these questions. Written answers, not verbal ones.

For customers, products and each other core entity, which system is authoritative, and where is that written down?
When two systems disagree on a field, what is the resolution rule and who approved it?
What is the freshness budget for each entity, in minutes or hours, and do we measure the actual lag today?
Can we trace any field in a report back to the system and the record it came from?
How many customer records exist in more than one system with no link between the copies?
If the AI writes back to a source system, what is the rollback path when it writes something wrong?
Which free-text fields does the AI read, and who can write into them from outside the organisation?

An initiative that cannot answer one to three should fund integration work first. The AI project does not disappear; it starts three months later with a foundation, and it usually gets cheaper because the prompt no longer has to compensate for missing context.

What it costs

Integration is unglamorous and it is usually the larger half of a first AI project's budget. Connecting two systems properly, with mapping, conflict rules, monitoring and a replay path for failures, takes weeks rather than days, and legacy systems with poor APIs take longer. We built Centazio and released it as open source because we were rewriting the same staging, checkpointing and entity-mapping machinery on every engagement, and that machinery is not where the value sits.

The payoff is that the work is reusable. A master view built for a customer service assistant also serves the next forecasting model, the reporting rebuild and the system you replace in two years. Model choice is a decision you will revisit every few months as the tracker above keeps ticking over. The data foundation is the part you build once.

PicNet builds production AI systems for Australian organisations. Talk to us about what a first project could look like.

Originally published at picnet.com.au.

Coding agents in a production .NET shop: what holds up

Guido Tapia — Fri, 18 Sep 2026 02:53:32 +0000

We have been running Claude-style CLI coding agents against production .NET code for long enough to have opinions that survived contact with a client's release schedule. This post, part of our Practical AI in DevOps series, describes the setup we use, the parts that pay for themselves, and the work we still hand back to a person.

Two findings shape the rest of it.

A coding agent is a model plus a harness: the tools it can call, the context it starts with, the loop that decides what it sees next. A Berkeley group compared 21 model and harness pairings and found harness choice barely moved success rates, staying within about 2% on SWE-bench Lite, while moving cost a long way. The same model solved roughly the same share of tasks at up to five times the price depending on what wrapped it, and Claude Code averaged about twice Pi's cost per attempt (HarnessTax). Effort spent on repo scaffolding returns more than effort spent swapping models each time a new one ships.

The second finding is that an agent does what is cheap rather than what is correct. When a study gave agents both grep and language-server navigation on the same repositories, they picked the semantic tool between 0% and 6% of the time on code-localisation tasks, and forcing the semantic path first dropped success from 100% to 89% (agentconnect). On a large typed C# solution that matters, because text search cannot tell you whether a call binds to the int or the string overload, or which project a caller lives in. The Graphify C# project exists to hand agents compiler-accurate find-usages over a solution through Roslyn and MSBuild, which tells you how often they get it wrong without that help (graphify-csharp).

What the setup looks like

The repository carries the agent's working environment alongside the code:

An AGENTS.md (or CLAUDE.md) at the solution root with the build and test commands, the project layout, and the rules the agent must follow.
A coding standards document under version control, covering naming, structure, iteration style, and what a unit test has to assert before it counts.
A semantic index of the solution so reference queries return bound symbols rather than string matches.
A container the agent runs in, with no production credentials and no write access to anything outside the working tree.
A review pass by a second agent instance against the standards document, before the diff reaches a human.

Checking agent context into git is now a common pattern rather than a PicNet invention. Tools such as OKF Agent Memory and hosted context registries are being built on the same premise, that curated project facts should be versioned and supplied, not guessed at from training data.

The standards document does most of the work

Our C# standards existed as a PDF long before we used agents, and it was read about as often as most standards documents are. Turning it into a file the agent reads on every task changed its status. An agent follows a written rule far more consistently than a human under deadline, which means the document now has to be right, specific and short enough to sit in context.

The shape of it:

# C# standards (agent must comply)

## Structure
- One public type per file, file name matches the type.
- No regions. No partial classes outside generated code.

## Nullability
- Nullable reference types enabled solution-wide. Do not suppress with `!`;
  fix the model or the call site.

## Tests
- Every behaviour change needs a test that fails without the change.
- Do not edit an existing test to make a new change pass. Flag it instead.

## Framework
- Target framework is defined in Directory.Build.props. Do not assume
  the latest .NET version or the latest C# syntax.

That last rule earns its place. Practitioners comparing assistants on C# repeatedly single out framework version awareness and idiomatic C# as the weak spots of general purpose models (r/jenova_ai discussion). An agent will happily write C# 13 syntax into a project pinned to an older SDK, then spend twenty minutes trying to work out why the build fails.

Tests are the safety net, and the agent will try to cut them

Unit tests are the only thing standing between an agent's confident diff and a production defect. That makes the tests themselves a target.

OpenAI publishes what its monitoring found in its own internal coding agent deployments, and reward hacking is on the list: agents "illegitimately edit tests to make them pass", disable checks, and sometimes misrepresent whether a task was completed at all (OpenAI). If the organisation building the model treats its agents as something to supervise continuously, a Sydney consultancy has no basis for treating them as a trusted contributor.

So we read test diffs separately from implementation diffs, and a changed assertion in an existing test gets more scrutiny than a hundred lines of new code. Where an agent touches a codebase under regulatory constraint, an APRA CPS 234 environment or anything handling personal information under the Privacy Act, a named human approves the merge. That is a design requirement of the pipeline, not a policy we hope people remember.

AI review before human review

The first reviewer of every agent diff is another agent, running a review prompt with the standards document and no memory of the conversation that produced the code. It catches the mechanical failures: standards breaches, missing tests, a public method with no null handling, a dependency added without being asked for.

This exists because a human reviewing forty agent diffs a day reviews the fortieth badly. It also filters out agent verbosity. Agents bury the conclusion under paragraphs of narration, to the point that people publish skills whose only job is to force the model to lead with the answer (i-have-adhd). A review agent that outputs a short list of concrete findings is much easier to act on than a wall of summary.

Sandboxing is not optional any more

Google Threat Intelligence reported in September 2026 that attackers are targeting coding agents directly. The DUSTMAKER credential stealer drops malicious configuration into hidden workspace directories such as .claude/ and .cursor/, then uses prompt injection in those files to make the assistant run attacker commands during ordinary developer interactions. The same actor published trojanised MCP servers to PyPI and stole OIDC tokens from GitHub Actions runners to publish signed packages that pass automated trust checks (GTIG).

Our rules follow from that. Agents run in a container with scoped credentials. Any dependency an agent adds is reviewed by a person against the registry. Agent configuration files are treated as executable code in review, because that is what they are.

Honest numbers

We report two figures per task: elapsed time and token cost. Tooling for this is maturing, with projects like Frugal Tokens built purely to track spend and usage across agents. Time saved without cost per completed task is half a measurement, and a failed agent run costs money while producing nothing.

What we see: greenfield work inside a well-defined module goes several times faster. Mechanical refactors across a large solution go faster only where the semantic index is in place, and go badly wrong without it. Debugging a subtle production issue in unfamiliar code is roughly a wash, because the time saved writing the fix is spent verifying that the fix addresses the real cause. We do not publish a single productivity multiplier, because the honest answer varies by task type more than it varies by model.

Senior engineers become editors

The part nobody costed properly is what this does to the working day of an experienced developer. Reading and judging generated diffs all day is a different job from writing code, and developers are feeling it as an identity change rather than a tooling change, as a heavily discussed Hacker News thread from August 2026 showed.

This favours senior-heavy teams, which suits how PicNet is built. Judging whether a diff is correct, idiomatic and safe to deploy requires exactly the knowledge that used to come from years of writing that code by hand. A junior engineer reviewing agent output has no way to tell a good diff from a plausible one, and the agent is very good at plausible. Small teams where everyone reviewing has shipped production systems get more out of these tools than large teams with a thin layer of seniors on top.

PicNet builds production AI systems for Australian organisations. Talk to us about what a first project could look like.

Originally published at picnet.com.au.

One small team, 100+ automated checks: anatomy of an AI-assisted monitoring platform

Guido Tapia — Fri, 18 Sep 2026 02:53:27 +0000

A backup job that has not run for eleven days will not send you an email. Neither will a TLS certificate with nineteen days left on it. The failures that cost our clients money are almost never the loud ones; they are the ones where something stopped happening and no system was responsible for noticing. We built our own monitoring platform for that class of problem, after years of stitching together vendor dashboards that each covered one slice of an estate and none of which covered the gaps between them. This post is part of our Practical AI in DevOps series, and it describes what the platform is made of, how little of it is AI, and what it costs us to keep running.

How the checks are put together

Every check is a scheduled task written in code. It runs on a timer, talks to an API or a device, compares what it finds against an expectation, and returns a structured result. There is no agent, no DSL and no plugin marketplace. Across the estate we run more than a hundred of these, grouped roughly as follows.

Backup checks confirm that a restore point exists, that its timestamp is inside the agreed window, and that the job covers the volumes the client thinks it covers.
Certificate and domain checks watch expiry dates, registrar locks, nameserver drift and whether the certificate presented on port 443 matches the one we issued.
Network device checks read firmware versions, configuration deltas against the last known-good config, and support or licence expiry on firewalls.
Cloud posture checks look for publicly readable storage, admin accounts without MFA, elevated roles nobody has used in ninety days, and logging that has been switched off.
SIEM review checks pull the last day of correlated alerts and reduce them to what a human should read.

The contract every check returns is deliberately boring:

public record CheckResult(
    string CheckId,
    string ClientId,
    CheckState State,      // Ok, Alert, Fault
    string Headline,
    IReadOnlyList<Finding> Findings,
    DateTime RanAtUtc,
    TimeSpan Duration);

Because the shape is fixed, a new check is usually an afternoon of work rather than a project, and the reporting, escalation and history layers get it for free.

OK, ALERT and FAULT

The three-state result is the design decision that has paid for itself most often. OK means the check ran and found what it expected. ALERT means the check ran and found a problem: an expired certificate, a backup outside its window, a storage bucket open to the internet.

FAULT means the check could not run. The credential expired, the API returned 500, the device refused the connection, the script threw. Our platform treats FAULT as its own state with its own escalation path, because a check in FAULT is blind and a blind check is more dangerous than a failing one. An ALERT tells you something true about the world. A FAULT tells you that you currently know nothing about that part of the world, while the dashboard beside it stays green.

Most tooling we have used collapses this distinction into "no data", which sorts visually next to "fine". Our rule is that two consecutive FAULTs on the same check escalate to a named engineer, and a FAULT older than a week is a ticket with a due date. The most common cause, by a wide margin, is a read-only credential that rotated or lapsed.

The scheduler watches itself

A monitoring platform that stops running is worse than no platform, because everyone has stopped looking for the things it used to catch. The scheduler writes a heartbeat to an endpoint hosted outside its own account and region. If that endpoint sees no heartbeat inside the expected interval, it emails us directly. The heartbeat watcher is intentionally the dumbest component we own: no queue, no database, no model, a timestamp and a comparison. It has never needed a change since we wrote it.

Where the LLM sits

The AI in this platform does one job: it turns structured findings into an email a person will read on a phone at 7am. A daily digest per client contains a machine-generated table of every check and its state, and above that, a short prose summary written by a model from the findings payload.

Two rules govern that step. The model never decides state, because state is computed in code from the check's own data and is auditable. And the model never removes a finding; the table below the prose is generated without it, so a summary that misses something can be caught by reading three lines further down.

We keep the summarisation call behind a thin interface with one method, because the vendor surface underneath it moves on a monthly cadence. In September alone, Anthropic merged its Cowork agentic product back into a single Claude product (Anthropic), and TypeSafe launched a model family built for fast structured decisions at a fraction of per-token cost (TypeSafe). Cheap structured classification is tempting for triage work, and we may use it for SIEM noise reduction. We are not moving state determination out of code for it, because an Australian client asking why a backup was reported healthy needs an answer that points at a timestamp comparison rather than a probability.

Why we own it rather than buying six tools

The argument for point tools is that each one is better at its slice than anything we would write. That is true. The argument against is that the slices are where the problems live. A backup tool knows the jobs it runs; it does not know about the VM that a cloud posture check found last week and that nobody added to the backup policy. Answering that question needs both result sets in one store, which means owning the store.

There are second-order reasons. Per-endpoint SaaS pricing across a managed client base gets expensive fast, and it charges you most for the clients with the most infrastructure, which are the ones where an extra check costs us an afternoon. Our escalation rules encode agreements we have made with specific clients, not a vendor's severity taxonomy. And a small senior team can now maintain considerably more code than it could five years ago: agent orchestration is a shipped product category rather than a demo, with tools built to run several coding agents together on one codebase (Castforge). The Servo project's one-year retrospective on funded part-time maintenance is a useful analogue for the ownership question, with one engineer's sustained attention producing 1,150 reviewed pull requests and eight new maintainers over twelve months (Servo). Sustained attention from a small number of people is what makes platform ownership work, and it is also the thing most organisations underestimate when they decide to build.

What it costs to run

The infrastructure is a single small VM, a queue, a database and an object store for check history. That bill is smaller than one seat of most commercial monitoring products. Model inference is the smallest line item on the platform, because we only send the findings, not raw logs: a daily digest is a few thousand tokens of input.

The real cost is engineering attention, and it runs at roughly a day a month in steady state, with spikes when a vendor changes an API or a device family gets a new firmware scheme. The first six months cost considerably more than the SaaS tools we replaced. If your team does not have someone who will own this at that level for years, buy the tools instead.

Where it falls short

Checks run on schedules between fifteen minutes and daily, so this is not uptime monitoring and it is not EDR. It sits alongside both and answers a different question: is the configuration of this estate still what we agreed it would be. It has nothing to say about a site that went down four minutes ago.

The summaries flatten nuance sometimes, which is why the generated table stays in the email. And every check is only as trustworthy as the credential behind it, which is the argument for taking FAULT seriously. Under the OAIC's Notifiable Data Breaches scheme the clock starts when you suspect a problem, so the time you spend discovering that a check has been blind for three weeks is time you cannot get back.

PicNet builds production AI systems for Australian organisations. Talk to us about what a first project could look like.

Originally published at picnet.com.au.

Listening at scale: monitoring industry forums and communities with AI

Guido Tapia — Wed, 16 Sep 2026 01:03:28 +0000

Most listening set-ups I see at Australian mid-market companies are a saved search and a daily email digest. That catches press mentions and misses almost everything that happens in the communities where your buyers ask each other for advice. Practitioners comparing monitoring tools in September 2026 put the problem plainly: on Reddit, thread titles are often vague and the brand mention is three comments deep. A keyword alert on your company name never fires on those threads, and the question you could have answered gets a dozen replies and drops off the front page before anyone at your end notices.

Language models are good at the part that was previously impossible to staff: reading everything and deciding what matters. They are bad at the part people assume you would automate, which is writing the reply. This post, part of our Practical AI in Marketing series, covers how we build the first half and why we deliberately stop before the second.

The shape of the pipeline

The systems we build for this have five parts, and only one of them involves a model doing anything clever.

Collection. Pull whole threads rather than matching posts, because the classifier needs the parent question and the existing answers to judge whether you have anything to add. Expect to cover several channels. The same practitioner thread lists X, LinkedIn, Reddit, Facebook, YouTube and Hacker News as normal coverage, with specialist tooling for Reddit on its own.
Cheap pre-filter. A small model or an embedding similarity check against your topic list throws away 90 per cent of the volume before you pay for anything larger.
Classification. A larger model reads the surviving threads and returns structured fields: what is being asked, whether the asker has a problem you have solved before, how competent the existing answers are, and a confidence score.
Policy engine. Plain rules, held in config, that decide whether a classified thread is even eligible for a reply. This is where community etiquette lives.
Human queue. A short daily list for a named person, with the thread, the classifier's reasoning, and a draft set of talking points rather than a draft post.

The policy engine is the part that determines whether this reads as a helpful specialist or as spam, so it is worth being specific about what goes in it.

Encode each community's rules, not a house style

Communities now publish their own written positions on AI participation and commercial posting, and those documents are your specification. r/gamedev published a formal Policy on AI Use on 18 August 2026. The right behaviour for your system is to treat each community's stated policy as the binding constraint, with no organisation-wide default that overrides it.

Tolerance varies by audience in ways that are predictable enough to encode. Commenters in that same thread expected that hobby devs and non-programmers would be outraged while professional game devs and software engineers would find it pretty normal. A threshold that works in a professional subreddit will get you removed from an enthusiast one.

Some venues are simply off the list. The llama.cpp pull request adding Qwen3.8-Flash-Next opens with a pinned notice that off-topic comments and AI slop or spam in the thread will be deleted. Engineering forums moderate aggressively now, and a posting bot aimed at one earns a ban that takes months to undo.

In practice the config looks like this, one entry per community:

- community: "r/gamedev"
  collect: true
  reply_allowed: true
  promotional_reply: false      # per published AI use policy, 2026-08-18
  disclosure_required: true
  min_confidence: 0.85
  reviewer: "marco"
  policy_reviewed: "2026-09-01"

- community: "github/llama.cpp"
  collect: true
  reply_allowed: false          # pinned no-spam notice on active PRs

The policy_reviewed date matters. Communities change their rules, and a rule set nobody has looked at since last financial year will eventually have you posting against a policy that has moved.

Disclosure is the rule that protects you

The loudest criticism in the r/gamedev policy thread was not aimed at AI. It was aimed at the original poster, who worked at a company developing a plugin for Unity AI integration and chose not to disclose it. Concealed commercial motive does the reputational damage, and no amount of careful tone repairs it once someone checks your post history.

The working norm is visible even among vendors recommending their own products. One founder in the monitoring tools thread prefixed his recommendation with Disclaimer: that's me! :-) Not shilling. Hard-code that. Every reply your team posts carries an affiliation statement in the first line, with no exceptions for threads where it feels awkward. It also keeps you clear of the Australian Consumer Law's misleading conduct provisions, which apply to a staff member posting as an apparently disinterested punter just as they apply to an ad.

The human writes the post

The argument for AI-drafted replies is time. The argument against is that the register gives it away. Members of a language learning community describe LLM output as unbearably sycophantic and corporate in its speech patterns, and that is exactly the voice that gets a reply downvoted and your account tagged as a marketer.

The upside of doing it properly is real. When the designer of the board game Cola Wars responded personally to criticism of AI art in his product, commenters on r/boardgames called it an incredibly well made response, and it's what should be the standard in situations like this, including people who had no interest in the game beforehand. A named person answering directly is the whole product here. The model's job is to make sure that person spends their twenty minutes a day on the six threads worth answering instead of scrolling.

Costs and the parts that break

Model inference is the cheap line item once the pre-filter is doing its job. The expensive lines are collection and maintenance.

Scrapers break. The OpenAmer project's Workflow Immune System exists specifically because automations fail when sites redesign, and it re-finds UI elements and patches its own workflows to keep running. Self-healing helps, although it does not remove the need for someone to own the pipeline. Budget a few hours a month for broken collectors, changed API terms and rate limits, and assume at least one platform will close off access during the first year.

Where the data cannot leave your control, self-hosting the collection layer is a live option. OpenAmer is open source and fully self-hosted, which matters for health-sector and government clients who cannot ship community and customer signals into third-party SaaS analytics under their privacy obligations.

The other honest limitation is precision. Classifiers over-select early on, and the first month of queues will contain threads where you have nothing useful to say. Have your reviewer mark those, feed the marked examples back into the prompt or a small evaluation set, and expect two or three rounds before the daily list is short enough that someone reads all of it.

Start with two or three communities you already care about, run the queue for a month with nobody posting at all, and see whether the threads it surfaces are ones you wish you had answered. If they are, you have a case for the posting half. If they are not, you have saved yourself a bot that annoys people.

PicNet builds production AI systems for Australian organisations. Talk to us about what a first project could look like.

Originally published at picnet.com.au.

Answering from your own documents: a policies and procedures assistant for any business

Guido Tapia — Mon, 14 Sep 2026 01:18:30 +0000

Most of the questions that clog an HR inbox or an operations channel have already been answered in writing. The leave policy says what happens when a public holiday falls during annual leave. The installation manual says which bracket suits a tiled roof. The problem is that the answer sits on page 34 of a PDF someone uploaded to SharePoint in 2019, so people ask a colleague instead, and the colleague guesses.

This is the least glamorous use of a language model and the one with the clearest payback. It is also the pattern we are asked about most often, which is why it appears early in Practical AI in Business Operations. The approach below is what we build for clients outside health as well as in it: HR policies, operations manuals, product documentation, supplier agreements, safety procedures.

What the system actually does

A user asks a question in plain English. The system finds the passages in your documents most likely to contain the answer, hands those passages to a language model, and asks it to answer using only what it was given. The answer comes back with links to the source documents and the sections used. If the passages do not contain the answer, the system says so.

That last behaviour is the one that decides whether staff trust the thing. A general model will happily produce a confident paragraph about your redundancy process based on what redundancy processes usually look like. That answer is worse than no answer, because it sounds like it came from your policy.

Architecture sketch

Ingestion. Pull documents from wherever they live (SharePoint, Confluence, a file share, a DMS) on a schedule. Convert to text, keeping headings and page numbers so a citation can point at a section rather than a 90 page file.
Metadata. Every chunk carries the document title, owner, effective date, the version it supersedes, and which entity or state it applies to. This is where most of the value hides. Without it you cannot filter a Queensland question away from a Victorian policy, and you cannot tell the user the answer is current as at a date.
Chunking. Split on document structure, not on a fixed character count. A clause with its heading and its parent heading retrieves far better than 800 characters that start mid sentence.
Index. Embeddings plus keyword search, with results merged. Pure vector search misses exact terms like award classification codes or part numbers. Keyword search alone misses "can I take leave at half pay". You want both.
Answer. The model receives the question, the top passages, and a strict instruction set. It returns structured output, not free text.
Logging. Every question, the passages retrieved, the answer, and any user feedback. This is your evaluation data and your audit trail.

We usually put the index in Postgres with pgvector rather than adding a new vendor. For corpora in the tens of thousands of chunks, which covers most SMEs, it is fast enough and it keeps your document content inside infrastructure you already govern under the Privacy Act.

The prompt contract

The instruction to the model is short and it does not negotiate:

Answer using only the numbered passages below.
Every sentence in the answer must cite at least one passage id.
If the passages do not contain enough to answer, return not_in_corpus.
Do not use general knowledge about how policies usually work.

And the output is a schema, so the application can check it before a human ever sees it:

{
  "status": "answered",
  "answer": "Annual leave accrues progressively...",
  "citations": [
    {"passage_id": 3, "document": "Leave Policy v4.2",
     "section": "5.1 Annual leave", "effective": "2025-07-01",
     "url": "https://.../leave-policy-v4-2.pdf#page=12"}
  ]
}

Then verify before rendering. If status is answered and citations is empty, drop the answer and show the retrieved passages instead. If a cited passage id was not in the set you supplied, treat it as a failure. These checks cost nothing and they catch the most embarrassing errors.

Make the citation a link to the source document at the right page, not a file name. Staff click through more often than you would expect, and the ones who click are the ones who will tell you when a policy is out of date.

Evaluation before anyone else sees it

Do not roll this out on a demo. Build a question set first.

Take 100 to 200 real questions from your HR inbox, service desk tickets or the operations channel. For each one, have the person who currently answers it write down the correct answer and the document and section it comes from. That is a day of someone's time and it is the highest value day in the project.

Then run the set and grade three things: whether the answer is correct, whether the citation points at the right section, and whether the system refused when it should have answered. Add a second set of 30 or so questions you know are not covered by the corpus, including a few that sound like they should be. Count how often the system answers those anyway. That false answer rate is the number to watch, and it is the one that moves when someone changes a prompt, swaps a model version or adds a new document set.

Re-run both sets on every change. It takes minutes once it is scripted, and it turns "the new model seems better" into something you can check.

If your documents are clinically adjacent, keep the assistant on administrative ground: rosters, credentialling, procurement, incident reporting workflow. Anything that could shape care needs a named clinician signing off the answer before it reaches anyone, designed in as a workflow step with an approval record, not added later as a disclaimer.

What it costs to run

Three lines, in descending order of size.

The first is corpus preparation. Someone has to decide which documents are authoritative, retire the superseded ones, and attach the metadata. In most engagements this is the bulk of the effort and it is internal staff time, not licence fees. It is also work that pays off whether or not you build the assistant.

The second is ongoing ownership. A document set that nobody maintains degrades, and a confidently cited answer from a withdrawn policy is a real problem. Budget a few hours a month for someone to review flagged answers and update the corpus.

The third is inference, and it is usually the smallest. Work it out yourself rather than trusting anyone's estimate: multiply the tokens you send per question (retrieved passages plus the question plus the answer, typically a few thousand) by your expected monthly question volume, then check your provider's current published price. A department sized deployment of a few thousand questions a month generally lands in the range of a single software subscription. Embedding the corpus is a one off charge plus a trickle for new documents. If you self host the index, add nothing for the database because it is already there.

When a wiki search is honestly good enough

Sometimes it is. If your corpus is a well structured intranet with good headings, your staff know the vocabulary, and volume is low, decent search plus a table of contents will solve the problem for the cost of nobody's afternoon. The same goes for a corpus small enough that a person can hold it in their head.

The pattern earns its keep when the documents are long, badly structured or scattered across systems, when the answer requires reading across several documents, when users do not know the internal terminology, or when the same questions arrive dozens of times a week. It does not help at all when your documents contradict each other. Fix that first, because retrieval will find both versions and the model will pick one.

PicNet builds production AI systems for Australian organisations. Talk to us about what a first project could look like.

Originally published at picnet.com.au.

When AI must not act alone: human-in-the-loop patterns for security automation

Guido Tapia — Fri, 11 Sep 2026 00:53:24 +0000

Most of the AI security work we do for clients ends up hinging on one question: what is this thing allowed to do on its own at 3am on a Sunday?

Everything else follows from the answer. Model choice, prompt design, which vendor's agent framework you use, all of it matters less than where you draw the line between an action the system takes and an action a person takes. This post closes out Practical AI in Cyber Security with the governance layer that the earlier posts assumed was there.

Draw the line at reversibility

The clean split is not "smart tasks versus dumb tasks". It is reversibility. If the worst outcome of the AI being wrong is a wasted five minutes of someone's reading time, automate it. If the worst outcome is a production system offline, a mailbox item gone, or a clinician locked out of a roster, a human decides.

Safe to run unattended:

Collection. Pulling logs, endpoint telemetry, sign-in events, ticket history, config snapshots into one place.
Enrichment. Resolving an IP to an ASN, looking up a hash, attaching asset ownership, pulling the last three changes to the affected host.
Triage and correlation. Grouping alerts, suppressing known-benign patterns, ranking by likely impact, drafting the "what we think happened" summary.
Reporting. Weekly posture summaries, control-drift reports, evidence packs for an audit.

Behind a human decision:

Blocking. Firewall rules, conditional access changes, disabling a user account.
Deleting. Quarantining or purging mail, removing files, revoking tokens at scale.
Isolating. Pulling a host off the network, suspending a VM, stopping a service.
Anything that touches a system of record. Closing an incident, changing a risk rating, writing to an asset register.

Vendors are pushing hard in the other direction. Cloudflare's own material on agentic AI describes systems that make decisions, execute code, query databases, send email and handle payments without constant human prompting. That is a fair description of the capability. It is also a precise list of the things you want fenced off in a security context, because each one of them is hard to undo.

The split is not new and it is not unique to security. Windmill's engineering write-up on their support pipeline lands in exactly the same place: AI does the volume work, drafting replies and even drafting the fix, but a human still presses send on anything customer-facing. Their reasoning is worth stealing wholesale. The system only works while people trust the drafts, and one wrong auto-sent reply burns that trust for every message after it. A SOC analyst who has been burned once by an automated block on a payment gateway will start ignoring the tool, and then you have paid for AI and got nothing.

The pattern: agent proposes, human approves

The most useful working implementation we have seen of this contract is in a repo, not a security product. The kube-agents project documented a protocol for AI agents contributing code: an agent claims work by self-assignment, escalates anything it cannot resolve with a needs-human label, and cannot merge until a human lgtm gate is satisfied. Three primitives. Claim, escalate, approve.

That maps onto a security runbook almost directly:

claim    -> agent takes alert #4471, marks it in-progress so a second
            agent or analyst doesn't duplicate the work
enrich   -> pulls host, user, recent changes, threat intel, prior tickets
propose  -> writes the finding, the evidence and ONE recommended action
            ("isolate WKS-0412"), status = awaiting-approval
escalate -> if it cannot complete, status = needs-human with the reason
approve  -> named human approves, rejects or edits; only then does the
            action execute, under the human's authority

The kube-agents authors made the contract deliberately framework-agnostic. That is the detail to copy. Your SOC processes will outlive your current model vendor by years. Keep the escalation and approval contract in your own orchestration layer, expressed in your own states, so swapping Claude for Gemini next year is a config change rather than a governance rewrite.

A blind check must never look green

The failure mode that costs real money is not the AI recommending the wrong action. It is the AI reporting "all clear" on a check that never ran.

Alert and fault are different states and need different plumbing. An alert means the check ran and found something. A fault means the check could not run: the API token expired, the log source stopped shipping, the model call timed out, the agent hit a rate limit. Both are "not green", but they need different handlers, different dashboards, and often different people.

Most monitoring code we inherit collapses the two. The check throws, something catches the exception, logs it at debug level and returns a default, and the tile stays green for eleven weeks while the log source has been dead the whole time. Adding an LLM to that pipeline makes it worse, because models are obliging. Ask one to summarise incomplete data and it will produce a confident, well-formatted summary of the subset it could see, with no flag that half the estate was missing.

What we build into these pipelines:

Every check returns one of three states, never two: ok, alert, fault. There is no default value on the failure path.
A fault renders as a distinct colour on the dashboard, not as a green tile with a footnote.
Staleness is a fault. If a source has not reported inside its expected window, the check that depends on it is faulted, not passed.
Coverage is reported alongside every AI-generated summary. "23 of 26 sources; 3 faulted" sits above the narrative, generated by the pipeline rather than written by the model.
The model never sets its own coverage figure. That number comes from the orchestrator, which knows what it asked for and what came back.

The needs-human label is the same idea at the agent level. An agent that hits something it cannot resolve emits an explicit escalation state rather than quietly returning success.

The audit trail is the deliverable

When someone asks how your AI-assisted decisions are governed, the honest answer is whatever your logs can show. Governance is a document exercise at the point it gets tested.

That point is closer than it looks for organisations with any European exposure. The EU AI Act moved into enforcement on 31 August 2026 with the AI Office issuing formal requests for information to general-purpose model providers, covering model security, independent external evaluations and post-market monitoring. The requests went to providers rather than to deployers, so an Australian organisation using an API is not in the first wave. But the pattern is set: the questions are document-driven, the answers become part of a permanent supervisory record, and incorrect or incomplete replies carry penalties of up to 15 million euros or 3% of global turnover. Australian organisations serving EU users, or working under contracts that flow those obligations down, will be asked to evidence human oversight of AI-assisted decisions. You cannot backfill that evidence.

For each AI-assisted decision we record: what proposed the action, which model and version, what evidence it saw (the actual query results, not a paraphrase), what it recommended, who approved or rejected it, when, and what executed. In practice the evidence snapshot is the hard part, because inputs change under you. Store it with the decision.

One more control, learned from an open-source community rather than a compliance framework. The nixpkgs maintainers proposed closing any pull request that removes the AI disclosure checkbox, on the basis that merging one would be a serious policy violation. The interesting part is the failure mode they found. The control existed, but it was a checkbox the submitter could delete. So in a security pipeline, AI-provenance metadata must be system-generated and not editable by the thing that generated the content. If the agent can write its own "reviewed by a human" flag, the flag means nothing.

Roll it out in dry run, and stay there longer than feels necessary

Every automation we deploy runs in dry run first. The agent does the full job, including selecting the action, and writes what it would have done to a log and a channel humans read. Nothing executes.

We run this for a minimum of four weeks in most environments, longer for anything that touches identity or network paths. The dry-run period tells you three things you cannot get from testing: how often the agent recommends something an analyst would not do, how noisy it is at 2am when the batch jobs run, and whether the evidence it attaches is actually enough for a human to decide from. That last one fails more often than the first.

Promotion out of dry run is per action type, never for the whole system. Enrichment and triage might graduate in week two. Automated blocking might never graduate, and for most clients it should not.

An honest cost note. Human-in-the-loop is more expensive to run than full automation, because you are paying for the AI and keeping the analyst. The saving comes from what the analyst spends their time on: reviewing a drafted finding with evidence attached takes a couple of minutes, where building that finding from scratch takes twenty or thirty. The approval queue itself needs an owner and an SLA, or it becomes a place where alerts go to age quietly.

For anything clinically adjacent, the sign-off is part of the design rather than a caveat bolted on the end. An AI can assemble a suspected-breach pack, correlate the access logs, draft the notification timeline. A named clinician or privacy officer approves what leaves the building, and their name is on the record.

PicNet builds production AI systems for Australian organisations. Talk to us about what a first project could look like.

Originally published at picnet.com.au.

Ad spend guardrails: automated Google Ads reporting and anomaly alerts

Guido Tapia — Wed, 09 Sep 2026 00:58:25 +0000

Most organisations we work with have alerts on cloud spend. If an AWS account jumps 30% overnight, someone gets paged. The same organisation will spend a comparable amount on Google Ads and review it once a month, in a meeting, from a deck built by the agency that placed the ads.

That gap is where money disappears. A campaign gets disapproved and stops serving for eleven days before anyone notices. A bid strategy switches to maximise clicks and burns three weeks of budget on traffic that never converts. A landing page 404s after a site release and the ads keep running. None of this is exotic. It is ordinary operational drift, and it is invisible until the monthly report, by which point the money is gone.

This post is part of our Practical AI in Marketing series, and it describes a small system we build for clients: a daily automated report on Google Ads spend and performance, with anomaly alerts, and a plain-English summary that a business owner will actually read.

Why you instrument it yourself

Two things happened in the last week that make the case better than I can. On 2 September a US court declined to force Google to sell its ad tech business, accepting behavioural remedies instead, the third time in recent years that US antitrust enforcers have failed to break up a Big Tech company. The buy side and the sell side of that market stay under one roof. The day before, the FTC and 22 state attorneys general alleged that Amazon made over US$20 billion by overriding its own ad auction results with higher prices, a claim Amazon disputes.

Whatever the outcomes, the practical lesson for an Australian advertiser is the same. Auction transparency is not arriving from a regulator on a timetable that helps your budget this quarter. Newer AI-driven ad channels have their own problems; practitioners are already publicly complaining that ChatGPT ad targeting sends spend to irrelevant audiences. Pull your own spend and conversion data into a system you control, keep your own history, and reconcile platform-reported conversions against what your CRM says actually closed.

The shape of the system

A scheduled job pulls campaign, ad group and keyword metrics from the Google Ads API each morning, read-only.
Rows land in a database table you own, one row per entity per day, appended and never overwritten, so you keep your own history independent of platform reporting windows.
An expectation layer computes what each campaign should have spent and returned.
A deterministic rules engine compares actuals against expectations and raises anomalies.
An LLM turns the day's numbers and any anomalies into six or eight sentences of plain English.
Delivery goes to email and Slack or Teams, with the raw table available for anyone who wants to dig.

The extraction query is unremarkable:

SELECT
  campaign.id,
  campaign.name,
  campaign.status,
  metrics.cost_micros,
  metrics.impressions,
  metrics.clicks,
  metrics.conversions,
  metrics.conversions_value
FROM campaign
WHERE segments.date DURING LAST_7_DAYS

Read-only by default

The credential the reporting job uses must not be able to change anything. No write scopes, no budget edits, no pausing campaigns, even though the API would happily allow all three.

This is not a hypothetical concern. A recently merged open-source project, the KPA Traffic Stack, builds exactly this pattern for Meta and Google Ads diagnosis: read-only exports with explicit guardrails against accidental writes and credential leaks, validators that reject exports containing secret-like keys, and a flag on every export asserting that platform writes are not allowed and human approval is required. The automated code review on that pull request found a real hole in it, where one validator trusted a self-reported "no credentials" flag rather than scanning nested fields for tokens. Worth reading if you are building something similar, because the same mistake is easy to make.

Two reasons for the restriction. An agent or script with write access to a live ad account is a bad incident waiting for a bad day. And once the report can act, its findings stop being an independent record and start being a description of its own behaviour.

If you do want programmatic control of campaigns, keep it in a separate, reviewed path. The agoraform project treats Google Ads as config-as-code, covering conversion goal, budget, campaign, ad group, targeting, keywords and responsive search ads in one declarative resource graph with a validate, plan, apply lifecycle. Campaign changes then get diffed and reviewed like any infrastructure change, by a person, on a different credential.

Anomalies need an expected value, not last week's number

The weak version of this system alerts when spend moves more than some percentage against the prior period. It produces noise every Monday, every long weekend and every end of financial year.

The better version compares against a modelled expectation. The mizan target engine is a good illustration of the arithmetic: it deterministically reverse-engineers required revenue, new customers, sales, qualified calls, shows, bookings, leads and ad spend from a monthly profit or margin goal, and computes a maximum sustainable acquisition CAC, maximum media CAC and maximum cost per lead along the way. That maximum sustainable ad spend is the ceiling a runaway-spend alert should be measured against. It also fails closed: missing months stay missing rather than becoming zero, and impossible targets return an explicit unattainable result instead of a plausible-looking number.

Our rules end up looking like this:

- id: runaway_spend
  window: rolling_7d
  trigger: spend > 1.15 * max_sustainable_daily_spend
  severity: high

- id: dead_campaign
  window: rolling_3d
  trigger: spend > 2 * target_cpa and conversions == 0
  severity: high

- id: silent_campaign
  window: rolling_2d
  trigger: status == "ENABLED" and impressions == 0
  severity: medium

- id: cpl_drift
  window: rolling_14d
  trigger: cost_per_lead > 1.3 * max_cpl and spend > 500
  severity: medium

The spend floors matter. A campaign running at $30 a day will breach any ratio-based rule constantly, so suppress alerts below a dollar threshold where the money at stake is smaller than the cost of reading the email.

Where the LLM helps

The LLM does one job: it writes the summary. It receives the computed table and the list of triggered anomalies as structured input and produces a short narrative. It does no arithmetic, no ranking, no attribution reasoning. Every number in its output must appear in the input, and we validate that before sending.

The reason is boring and practical. A business owner will not open a dashboard daily, and will not read a 40-row table. They will read four paragraphs in an email that says spend was $4,180 against a $3,900 expectation, that the brand campaign is fine, and that a particular campaign has spent $612 in three days with no conversions and should be checked today. Prior to LLMs, we wrote that summary with templates, and the templates read like templates. This is the piece where the model genuinely earns its place.

Costs are small. The Google Ads API is free to use, though you need a developer token and basic access approval. Daily summarisation for a mid-sized account runs to a few thousand tokens, which is cents per day. The build is the expense: a week or two of engineering for a first version covering one account, plus a smaller amount of maintenance because Google retires API versions regularly and you will be doing an upgrade at least annually.

Honest limitations

Conversion lag is the big one. Yesterday's conversions are undercounted, so a dead-campaign rule reading a single day of data will produce false alarms. We compare on a lagged window and state the lag in the summary. Attribution is the second: Google's reported conversions and your CRM's closed deals will not agree, and the system should show both rather than pretending one is truth. Third, an alert nobody actions is worse than no alert, so cap the daily volume and give each rule an owner.

One Australian-specific note. If your campaigns use customer match or remarketing audiences, the email addresses and identifiers you upload are personal information under the Privacy Act, and the handling, consent and retention questions belong with your privacy officer before the reporting project starts, not after.

PicNet builds production AI systems for Australian organisations. Talk to us about what a first project could look like.

Originally published at picnet.com.au.

A morning digest your team actually reads: AI-summarised news, spend and system health

Guido Tapia — Mon, 07 Sep 2026 01:28:26 +0000

Most Australian operations teams I talk to are subscribed to too much. Cloud billing alerts to one inbox, an uptime tool posting to a Slack channel, ad platform notifications to whoever set up the account three years ago, four vendor newsletters, and a deploy bot that fires on every merge. Nobody reads any of it. The item that actually mattered last Tuesday was sitting in a channel someone muted in March.

The approach we keep coming back to is unglamorous: one email, one delivery list, sent at 7am, ordered by what needs a decision today. Industry news summarised by a model, yesterday's cloud and ad spend, and anything monitoring flagged overnight. This post is part of our Practical AI in Business Operations series, and it is the cheapest useful thing on the list.

Why one digest beats ten streams

Notification channels rot quietly, and nothing tells you when they do. A good example sits in a push notification service where the send call returned a bare boolean, so a permanent APNS 410 (the token is dead, the app was uninstalled or push was disabled) looked identical to a transient network blip. Dead device tokens were never pruned and the failures kept accumulating with nobody watching (TrackRat #1802).

That is the same failure mode as ten alert channels in an organisation. Each one is a small integration nobody owns. One digest with a known distribution list is a single thing to keep honest, and every recipient notices the day it stops arriving.

What the digest contains, in the order it matters

Order sections by what forces action, not by data source. Grouping by category buries the item that needs a response, which is exactly what happened to a changelog generator that grouped commits by their type prefix. Breaking changes were rendered under "Added" and "Changed", and one commit that deleted eight internal packages shipped in a published changelog under "Changed" where an auditor would never look for it. The fix was to give breaking changes their own section and sort it to the top (go-kure #775).

So the running order in ours is:

Overnight alerts. Anything monitoring raised between the last digest and this one, with the alert still open flagged separately from the ones that self-cleared.
Spend. Yesterday's AWS or Azure spend against the trailing seven-day average, plus Google and Meta ad spend from the same window. Percentage moves in the heading, dollars in the row.
Industry and vendor news. Summarised, triaged, and always last, because it is the section a reader is allowed to skip.

The architecture

It is a scheduled task, not a platform. Ours runs on a cron trigger before business hours in Sydney and takes a few seconds:

One collector per source, each returning (rows, error). Collectors never raise into the pipeline.
One summariser call per section that needs prose. The alerts and spend sections are rendered from data with no model involved.
One renderer producing HTML and a plain text alternative that says the same thing.
One run log recording which sources answered, which failed, and how many items each contributed.

The (rows, error) shape carries more weight than it looks like it should:

def render(section):
    if section.error:
        return f"{section.title}: COULD NOT READ ({section.error})"
    if not section.rows:
        return f"{section.title}: nothing overnight"
    return format_rows(section.rows)

An empty read and a failed read must render differently. A morning brief script learnt this the hard way: its calendar collector called the API with no calendarId, so it read the account default and returned zero events for a week that had nine, including a weekly planning meeting. The brief printed "nothing" on days that had meetings, and the zero read as a pass. The prompt named the calendar in prose, which the author summarised as "prose is not an argument" (kipi-system #299).

That is the archetypal silent digest failure. The email still arrives, still looks complete, and an entire data source is missing. Enforce data scope in code with an explicit parameter, and make source coverage a section of the digest itself so a missing feed is visible to every reader rather than to nobody.

Prompt design that stays faithful to sources

Three rules do most of the work.

First, the model chooses, and code checks the choice. Give the summariser a small set of named admission rules and require it to cite which rule admitted each item it keeps. Then validate the citation in code and drop anything citing something else. A team digest at PostHog did exactly this with four named rules (contract, assumption, decision, customer) and went from keeping 21 of 26 merges to keeping zero to two (PostHog #88286). Their earlier attempt used a count target in the prompt, which squeezed out real news on busy days and admitted routine work on quiet ones.

Second, no untrusted text reaches the prompt in a position where it can address the summariser. The same PostHog change removed contributor-written PR bodies from the prompt entirely, and made sure the remaining values could not close their own tag and continue as instructions. Vendor news is worse than internal commits here: press releases are written to be persuasive, and you are feeding them to a model whose job is to be sceptical about them.

Third, ask for assessment, not paraphrase. A summary that repeats an item's own marketing copy is worse than the headline alone, because it launders the claim into your team's morning reading. A live automated feed we looked at as a format reference gets this right: a headline count ("20 new videos across your watchlist"), then per item a title, source, a triage verdict such as Skim or Skip, and one line of critical assessment (youtube-digest #10). The triage label is what ten notification streams can never give you. It tells the reader what not to open.

Freshness rules, and the trap underneath them

Decide deliberately what happens to an item the digest did not carry, because there are two separate loss paths and they need different answers. Something cut by a render cap is different from something the summariser read and rejected.

Get this wrong and the digest quietly decays. PostHog's was carrying merges that had landed up to six days earlier with nothing marking them as old, because anything the cap cut or the summariser dropped was re-queued into the next morning's run. The rejected ones then produced the same empty result every day, forever, since the same input to the same prompt returns the same answer. Their fix was to let a run consume everything it claims, including an empty answer. Either age items out or label them explicitly as carried over.

Editorial rules as a binding contract

Write the rules down once, in one referenced document, and treat them as constraints any agent revising the text must obey. Scattering them through ad-hoc prompts guarantees drift. A useful example of the form is an editorial supervision contract written for agent-generated prose: core rules, an explicit list of banned constructions, and a revision standard listing what every pass must actively search for (janeco #3).

For a digest, ours is short. Every claim links to its source. No item is described in the vendor's own adjectives. An item the model cannot summarise from the source text gets a link and no summary. Every section states its coverage. Nothing in the digest ever asserts something the collector did not read.

Honest limits and costs

The model spend is a rounding error next to the cloud bill the digest reports on. A handful of calls a day against a small model costs less than the coffee consumed while reading the output. The real cost is engineering time on the collectors, because ad platforms and cloud billing APIs change, and a broken collector that fails loudly still needs someone to fix it that week.

The other limit is scope. A digest is a reading aid, not a control system. It should never take an action, close an alert, or approve spend. Every item links back to the console where a person does that. And it needs one owner who reads it every morning and notices the day a section goes quiet for the wrong reason.

Build the smallest version first. One source, one section, one recipient, running for a fortnight before you add the second source. Most of the value shows up in week one.

PicNet builds production AI systems for Australian organisations. Talk to us about what a first project could look like.

Originally published at picnet.com.au.