DEV Community: Guido Tapia

Listening at scale: monitoring industry forums and communities with AI

Guido Tapia — Wed, 16 Sep 2026 01:03:28 +0000

Most listening set-ups I see at Australian mid-market companies are a saved search and a daily email digest. That catches press mentions and misses almost everything that happens in the communities where your buyers ask each other for advice. Practitioners comparing monitoring tools in September 2026 put the problem plainly: on Reddit, thread titles are often vague and the brand mention is three comments deep. A keyword alert on your company name never fires on those threads, and the question you could have answered gets a dozen replies and drops off the front page before anyone at your end notices.

Language models are good at the part that was previously impossible to staff: reading everything and deciding what matters. They are bad at the part people assume you would automate, which is writing the reply. This post, part of our Practical AI in Marketing series, covers how we build the first half and why we deliberately stop before the second.

The shape of the pipeline

The systems we build for this have five parts, and only one of them involves a model doing anything clever.

Collection. Pull whole threads rather than matching posts, because the classifier needs the parent question and the existing answers to judge whether you have anything to add. Expect to cover several channels. The same practitioner thread lists X, LinkedIn, Reddit, Facebook, YouTube and Hacker News as normal coverage, with specialist tooling for Reddit on its own.
Cheap pre-filter. A small model or an embedding similarity check against your topic list throws away 90 per cent of the volume before you pay for anything larger.
Classification. A larger model reads the surviving threads and returns structured fields: what is being asked, whether the asker has a problem you have solved before, how competent the existing answers are, and a confidence score.
Policy engine. Plain rules, held in config, that decide whether a classified thread is even eligible for a reply. This is where community etiquette lives.
Human queue. A short daily list for a named person, with the thread, the classifier's reasoning, and a draft set of talking points rather than a draft post.

The policy engine is the part that determines whether this reads as a helpful specialist or as spam, so it is worth being specific about what goes in it.

Encode each community's rules, not a house style

Communities now publish their own written positions on AI participation and commercial posting, and those documents are your specification. r/gamedev published a formal Policy on AI Use on 18 August 2026. The right behaviour for your system is to treat each community's stated policy as the binding constraint, with no organisation-wide default that overrides it.

Tolerance varies by audience in ways that are predictable enough to encode. Commenters in that same thread expected that hobby devs and non-programmers would be outraged while professional game devs and software engineers would find it pretty normal. A threshold that works in a professional subreddit will get you removed from an enthusiast one.

Some venues are simply off the list. The llama.cpp pull request adding Qwen3.8-Flash-Next opens with a pinned notice that off-topic comments and AI slop or spam in the thread will be deleted. Engineering forums moderate aggressively now, and a posting bot aimed at one earns a ban that takes months to undo.

In practice the config looks like this, one entry per community:

- community: "r/gamedev"
  collect: true
  reply_allowed: true
  promotional_reply: false      # per published AI use policy, 2026-08-18
  disclosure_required: true
  min_confidence: 0.85
  reviewer: "marco"
  policy_reviewed: "2026-09-01"

- community: "github/llama.cpp"
  collect: true
  reply_allowed: false          # pinned no-spam notice on active PRs

The policy_reviewed date matters. Communities change their rules, and a rule set nobody has looked at since last financial year will eventually have you posting against a policy that has moved.

Disclosure is the rule that protects you

The loudest criticism in the r/gamedev policy thread was not aimed at AI. It was aimed at the original poster, who worked at a company developing a plugin for Unity AI integration and chose not to disclose it. Concealed commercial motive does the reputational damage, and no amount of careful tone repairs it once someone checks your post history.

The working norm is visible even among vendors recommending their own products. One founder in the monitoring tools thread prefixed his recommendation with Disclaimer: that's me! :-) Not shilling. Hard-code that. Every reply your team posts carries an affiliation statement in the first line, with no exceptions for threads where it feels awkward. It also keeps you clear of the Australian Consumer Law's misleading conduct provisions, which apply to a staff member posting as an apparently disinterested punter just as they apply to an ad.

The human writes the post

The argument for AI-drafted replies is time. The argument against is that the register gives it away. Members of a language learning community describe LLM output as unbearably sycophantic and corporate in its speech patterns, and that is exactly the voice that gets a reply downvoted and your account tagged as a marketer.

The upside of doing it properly is real. When the designer of the board game Cola Wars responded personally to criticism of AI art in his product, commenters on r/boardgames called it an incredibly well made response, and it's what should be the standard in situations like this, including people who had no interest in the game beforehand. A named person answering directly is the whole product here. The model's job is to make sure that person spends their twenty minutes a day on the six threads worth answering instead of scrolling.

Costs and the parts that break

Model inference is the cheap line item once the pre-filter is doing its job. The expensive lines are collection and maintenance.

Scrapers break. The OpenAmer project's Workflow Immune System exists specifically because automations fail when sites redesign, and it re-finds UI elements and patches its own workflows to keep running. Self-healing helps, although it does not remove the need for someone to own the pipeline. Budget a few hours a month for broken collectors, changed API terms and rate limits, and assume at least one platform will close off access during the first year.

Where the data cannot leave your control, self-hosting the collection layer is a live option. OpenAmer is open source and fully self-hosted, which matters for health-sector and government clients who cannot ship community and customer signals into third-party SaaS analytics under their privacy obligations.

The other honest limitation is precision. Classifiers over-select early on, and the first month of queues will contain threads where you have nothing useful to say. Have your reviewer mark those, feed the marked examples back into the prompt or a small evaluation set, and expect two or three rounds before the daily list is short enough that someone reads all of it.

Start with two or three communities you already care about, run the queue for a month with nobody posting at all, and see whether the threads it surfaces are ones you wish you had answered. If they are, you have a case for the posting half. If they are not, you have saved yourself a bot that annoys people.

PicNet builds production AI systems for Australian organisations. Talk to us about what a first project could look like.

Originally published at picnet.com.au.

Answering from your own documents: a policies and procedures assistant for any business

Guido Tapia — Mon, 14 Sep 2026 01:18:30 +0000

Most of the questions that clog an HR inbox or an operations channel have already been answered in writing. The leave policy says what happens when a public holiday falls during annual leave. The installation manual says which bracket suits a tiled roof. The problem is that the answer sits on page 34 of a PDF someone uploaded to SharePoint in 2019, so people ask a colleague instead, and the colleague guesses.

This is the least glamorous use of a language model and the one with the clearest payback. It is also the pattern we are asked about most often, which is why it appears early in Practical AI in Business Operations. The approach below is what we build for clients outside health as well as in it: HR policies, operations manuals, product documentation, supplier agreements, safety procedures.

What the system actually does

A user asks a question in plain English. The system finds the passages in your documents most likely to contain the answer, hands those passages to a language model, and asks it to answer using only what it was given. The answer comes back with links to the source documents and the sections used. If the passages do not contain the answer, the system says so.

That last behaviour is the one that decides whether staff trust the thing. A general model will happily produce a confident paragraph about your redundancy process based on what redundancy processes usually look like. That answer is worse than no answer, because it sounds like it came from your policy.

Architecture sketch

Ingestion. Pull documents from wherever they live (SharePoint, Confluence, a file share, a DMS) on a schedule. Convert to text, keeping headings and page numbers so a citation can point at a section rather than a 90 page file.
Metadata. Every chunk carries the document title, owner, effective date, the version it supersedes, and which entity or state it applies to. This is where most of the value hides. Without it you cannot filter a Queensland question away from a Victorian policy, and you cannot tell the user the answer is current as at a date.
Chunking. Split on document structure, not on a fixed character count. A clause with its heading and its parent heading retrieves far better than 800 characters that start mid sentence.
Index. Embeddings plus keyword search, with results merged. Pure vector search misses exact terms like award classification codes or part numbers. Keyword search alone misses "can I take leave at half pay". You want both.
Answer. The model receives the question, the top passages, and a strict instruction set. It returns structured output, not free text.
Logging. Every question, the passages retrieved, the answer, and any user feedback. This is your evaluation data and your audit trail.

We usually put the index in Postgres with pgvector rather than adding a new vendor. For corpora in the tens of thousands of chunks, which covers most SMEs, it is fast enough and it keeps your document content inside infrastructure you already govern under the Privacy Act.

The prompt contract

The instruction to the model is short and it does not negotiate:

Answer using only the numbered passages below.
Every sentence in the answer must cite at least one passage id.
If the passages do not contain enough to answer, return not_in_corpus.
Do not use general knowledge about how policies usually work.

And the output is a schema, so the application can check it before a human ever sees it:

{
  "status": "answered",
  "answer": "Annual leave accrues progressively...",
  "citations": [
    {"passage_id": 3, "document": "Leave Policy v4.2",
     "section": "5.1 Annual leave", "effective": "2025-07-01",
     "url": "https://.../leave-policy-v4-2.pdf#page=12"}
  ]
}

Then verify before rendering. If status is answered and citations is empty, drop the answer and show the retrieved passages instead. If a cited passage id was not in the set you supplied, treat it as a failure. These checks cost nothing and they catch the most embarrassing errors.

Make the citation a link to the source document at the right page, not a file name. Staff click through more often than you would expect, and the ones who click are the ones who will tell you when a policy is out of date.

Evaluation before anyone else sees it

Do not roll this out on a demo. Build a question set first.

Take 100 to 200 real questions from your HR inbox, service desk tickets or the operations channel. For each one, have the person who currently answers it write down the correct answer and the document and section it comes from. That is a day of someone's time and it is the highest value day in the project.

Then run the set and grade three things: whether the answer is correct, whether the citation points at the right section, and whether the system refused when it should have answered. Add a second set of 30 or so questions you know are not covered by the corpus, including a few that sound like they should be. Count how often the system answers those anyway. That false answer rate is the number to watch, and it is the one that moves when someone changes a prompt, swaps a model version or adds a new document set.

Re-run both sets on every change. It takes minutes once it is scripted, and it turns "the new model seems better" into something you can check.

If your documents are clinically adjacent, keep the assistant on administrative ground: rosters, credentialling, procurement, incident reporting workflow. Anything that could shape care needs a named clinician signing off the answer before it reaches anyone, designed in as a workflow step with an approval record, not added later as a disclaimer.

What it costs to run

Three lines, in descending order of size.

The first is corpus preparation. Someone has to decide which documents are authoritative, retire the superseded ones, and attach the metadata. In most engagements this is the bulk of the effort and it is internal staff time, not licence fees. It is also work that pays off whether or not you build the assistant.

The second is ongoing ownership. A document set that nobody maintains degrades, and a confidently cited answer from a withdrawn policy is a real problem. Budget a few hours a month for someone to review flagged answers and update the corpus.

The third is inference, and it is usually the smallest. Work it out yourself rather than trusting anyone's estimate: multiply the tokens you send per question (retrieved passages plus the question plus the answer, typically a few thousand) by your expected monthly question volume, then check your provider's current published price. A department sized deployment of a few thousand questions a month generally lands in the range of a single software subscription. Embedding the corpus is a one off charge plus a trickle for new documents. If you self host the index, add nothing for the database because it is already there.

When a wiki search is honestly good enough

Sometimes it is. If your corpus is a well structured intranet with good headings, your staff know the vocabulary, and volume is low, decent search plus a table of contents will solve the problem for the cost of nobody's afternoon. The same goes for a corpus small enough that a person can hold it in their head.

The pattern earns its keep when the documents are long, badly structured or scattered across systems, when the answer requires reading across several documents, when users do not know the internal terminology, or when the same questions arrive dozens of times a week. It does not help at all when your documents contradict each other. Fix that first, because retrieval will find both versions and the model will pick one.

PicNet builds production AI systems for Australian organisations. Talk to us about what a first project could look like.

Originally published at picnet.com.au.

When AI must not act alone: human-in-the-loop patterns for security automation

Guido Tapia — Fri, 11 Sep 2026 00:53:24 +0000

Most of the AI security work we do for clients ends up hinging on one question: what is this thing allowed to do on its own at 3am on a Sunday?

Everything else follows from the answer. Model choice, prompt design, which vendor's agent framework you use, all of it matters less than where you draw the line between an action the system takes and an action a person takes. This post closes out Practical AI in Cyber Security with the governance layer that the earlier posts assumed was there.

Draw the line at reversibility

The clean split is not "smart tasks versus dumb tasks". It is reversibility. If the worst outcome of the AI being wrong is a wasted five minutes of someone's reading time, automate it. If the worst outcome is a production system offline, a mailbox item gone, or a clinician locked out of a roster, a human decides.

Safe to run unattended:

Collection. Pulling logs, endpoint telemetry, sign-in events, ticket history, config snapshots into one place.
Enrichment. Resolving an IP to an ASN, looking up a hash, attaching asset ownership, pulling the last three changes to the affected host.
Triage and correlation. Grouping alerts, suppressing known-benign patterns, ranking by likely impact, drafting the "what we think happened" summary.
Reporting. Weekly posture summaries, control-drift reports, evidence packs for an audit.

Behind a human decision:

Blocking. Firewall rules, conditional access changes, disabling a user account.
Deleting. Quarantining or purging mail, removing files, revoking tokens at scale.
Isolating. Pulling a host off the network, suspending a VM, stopping a service.
Anything that touches a system of record. Closing an incident, changing a risk rating, writing to an asset register.

Vendors are pushing hard in the other direction. Cloudflare's own material on agentic AI describes systems that make decisions, execute code, query databases, send email and handle payments without constant human prompting. That is a fair description of the capability. It is also a precise list of the things you want fenced off in a security context, because each one of them is hard to undo.

The split is not new and it is not unique to security. Windmill's engineering write-up on their support pipeline lands in exactly the same place: AI does the volume work, drafting replies and even drafting the fix, but a human still presses send on anything customer-facing. Their reasoning is worth stealing wholesale. The system only works while people trust the drafts, and one wrong auto-sent reply burns that trust for every message after it. A SOC analyst who has been burned once by an automated block on a payment gateway will start ignoring the tool, and then you have paid for AI and got nothing.

The pattern: agent proposes, human approves

The most useful working implementation we have seen of this contract is in a repo, not a security product. The kube-agents project documented a protocol for AI agents contributing code: an agent claims work by self-assignment, escalates anything it cannot resolve with a needs-human label, and cannot merge until a human lgtm gate is satisfied. Three primitives. Claim, escalate, approve.

That maps onto a security runbook almost directly:

claim    -> agent takes alert #4471, marks it in-progress so a second
            agent or analyst doesn't duplicate the work
enrich   -> pulls host, user, recent changes, threat intel, prior tickets
propose  -> writes the finding, the evidence and ONE recommended action
            ("isolate WKS-0412"), status = awaiting-approval
escalate -> if it cannot complete, status = needs-human with the reason
approve  -> named human approves, rejects or edits; only then does the
            action execute, under the human's authority

The kube-agents authors made the contract deliberately framework-agnostic. That is the detail to copy. Your SOC processes will outlive your current model vendor by years. Keep the escalation and approval contract in your own orchestration layer, expressed in your own states, so swapping Claude for Gemini next year is a config change rather than a governance rewrite.

A blind check must never look green

The failure mode that costs real money is not the AI recommending the wrong action. It is the AI reporting "all clear" on a check that never ran.

Alert and fault are different states and need different plumbing. An alert means the check ran and found something. A fault means the check could not run: the API token expired, the log source stopped shipping, the model call timed out, the agent hit a rate limit. Both are "not green", but they need different handlers, different dashboards, and often different people.

Most monitoring code we inherit collapses the two. The check throws, something catches the exception, logs it at debug level and returns a default, and the tile stays green for eleven weeks while the log source has been dead the whole time. Adding an LLM to that pipeline makes it worse, because models are obliging. Ask one to summarise incomplete data and it will produce a confident, well-formatted summary of the subset it could see, with no flag that half the estate was missing.

What we build into these pipelines:

Every check returns one of three states, never two: ok, alert, fault. There is no default value on the failure path.
A fault renders as a distinct colour on the dashboard, not as a green tile with a footnote.
Staleness is a fault. If a source has not reported inside its expected window, the check that depends on it is faulted, not passed.
Coverage is reported alongside every AI-generated summary. "23 of 26 sources; 3 faulted" sits above the narrative, generated by the pipeline rather than written by the model.
The model never sets its own coverage figure. That number comes from the orchestrator, which knows what it asked for and what came back.

The needs-human label is the same idea at the agent level. An agent that hits something it cannot resolve emits an explicit escalation state rather than quietly returning success.

The audit trail is the deliverable

When someone asks how your AI-assisted decisions are governed, the honest answer is whatever your logs can show. Governance is a document exercise at the point it gets tested.

That point is closer than it looks for organisations with any European exposure. The EU AI Act moved into enforcement on 31 August 2026 with the AI Office issuing formal requests for information to general-purpose model providers, covering model security, independent external evaluations and post-market monitoring. The requests went to providers rather than to deployers, so an Australian organisation using an API is not in the first wave. But the pattern is set: the questions are document-driven, the answers become part of a permanent supervisory record, and incorrect or incomplete replies carry penalties of up to 15 million euros or 3% of global turnover. Australian organisations serving EU users, or working under contracts that flow those obligations down, will be asked to evidence human oversight of AI-assisted decisions. You cannot backfill that evidence.

For each AI-assisted decision we record: what proposed the action, which model and version, what evidence it saw (the actual query results, not a paraphrase), what it recommended, who approved or rejected it, when, and what executed. In practice the evidence snapshot is the hard part, because inputs change under you. Store it with the decision.

One more control, learned from an open-source community rather than a compliance framework. The nixpkgs maintainers proposed closing any pull request that removes the AI disclosure checkbox, on the basis that merging one would be a serious policy violation. The interesting part is the failure mode they found. The control existed, but it was a checkbox the submitter could delete. So in a security pipeline, AI-provenance metadata must be system-generated and not editable by the thing that generated the content. If the agent can write its own "reviewed by a human" flag, the flag means nothing.

Roll it out in dry run, and stay there longer than feels necessary

Every automation we deploy runs in dry run first. The agent does the full job, including selecting the action, and writes what it would have done to a log and a channel humans read. Nothing executes.

We run this for a minimum of four weeks in most environments, longer for anything that touches identity or network paths. The dry-run period tells you three things you cannot get from testing: how often the agent recommends something an analyst would not do, how noisy it is at 2am when the batch jobs run, and whether the evidence it attaches is actually enough for a human to decide from. That last one fails more often than the first.

Promotion out of dry run is per action type, never for the whole system. Enrichment and triage might graduate in week two. Automated blocking might never graduate, and for most clients it should not.

An honest cost note. Human-in-the-loop is more expensive to run than full automation, because you are paying for the AI and keeping the analyst. The saving comes from what the analyst spends their time on: reviewing a drafted finding with evidence attached takes a couple of minutes, where building that finding from scratch takes twenty or thirty. The approval queue itself needs an owner and an SLA, or it becomes a place where alerts go to age quietly.

For anything clinically adjacent, the sign-off is part of the design rather than a caveat bolted on the end. An AI can assemble a suspected-breach pack, correlate the access logs, draft the notification timeline. A named clinician or privacy officer approves what leaves the building, and their name is on the record.

PicNet builds production AI systems for Australian organisations. Talk to us about what a first project could look like.

Originally published at picnet.com.au.

Ad spend guardrails: automated Google Ads reporting and anomaly alerts

Guido Tapia — Wed, 09 Sep 2026 00:58:25 +0000

Most organisations we work with have alerts on cloud spend. If an AWS account jumps 30% overnight, someone gets paged. The same organisation will spend a comparable amount on Google Ads and review it once a month, in a meeting, from a deck built by the agency that placed the ads.

That gap is where money disappears. A campaign gets disapproved and stops serving for eleven days before anyone notices. A bid strategy switches to maximise clicks and burns three weeks of budget on traffic that never converts. A landing page 404s after a site release and the ads keep running. None of this is exotic. It is ordinary operational drift, and it is invisible until the monthly report, by which point the money is gone.

This post is part of our Practical AI in Marketing series, and it describes a small system we build for clients: a daily automated report on Google Ads spend and performance, with anomaly alerts, and a plain-English summary that a business owner will actually read.

Why you instrument it yourself

Two things happened in the last week that make the case better than I can. On 2 September a US court declined to force Google to sell its ad tech business, accepting behavioural remedies instead, the third time in recent years that US antitrust enforcers have failed to break up a Big Tech company. The buy side and the sell side of that market stay under one roof. The day before, the FTC and 22 state attorneys general alleged that Amazon made over US$20 billion by overriding its own ad auction results with higher prices, a claim Amazon disputes.

Whatever the outcomes, the practical lesson for an Australian advertiser is the same. Auction transparency is not arriving from a regulator on a timetable that helps your budget this quarter. Newer AI-driven ad channels have their own problems; practitioners are already publicly complaining that ChatGPT ad targeting sends spend to irrelevant audiences. Pull your own spend and conversion data into a system you control, keep your own history, and reconcile platform-reported conversions against what your CRM says actually closed.

The shape of the system

A scheduled job pulls campaign, ad group and keyword metrics from the Google Ads API each morning, read-only.
Rows land in a database table you own, one row per entity per day, appended and never overwritten, so you keep your own history independent of platform reporting windows.
An expectation layer computes what each campaign should have spent and returned.
A deterministic rules engine compares actuals against expectations and raises anomalies.
An LLM turns the day's numbers and any anomalies into six or eight sentences of plain English.
Delivery goes to email and Slack or Teams, with the raw table available for anyone who wants to dig.

The extraction query is unremarkable:

SELECT
  campaign.id,
  campaign.name,
  campaign.status,
  metrics.cost_micros,
  metrics.impressions,
  metrics.clicks,
  metrics.conversions,
  metrics.conversions_value
FROM campaign
WHERE segments.date DURING LAST_7_DAYS

Read-only by default

The credential the reporting job uses must not be able to change anything. No write scopes, no budget edits, no pausing campaigns, even though the API would happily allow all three.

This is not a hypothetical concern. A recently merged open-source project, the KPA Traffic Stack, builds exactly this pattern for Meta and Google Ads diagnosis: read-only exports with explicit guardrails against accidental writes and credential leaks, validators that reject exports containing secret-like keys, and a flag on every export asserting that platform writes are not allowed and human approval is required. The automated code review on that pull request found a real hole in it, where one validator trusted a self-reported "no credentials" flag rather than scanning nested fields for tokens. Worth reading if you are building something similar, because the same mistake is easy to make.

Two reasons for the restriction. An agent or script with write access to a live ad account is a bad incident waiting for a bad day. And once the report can act, its findings stop being an independent record and start being a description of its own behaviour.

If you do want programmatic control of campaigns, keep it in a separate, reviewed path. The agoraform project treats Google Ads as config-as-code, covering conversion goal, budget, campaign, ad group, targeting, keywords and responsive search ads in one declarative resource graph with a validate, plan, apply lifecycle. Campaign changes then get diffed and reviewed like any infrastructure change, by a person, on a different credential.

Anomalies need an expected value, not last week's number

The weak version of this system alerts when spend moves more than some percentage against the prior period. It produces noise every Monday, every long weekend and every end of financial year.

The better version compares against a modelled expectation. The mizan target engine is a good illustration of the arithmetic: it deterministically reverse-engineers required revenue, new customers, sales, qualified calls, shows, bookings, leads and ad spend from a monthly profit or margin goal, and computes a maximum sustainable acquisition CAC, maximum media CAC and maximum cost per lead along the way. That maximum sustainable ad spend is the ceiling a runaway-spend alert should be measured against. It also fails closed: missing months stay missing rather than becoming zero, and impossible targets return an explicit unattainable result instead of a plausible-looking number.

Our rules end up looking like this:

- id: runaway_spend
  window: rolling_7d
  trigger: spend > 1.15 * max_sustainable_daily_spend
  severity: high

- id: dead_campaign
  window: rolling_3d
  trigger: spend > 2 * target_cpa and conversions == 0
  severity: high

- id: silent_campaign
  window: rolling_2d
  trigger: status == "ENABLED" and impressions == 0
  severity: medium

- id: cpl_drift
  window: rolling_14d
  trigger: cost_per_lead > 1.3 * max_cpl and spend > 500
  severity: medium

The spend floors matter. A campaign running at $30 a day will breach any ratio-based rule constantly, so suppress alerts below a dollar threshold where the money at stake is smaller than the cost of reading the email.

Where the LLM helps

The LLM does one job: it writes the summary. It receives the computed table and the list of triggered anomalies as structured input and produces a short narrative. It does no arithmetic, no ranking, no attribution reasoning. Every number in its output must appear in the input, and we validate that before sending.

The reason is boring and practical. A business owner will not open a dashboard daily, and will not read a 40-row table. They will read four paragraphs in an email that says spend was $4,180 against a $3,900 expectation, that the brand campaign is fine, and that a particular campaign has spent $612 in three days with no conversions and should be checked today. Prior to LLMs, we wrote that summary with templates, and the templates read like templates. This is the piece where the model genuinely earns its place.

Costs are small. The Google Ads API is free to use, though you need a developer token and basic access approval. Daily summarisation for a mid-sized account runs to a few thousand tokens, which is cents per day. The build is the expense: a week or two of engineering for a first version covering one account, plus a smaller amount of maintenance because Google retires API versions regularly and you will be doing an upgrade at least annually.

Honest limitations

Conversion lag is the big one. Yesterday's conversions are undercounted, so a dead-campaign rule reading a single day of data will produce false alarms. We compare on a lagged window and state the lag in the summary. Attribution is the second: Google's reported conversions and your CRM's closed deals will not agree, and the system should show both rather than pretending one is truth. Third, an alert nobody actions is worse than no alert, so cap the daily volume and give each rule an owner.

One Australian-specific note. If your campaigns use customer match or remarketing audiences, the email addresses and identifiers you upload are personal information under the Privacy Act, and the handling, consent and retention questions belong with your privacy officer before the reporting project starts, not after.

PicNet builds production AI systems for Australian organisations. Talk to us about what a first project could look like.

Originally published at picnet.com.au.

A morning digest your team actually reads: AI-summarised news, spend and system health

Guido Tapia — Mon, 07 Sep 2026 01:28:26 +0000

Most Australian operations teams I talk to are subscribed to too much. Cloud billing alerts to one inbox, an uptime tool posting to a Slack channel, ad platform notifications to whoever set up the account three years ago, four vendor newsletters, and a deploy bot that fires on every merge. Nobody reads any of it. The item that actually mattered last Tuesday was sitting in a channel someone muted in March.

The approach we keep coming back to is unglamorous: one email, one delivery list, sent at 7am, ordered by what needs a decision today. Industry news summarised by a model, yesterday's cloud and ad spend, and anything monitoring flagged overnight. This post is part of our Practical AI in Business Operations series, and it is the cheapest useful thing on the list.

Why one digest beats ten streams

Notification channels rot quietly, and nothing tells you when they do. A good example sits in a push notification service where the send call returned a bare boolean, so a permanent APNS 410 (the token is dead, the app was uninstalled or push was disabled) looked identical to a transient network blip. Dead device tokens were never pruned and the failures kept accumulating with nobody watching (TrackRat #1802).

That is the same failure mode as ten alert channels in an organisation. Each one is a small integration nobody owns. One digest with a known distribution list is a single thing to keep honest, and every recipient notices the day it stops arriving.

What the digest contains, in the order it matters

Order sections by what forces action, not by data source. Grouping by category buries the item that needs a response, which is exactly what happened to a changelog generator that grouped commits by their type prefix. Breaking changes were rendered under "Added" and "Changed", and one commit that deleted eight internal packages shipped in a published changelog under "Changed" where an auditor would never look for it. The fix was to give breaking changes their own section and sort it to the top (go-kure #775).

So the running order in ours is:

Overnight alerts. Anything monitoring raised between the last digest and this one, with the alert still open flagged separately from the ones that self-cleared.
Spend. Yesterday's AWS or Azure spend against the trailing seven-day average, plus Google and Meta ad spend from the same window. Percentage moves in the heading, dollars in the row.
Industry and vendor news. Summarised, triaged, and always last, because it is the section a reader is allowed to skip.

The architecture

It is a scheduled task, not a platform. Ours runs on a cron trigger before business hours in Sydney and takes a few seconds:

One collector per source, each returning (rows, error). Collectors never raise into the pipeline.
One summariser call per section that needs prose. The alerts and spend sections are rendered from data with no model involved.
One renderer producing HTML and a plain text alternative that says the same thing.
One run log recording which sources answered, which failed, and how many items each contributed.

The (rows, error) shape carries more weight than it looks like it should:

def render(section):
    if section.error:
        return f"{section.title}: COULD NOT READ ({section.error})"
    if not section.rows:
        return f"{section.title}: nothing overnight"
    return format_rows(section.rows)

An empty read and a failed read must render differently. A morning brief script learnt this the hard way: its calendar collector called the API with no calendarId, so it read the account default and returned zero events for a week that had nine, including a weekly planning meeting. The brief printed "nothing" on days that had meetings, and the zero read as a pass. The prompt named the calendar in prose, which the author summarised as "prose is not an argument" (kipi-system #299).

That is the archetypal silent digest failure. The email still arrives, still looks complete, and an entire data source is missing. Enforce data scope in code with an explicit parameter, and make source coverage a section of the digest itself so a missing feed is visible to every reader rather than to nobody.

Prompt design that stays faithful to sources

Three rules do most of the work.

First, the model chooses, and code checks the choice. Give the summariser a small set of named admission rules and require it to cite which rule admitted each item it keeps. Then validate the citation in code and drop anything citing something else. A team digest at PostHog did exactly this with four named rules (contract, assumption, decision, customer) and went from keeping 21 of 26 merges to keeping zero to two (PostHog #88286). Their earlier attempt used a count target in the prompt, which squeezed out real news on busy days and admitted routine work on quiet ones.

Second, no untrusted text reaches the prompt in a position where it can address the summariser. The same PostHog change removed contributor-written PR bodies from the prompt entirely, and made sure the remaining values could not close their own tag and continue as instructions. Vendor news is worse than internal commits here: press releases are written to be persuasive, and you are feeding them to a model whose job is to be sceptical about them.

Third, ask for assessment, not paraphrase. A summary that repeats an item's own marketing copy is worse than the headline alone, because it launders the claim into your team's morning reading. A live automated feed we looked at as a format reference gets this right: a headline count ("20 new videos across your watchlist"), then per item a title, source, a triage verdict such as Skim or Skip, and one line of critical assessment (youtube-digest #10). The triage label is what ten notification streams can never give you. It tells the reader what not to open.

Freshness rules, and the trap underneath them

Decide deliberately what happens to an item the digest did not carry, because there are two separate loss paths and they need different answers. Something cut by a render cap is different from something the summariser read and rejected.

Get this wrong and the digest quietly decays. PostHog's was carrying merges that had landed up to six days earlier with nothing marking them as old, because anything the cap cut or the summariser dropped was re-queued into the next morning's run. The rejected ones then produced the same empty result every day, forever, since the same input to the same prompt returns the same answer. Their fix was to let a run consume everything it claims, including an empty answer. Either age items out or label them explicitly as carried over.

Editorial rules as a binding contract

Write the rules down once, in one referenced document, and treat them as constraints any agent revising the text must obey. Scattering them through ad-hoc prompts guarantees drift. A useful example of the form is an editorial supervision contract written for agent-generated prose: core rules, an explicit list of banned constructions, and a revision standard listing what every pass must actively search for (janeco #3).

For a digest, ours is short. Every claim links to its source. No item is described in the vendor's own adjectives. An item the model cannot summarise from the source text gets a link and no summary. Every section states its coverage. Nothing in the digest ever asserts something the collector did not read.

Honest limits and costs

The model spend is a rounding error next to the cloud bill the digest reports on. A handful of calls a day against a small model costs less than the coffee consumed while reading the output. The real cost is engineering time on the collectors, because ad platforms and cloud billing APIs change, and a broken collector that fails loudly still needs someone to fix it that week.

The other limit is scope. A digest is a reading aid, not a control system. It should never take an action, close an alert, or approve spend. Every item links back to the console where a person does that. And it needs one owner who reads it every morning and notices the day a section goes quiet for the wrong reason.

Build the smallest version first. One source, one section, one recipient, running for a fortnight before you add the second source. Most of the value shows up in week one.

PicNet builds production AI systems for Australian organisations. Talk to us about what a first project could look like.

Originally published at picnet.com.au.

Cloud posture on autopilot: continuously auditing AWS, Azure and Microsoft 365

Guido Tapia — Fri, 04 Sep 2026 00:58:28 +0000

Most of the cloud security incidents we get called into have nothing to do with a novel exploit. Someone opened a security group to 0.0.0.0/0 for a vendor's remote session and never closed it. A storage account got flipped to public so a contractor could pull a file. An admin was excluded from conditional access "temporarily" in March. A mailbox forwarding rule was added and nobody noticed because nobody was looking.

None of that shows up in a vulnerability scan. It shows up in configuration, and configuration changes constantly, in three or four different consoles, made by people who had a good reason at the time. This post is part of our Practical AI in Cyber Security series, and it covers the service we run for clients: a scheduled job that reads the current state of AWS, Azure and Microsoft 365, compares it to a stored baseline, and emails a human when something has moved.

Why point-in-time auditing stopped working

The traditional pattern is an annual review where someone screenshots portal settings for the auditor. Cloud-first teams have been moving away from that towards continuous, code-driven checks that run against AWS and Azure on a schedule, so the audit is a job that runs rather than a scramble in the weeks before certification (r/Information_Security discussion, Aug 2026).

The control pattern that thread describes is worth stealing directly: state the requirement, tag the resource, run the check, report the result. It is boring and it works across every control domain we care about. Each of our checks is written that way, as a small independent unit with a stated requirement and a pass or fail, rather than as a giant "security score" that nobody can act on.

There is a second reason to build this yourself rather than lean entirely on the native tools. Most Australian organisations we work with are not on one platform. They have Microsoft 365 for identity and email, some Azure, and AWS from an acquisition or a dev team that got there first, and the relative growth of the big three suggests that mix is not resolving itself any time soon (r/Mag7shares, Aug 2026). Three consoles with three different opinions about what "secure" means is how things get missed. One consolidated report, one owner, one weekly read.

What we actually check

The check list is deliberately narrow. These are the things that drift, that matter, and that a person can fix in an afternoon.

Identity in AWS and Azure: IAM users with console access and no MFA, long-lived access keys, keys that have not rotated, roles with wildcard permissions, orphaned service principals.
Network exposure: security groups and NSGs allowing inbound from anywhere, especially on 22, 3389, 1433 and 3306. Public IPs on resources that had none last week.
Storage: S3 buckets and Azure blob containers with public access, anonymous read enabled, or encryption settings changed.
Microsoft 365 identity: conditional access policies added, disabled or scoped down, break-glass exclusions, MFA registration state versus MFA enforcement (they are not the same thing and the gap is where people live), legacy authentication re-enabled.
Exchange Online: mailbox forwarding rules to external domains, new transport rules, mailbox delegation changes.
Entra ID app surface: new enterprise app registrations, OAuth consent grants and the scopes they were granted, changes to admin role membership including eligible assignments in PIM.
Cost and orphan drift: resources that stopped being used but did not stop being billed. One operator running an identical 24/7 workload found AWS came in around 30% more expensive than the alternatives (r/automation, Sep 2026), which is a fair reminder that an unnoticed left-running instance is a real cost, not just an untidy inventory.

That last one is not strictly a security check, but it uses the same snapshot and it is the item that most often gets a client to actually read the email.

The architecture

The design is simple enough to describe in a paragraph, which is part of the point.

A scheduled worker (we run these hourly for identity checks, daily for everything else) authenticates to each tenant using a read-only role: an AWS IAM role assumed cross-account, an Azure reader plus a Graph app registration with directory and policy read scopes. No write permissions anywhere.
Each collector pulls the relevant state and normalises it into a flat JSON document per platform. Resource identifier, control name, observed value, timestamp.
That document is diffed against the last stored snapshot. Any difference is a drift event with a before value and an after value.
Drift events go to a model that writes the summary a human reads: what changed, who changed it if the audit log says, and why it matters for this control.
The report goes to email and to a Teams channel. Nothing is remediated automatically.

Where clients already have infrastructure as code, the diff is even cleaner, because the declared template is the baseline and live state is the thing that has wandered. Practitioners in that same ISO 27001 thread make the point about Bicep specifically. Full IaC coverage is rare in the environments we inherit, so snapshots plus drift detection give you the same signal without requiring a rebuild first.

Where the AI part earns its keep

The detection is deterministic. Rules, diffs, comparisons. We do not use a model to decide whether something is a finding, because a model that occasionally misses a public bucket is worse than no check at all.

The model writes the explanation. Raw drift output is unreadable: a JSON blob showing that conditionalAccessPolicy/7f3a... moved from enabled to enabledForReportingButNotEnforced. The summary the client gets says that the policy requiring MFA for admin sign-ins from outside Australia was switched to report-only mode on Tuesday afternoon, that it currently blocks nothing, and that the change was made by a named admin account. Same fact, but one of them gets actioned.

The model also groups related drift. Twelve NSG changes from a single deployment become one paragraph rather than twelve alerts, which is the difference between a report people read and a filter rule people write.

We keep the model on a short leash. It gets the drift record and the control description, and it writes prose from those. It does not query the tenant, it does not decide severity on its own, and every summary sits next to the raw before and after values so the reader can check it. When the model is wrong, it is wrong about wording, not about whether something changed.

Detect and notify, not auto-fix

We deliberately do not remediate automatically. Experienced operators are sceptical of automated remediation for good reason, as one put it in that thread: "I'm personally not a fan of automated remediations as it's hard to do."

Our experience matches. A rule that automatically closes an open security group will eventually close the one that a legitimate integration depends on, at 2am, in the environment where nobody documented the dependency. In tenants with clinical or health administration workloads, that risk is not acceptable at all. Any change to an operational system needs a person to sign it off before it happens. That is a design requirement for us, not a footnote.

So the output is a decision point, not an action: here is what changed, here is what we think it means, here is the button that says "expected, add to baseline" or "not expected, raise a ticket".

The first run rule: never silently seed

This one is worth stating on its own because it is the mistake we see most often in home-grown versions.

The first time the service runs against a tenant, there is no prior snapshot, so nothing is technically "drift". The tempting behaviour is to quietly store that state as the baseline and start alerting from run two. That means every pre-existing misconfiguration, every public bucket and every stale global admin becomes invisible by definition, permanently blessed as normal.

Our first run emails the complete baseline instead. Every public storage container, every account without MFA, every open security group, every external forwarding rule that exists on day one. It is usually an uncomfortable document, it is often the most valuable single report a client gets from us, and it forces an explicit decision on each item before anything becomes "normal".

Costs and limitations

Being honest about what this does not do:

It only sees what the APIs expose. Graph rate limits mean a large tenant's full Microsoft 365 sweep takes longer than you would like, and some settings are only readable through interfaces that change without notice. Collectors break, so we monitor the collectors themselves and treat a silent run as a failure.

It is noisy in environments with active deployment pipelines until the rules are tuned. Expect two to four weeks of tuning before the signal to noise ratio is good, and expect to keep tuning as the environment changes.

New rules should never fire their first alert against production. Local emulators for AWS, Azure and GCP let you develop and regression-test drift rules offline (flowg.cloud, Aug 2026), which we use for the AWS side in particular. The same write-up is candid that emulators do not validate real authentication, so the last test always happens against a real tenant, in read-only mode, before a rule goes live.

Cost-wise, the compute is trivial. A scheduled worker plus snapshot storage runs at a few dollars a month. The model calls for summarisation are similarly small because they only fire when there is drift. The real cost is the human time in the first month, building the check set for your environment and clearing the day-one baseline. Budget for that honestly, because a posture service nobody triages is just an expensive mail rule.

PicNet builds production AI systems for Australian organisations. Talk to us about what a first project could look like.

Originally published at picnet.com.au.

Tracking search rankings and AI Overviews without an agency

Guido Tapia — Wed, 02 Sep 2026 01:03:29 +0000

The monthly agency deck lands, forty keywords are green, and somebody in the room asks the question the deck cannot answer: when a buyer asks ChatGPT or Gemini who does this work in Sydney, do we come up? Ranking well in the ten blue links and being absent from AI answers are now two different states, and a brand can sit in one while assuming the other (RadarKit vs Peec AI vs Profound). That gap is the reason to run your own telemetry. This post is part of our Practical AI in Marketing series, and it covers the smallest thing that works: a scheduled job, a table you own, and a weekly report.

Record presence separately from position

The main design decision is in the schema. Position and AI-surface presence are different measurements and collapsing them into one "visibility score" throws away the signal you actually want.

create table serp_check (
  checked_at  timestamptz not null,
  keyword     text        not null,
  locale      text        not null,  -- gl=au, hl=en-AU, Sydney or Melbourne
  engine      text        not null,  -- google, google_ai_mode, chatgpt, perplexity
  position    int,                   -- null when absent, never 0
  ai_present  boolean,               -- was an AI answer rendered at all?
  cited       boolean,               -- were we named or linked in it?
  competitors text[],                -- who was named instead
  raw         jsonb       not null   -- the full response, kept forever
);

Two details matter. Use null for absence rather than 0 or 101, because averages over a sentinel value quietly lie to you. And keep the raw payload. The tool market is churning hard enough that August 2026 alone produced round-ups of 17 AI rank trackers plus separate lists of LLM visibility tools and vendor alternatives. Any of those products may not exist in two years. Your history should live in your database, not theirs.

Set the locale properly. Australian result sets differ from US ones, and a national keyword checked from a US data centre will tell you about a market you do not sell into.

The job itself

The architecture is deliberately boring:

A keyword and prompt list in a config file in the repo, reviewed by whoever owns the site.
A scheduled runner: cron on a small VM, an Azure Function on a timer, or a GitHub Actions workflow if you want the config and the history in one place.
A SERP API call per keyword for classic Google, returning positions and any AI Overview block.
Optional web-grounded calls to the LLM assistants your buyers actually use, one row per prompt per engine.
An append-only write to Postgres or SQLite, plus a generated weekly markdown or HTML report.

None of this is novel work. A public example repository posted in August 2026 does the whole loop with a prompt bank, four optional engines, JSONL history and a weekly workflow, and Lettertrace ships an MIT-licensed bring-your-own-key version where your data sits in your own Supabase. The tracking layer is commoditised. Fork something, or write it in a day.

What it costs

Per-query pricing is the easy part. SERP APIs bill per successful query in the fractions-of-a-cent range at the volumes a single company needs, so the arithmetic looks like this: 60 keywords checked daily is roughly 1,800 queries a month, which at half a cent a query is under $10 AUD. Add three AI surfaces on a weekly cadence and you are still in tens of dollars, not hundreds. Grounded LLM calls cost more per query than raw SERP fetches, so run those weekly rather than daily.

Verify tier pricing against official sites before you budget. One August 2026 comparison listed Ahrefs at $199 per month per index and was corrected in its own comments to a $50 starting price, which the author acknowledged (9 best LLM visibility tracking tools). Round-up pricing is unreliable in both directions.

The real cost is engineering time: a day or two to build, then an hour or two a month keeping it alive. Budget that hour honestly, because it is the line item people forget.

Rate limits and failure handling

Providers cap concurrency well below what a naive loop will attempt. Practical defaults we use:

Queue the keywords and run them with bounded concurrency, not a parallel map over the whole list.
Retry on 429 and 5xx with exponential backoff and a hard attempt limit.
Add a few minutes of random jitter to the scheduled start so you are not hammering the API at the same second as everyone else's cron.
Write a row for every failure with a status field. A missing row and a genuine absence must never look the same in the data.

That last point catches the worst failure mode. If your job dies halfway through Tuesday and you silently store nothing, next week's chart shows a ranking collapse that never happened.

Parsers break, and that is normal

The AI surfaces change shape month to month. On 28 August 2026 Google AI Mode added flight price tracking, points and mile rates, and hotel booking. Layout changes of that kind break whatever selector or heuristic you use to decide "AI Overview present?".

Two cheap defences. First, a canary: pick two or three keywords that reliably trigger an AI answer and alert if they all report absent on the same day, because a global drop is a parser bug far more often than a ranking event. Second, put a recurring maintenance task in the backlog rather than waiting for the dashboard to look wrong.

Reading the trend

Daily numbers jitter for reasons that have nothing to do with your site: test buckets, location, index refreshes, and in the case of LLM answers, plain non-determinism between runs. Rules that keep people calm:

Report a seven-day rolling median position, not yesterday's number.
Treat AI presence as a rate, not an event: "cited in 9 of 28 checks this month" is a number you can act on.
Only investigate a move that holds for four or more consecutive checks.
Alert on the competitor field, not just your own. A rival appearing in six answers where you appear in none is the signal worth a meeting.

Where DIY wins, and what an agency still adds

There are two jobs here and only one of them is a script. Counting how often you appear is a dashboard. Working out why you are missing from AI answers across Google AI Overviews, ChatGPT Search, Perplexity, Copilot and Gemini is diagnosis, and it is where an agency or consultant still earns the fee (AI GEO strategist thread). The method is not the moat either: search engines document their own systems and large agencies publish their manuals for free (how to learn SEO and AI search for free), so what you are buying is judgement and follow-through, not secret knowledge.

Judge your internal dashboard against the three questions agencies are being asked to answer in 2026: does the brand appear in AI-generated answers, which competitors are recommended instead, and what would change that (AI visibility trackers for agencies). A script answers the first two well and the third not at all. If you want a template for splitting the work formally, an August 2026 engineering ticket ran a gap analysis of an in-house visibility surface against a commercial product and produced an explicit build list. That is the right shape of decision: build the collection and the history, licence or outsource the analysis.

Honest limitations

API results are not what a logged-in customer in Parramatta sees on their phone. LLM answers vary between identical runs, so any single check is close to meaningless and only the rate over weeks means anything. Coverage of AI surfaces varies by provider and lags each layout change. You own the maintenance, and telemetry does not connect to revenue on its own. What it does buy you is a defensible measurement you control, at a cost that rounds to nothing, so the conversation with your agency starts from shared numbers instead of their slide deck.

PicNet builds production AI systems for Australian organisations. Talk to us about what a first project could look like.

Originally published at picnet.com.au.

Document hygiene at scale: finding stale, duplicated and over-shared files

Guido Tapia — Mon, 31 Aug 2026 01:05:58 +0000

Most Microsoft 365 tenants we are asked to look at have the same shape. A few thousand active documents that people use every week, and somewhere behind them a much larger pile that nobody has opened since the project closed. Old tender responses. Three versions of the same policy in three team sites. A OneDrive belonging to someone who left in 2022, still holding a CSV export of the customer database.

None of that is a crisis on any given day. It becomes one when a mailbox is compromised and you have to work out what the attacker could reach, or when someone asks you to prove that personal information has actually been deleted rather than merely forgotten. The honest answer in most organisations is that nobody knows what is in there.

This post is about how we build a scheduled audit that answers the question, and how we roll it out without deleting something a court later asks for.

Three signals worth checking on a schedule

Rather than trying to classify every file, we look for three patterns that reliably indicate a problem.

The first is subtree staleness. A single old file means nothing. A folder where nothing at any depth has been modified or opened in eighteen months, with no retention label and no legal hold, is a different animal. It is usually a finished project, and the whole subtree can be dealt with as one decision instead of four hundred.

The second is sharing links that outlived their purpose. Anonymous links created for a tender that closed two years ago. Guest accounts from a consultancy engagement that ended. Links with edit rights where view rights would have done. These are cheap to find because the permission graph is right there in the API, and they are the findings that most often make an IT manager sit up.

The third is personal-drive data that should not exist. System exports, copies of finance workbooks, HR documents in an individual's OneDrive rather than the site that governs them. This is the least popular part of the audit and usually the most valuable.

The architecture

The system is deliberately boring. Most of the work is enumeration and diffing, and only a thin slice of it needs a model.

A scheduled job walks every site and drive through the Microsoft Graph, using delta queries so subsequent runs only pull what changed. Metadata only at this stage: path, size, owner, timestamps, sensitivity and retention labels, version count, permissions and sharing links.
Each run is written as a snapshot. An embedded store with JSON support is enough for this. SQLite will accept the raw Graph JSON and let you index fields out of it with generated columns, so you can add a new attribute to the audit without rebuilding the schema.
Deterministic rules run over the snapshot first, because they are free. Exact duplicates by content hash, stale subtrees, links with no expiry, external sharing on sites marked internal, orphaned drives.
Near-duplicate detection uses embeddings rather than hashes. Two copies of the same contract that differ by a filename and a date field will never match on a hash, but they sit next to each other in vector space. That is the same property behind dense retrieval work such as HyDE: documents get compared by meaning, not by bytes.
Content classification runs last and only over the ambiguous remainder. Files that need parsing before anything can be said about them go through a document parsing model, then a classifier that answers narrow questions: does this contain personal information, is it a record or a working copy, which business function does it belong to.
Everything ends in a per-owner report and a proposed-actions table, which is where the rollout begins.

Diffing snapshots is the part people underestimate. A single run tells you the tenant is messy. Two runs a month apart tell you which parts of the mess are growing, and that is what gets budget.

-- folders where nothing in the subtree has changed in 18 months
SELECT folder_path,
       COUNT(*) AS files,
       SUM(size_bytes) / 1024 / 1024 AS mb,
       MAX(modified) AS last_touched
FROM items
WHERE retention_label IS NULL
  AND legal_hold = 0
GROUP BY folder_path
HAVING MAX(modified) < date('now', '-18 months')
ORDER BY mb DESC;

Where the model earns its place, and where it does not

Most findings come out of rules and joins. The model is worth paying for on the slice where a human would otherwise have to open the file: is this scanned PDF an executed contract or a draft, does this spreadsheet hold customer records, are these two documents the same thing.

Cost is manageable if you keep that slice small. Cohere prices its Parse model at USD $1.50 per 1,000 pages through its API, which is a useful anchor for what tenant-scale parsing costs; vendors are now pitching document intelligence at whole organisational corpora rather than single files. At that order of magnitude, parsing a hundred thousand pages is a few hundred dollars, and re-parsing them every night is a waste. Parse once, cache by content hash, and only reprocess when the hash changes.

Where the model does not belong is the delete decision. Classification is an input to a rule, not a substitute for one. If the pipeline cannot explain a proposed action in terms a records officer can read, it does not get to propose it.

Rollout: audit, dry run, then enforce

We have never had a good outcome from turning deletion on early. The sequence that works:

Audit only, for at least a month. The system reports and does nothing else. This is where you discover that the "stale" site is the one Finance opens each June, and that half the external links belong to an auditor who is still engaged.

Dry-run deletes, for several weeks. Every action the system would have taken is written to a ledger with its rule, its evidence and its owner. Owners get a weekly digest of what would have happened to their content. The rules get tuned against real objections rather than imagined ones. This phase is deliberately long, because seasonal work is invisible in a two-week window.

Enforcement, scoped narrowly and reversibly at first. Start with the actions nobody defends: expiring sharing links, removing guest access for closed engagements, archiving stale subtrees to cool storage rather than deleting them. Keep the recycle bin window generous and keep the ledger. Actual destruction should come last, run against retention labels, and generate a record of what was destroyed and under which rule.

Privacy and retention, in Australian terms

Two things matter here for Australian organisations.

The first is that the audit is the evidence. Retention policies are easy to write and hard to prove. A dated snapshot, a rule, a dry-run ledger and a destruction record together demonstrate that the policy is operating, which is a far better position than an assertion. If your organisation holds personal information it no longer needs for a permitted purpose, the audit is what tells you where it is, and the ledger is what shows you dealt with it. Take the specifics of retention periods from your privacy officer or legal counsel, not from the pipeline.

The second is that the audit is itself a privacy surface. Paths and filenames leak content, sometimes including names and case identifiers. The index is a concentrated map of everywhere sensitive material lives, so it needs the same controls as the data it describes: Australian region, tenant-controlled keys, restricted access, its own retention period. If you send content to a hosted model for classification, know which jurisdiction it lands in and what the vendor retains. Private or on-premise deployment is a real option for regulated workloads.

For health, legal and financial records, keep the pipeline administrative. Inventory, duplication, sharing exposure and retention status are safe ground. Any action touching clinical or case records goes to a named human for sign-off before it executes, and that sign-off is designed into the workflow, not bolted on as a warning.

Limitations worth stating up front

Access timestamps in Microsoft 365 are less reliable than they look, so treat staleness as a prompt for a human, not proof of abandonment. Near-duplicate detection has false positives on templated documents, where twenty tender responses are legitimately ninety per cent identical. Legal holds and matter-based retention must be resolved before any rule runs, because a wrong deletion there is far more expensive than a thousand stale files. And no audit fixes the behaviour that created the mess. Expiring sharing links by default and giving teams somewhere sensible to put finished work does more for hygiene over a year than any cleanup run.

This post is part of our Practical AI in Business Operations series, which covers the unglamorous internal systems where this kind of work tends to pay off first. A document hygiene audit is a good first project precisely because it is measurable: you can count the links you closed and the gigabytes you retired.

PicNet builds production AI systems for Australian organisations. Talk to us about what a first project could look like.

Originally published at picnet.com.au.

Supply-chain attacks: package cooldowns and policy checks for npm, PyPI and NuGet

Guido Tapia — Fri, 28 Aug 2026 00:50:59 +0000

On 4 August 2026, a self-propagating worm called ChainDrop tore through npm. BleepingComputer reported more than 1,300 compromised packages with about two billion monthly downloads between them, including keyv, cacheable, flat-cache and file-entry-cache. The poisoned releases carried a "preinstall": "node setup.mjs" entry, so anyone who ran npm install against an affected version executed the dropper before the install even finished. It collected environment variables, GitHub tokens, npm tokens, AWS credentials, Kubernetes secrets, Vault tokens and database credentials from developer machines and CI runners.

The detail that matters for how you defend against this: the attacker got in through a legitimate maintainer's compromised GitHub account, not a hole in npm or GitHub. The packages were built and published through their real GitHub Actions workflows, so the malicious releases carried valid provenance. Reputation checks, provenance attestation and "do we trust this maintainer" reviews all passed.

So "is this package reputable?" is the wrong question. The question is whether this specific version is safe, and nobody can answer that at install time. Wiz was tracking the keyv and cacheable compromise as an in-progress campaign, which means every organisation that installed during the window was exposed before any advisory existed to warn them.

This post is part of our Practical AI in Cyber Security series, and it covers the least glamorous control in it. There is no model here. Just two mechanical checks we run across every repository we touch.

Control one: the cooldown window

Never install a package version younger than N days. We use 14 days as the default.

The logic is boring and that is the point. Nearly all of these compromises get detected and the malicious versions yanked within days of publication. The attacker's whole business model is speed: publish, get pulled into as many downstream lockfiles as possible, harvest credentials before anyone notices. A time delay between publication and installation closes exactly the window the attack depends on. You do not need to detect anything. You just need to arrive late.

Fourteen days is not a number we invented. Security architects at a multi-national audit firm, after handling two supply-chain attacks themselves, gave guidance of exactly two options: pin your versions, or only accept dependencies older than two weeks. We run both together.

GitHub has moved in the same direction. Dependabot version updates now wait three days by default before opening a pull request, and security updates stay exempt so real fixes are not delayed. Three days is better than nothing. We think two weeks is a better fit for business systems, where a fortnight's delay on a minor version bump costs nothing.

How you implement it depends on the ecosystem:

npm/pnpm: minimumReleaseAge in pnpm-workspace.yaml (20160 minutes for 14 days), plus cooldown.default-days: 14 in .github/dependabot.yml
PyPI: enforced at the resolver or proxy layer, since pip has no native age gate. If you run an internal index, the age check belongs there
NuGet: same pattern, enforced through your feed rather than the client

Two things about the cooldown are easy to get wrong. First, exclude your own first-party packages, otherwise you cannot ship your own code for a fortnight. Second, the gate governs lockfile regeneration, not installs from an existing lockfile. CI running --frozen-lockfile is unaffected. That is correct behaviour: the regen moment is the attack surface, because that is when a floating range silently resolves to something new.

Control two: policy checks on every manifest

The cooldown handles timing. Policy checks handle everything else. We look for four things.

Lockfiles are required and committed. No lockfile means every build resolves fresh, which means every build is a roll of the dice against whatever was published in the last hour.

Install scripts are disabled. ChainDrop ran through preinstall. Attackers use install-time scripts because they fire immediately, before the package's code is ever called at runtime. npm v12 disables install scripts by default with an approval list for the packages that legitimately need them, and disables git and remote-URL dependencies too. Do not wait for the version bump to force it. Set it now.

Floating versions are flagged. A ^ or ~ range is a standing authorisation for code that does not exist yet and nobody has reviewed. The public pull request implementing this guidance is instructive: the project had 37 of 39 direct dependencies on flexible specifiers, with a frozen lockfile in CI as the only protection. Pinning direct dependencies to exact versions pushes every future bump into a reviewable package.json diff instead of burying it in a lockfile where a poisoned transitive bump is easy to skim past.

Source mapping. Every dependency maps to a declared upstream source. When an advisory lands, the first question is always "where do we have this, and in which versions?" If answering takes three days of grep, you have already lost the response window.

One extension worth noting: a Python project template hardened in August 2026 added dependency scanning on the principle of treating your scanners as untrusted software. The tools you install to inspect dependencies are themselves dependencies. Same cooldown, same pins, same install-script rules. Your scanner does not get an exemption.

Why this is monitored, not audited

Two campaigns of this scale landed inside a single week in August 2026. An annual dependency audit misses both. A quarterly one probably misses both. The exposure window in the keyv/cacheable case was measured in hours.

So we run recursive discovery across every repository in an organisation: walk the tree, find every package.json, pnpm-lock.yaml, requirements.txt, pyproject.toml, poetry.lock, *.csproj and packages.lock.json, including the ones inside sample directories, archived services and that internal tool one person maintains. Then apply the policy checks to all of them on a schedule and report the drift.

The unglamorous finding is almost always the same. Organisations know their main product repositories reasonably well. They do not know about the twelve other repositories with manifests in them, and the abandoned build pipeline whose runner still holds a valid deployment credential is a better target than the flagship app.

Honest limitations

A cooldown does not help against a slow attacker. If a malicious version sits undetected for three weeks, 14 days buys you nothing. It also does not protect you against a compromise you install on day 15 because no advisory ever appeared. These controls raise the cost and shrink the window. They do not close it.

Pinning has a real cost too: you now own your upgrade cadence. Pinned dependencies do not update themselves, and a project that pins without a maintenance rhythm ends up on ancient, genuinely vulnerable versions, which is a worse position than where it started. Budget for the upgrade work.

There is a per-upgrade review practice emerging as a complement, reading the actual diff of a package upgrade before accepting it, which is where AI assistance is starting to be useful in ecosystems without strong registry-side attestation. It is early, and it does not replace the cooldown.

And if you ran an affected install, none of this is remediation. Treat the workstation or CI runner as compromised even if the package was removed: rebuild, rotate every token reachable from that environment, and check logs and repositories for unexpected access and commits.

The rest is cheap. Two config settings and a scheduled job across your repositories will cost you a day of engineering time. Both of the August incidents would have been non-events for anyone running them.

PicNet builds production AI systems for Australian organisations. Talk to us about what a first project could look like.

Originally published at picnet.com.au.

An AI blog pipeline with a human editor: how this post was made

Guido Tapia — Wed, 26 Aug 2026 02:00:47 +0000

This post was drafted by a machine. A person read it, changed things, and merged it before you saw it. Since we spend this series describing how other organisations should apply AI to marketing work, it seemed reasonable to open up the pipeline that produces the series itself. This is part of our Practical AI in Marketing series, and it is the only post in it about us.

The backlog is a JSON file

There is no content calendar in a spreadsheet and no monthly planning meeting. Topics live in a JSON file in the same repository as the site, and each entry carries enough context for an agent to write from it:

{
  "slug": "an-ai-blog-pipeline-with-a-human-editor-how-this-post-was-made",
  "series": "practical-ai-in-marketing",
  "title": "An AI blog pipeline with a human editor: how this post was made",
  "notes": "Transparency meta-post. Backlog JSON, weekly research, humanised draft on a review branch, LinkedIn copy alongside, human edits and merges. Human gate is the point. Costs per post vs agency.",
  "status": "drafted"
}

The notes field does most of the work. It is written by a person who knows what we actually do, which is why the pipeline can produce a post about our own process without me having to write the process down twice. Anyone at PicNet can add an entry. Adding one takes about two minutes, which matters more than it sounds: the old bottleneck was not writing, it was that good topic ideas evaporated between the conversation where they came up and the next time anyone opened the blog folder.

What runs each week

The weekly job is deliberately boring:

Pick the next backlog item that is ready.
Run web searches on the topic, scrape the promising results, and write everything into a research file next to the draft. Sources, notes, dates, full text where we can get it.
Draft the post against a system prompt holding the house rules: word count, Australian English, byline, what may and may not be claimed, which client names are cleared for public use and which are not.
Run a humanising pass that strips the stock AI phrasing.
Draft the LinkedIn copy from the same research, in the same run, so the social post and the article do not drift apart.
Commit the draft, the research file and the social copy to a review branch and open it for review.

Nothing in that list touches the live site. The last step is a person reading the branch and merging it, or not.

Putting content through git rather than a CMS was the single best decision in the build. Every post has a diff, a branch and a merge, and every claim in it can be traced back to the research file sitting beside it in the same commit. Teams are already reworking their software lifecycle around agent output (Ask HN, August 2026), and the review discipline transfers to a content repository without modification. The editor is doing a code review on prose.

Why the human gate is the point

The clearest statement of the problem I have read comes from someone who built a marketing AI system for a 60-person team: "generation is free now... that sounds like good news, and for about three weeks it was." Their conclusion after the build was that the most valuable layer in the system is the one that refuses to write. Once anyone can produce a competent blog post in ninety seconds, the constraint moves to the part of the process nobody had staffed, which is review and editorial judgement.

Engineering hit the same wall first. Sylvain Kalache's framing, AI writes the code, but humans can't review it all, describes teams generating ten times faster while reviewing at human speed, and cites survey data showing high-adoption teams merging 98% more pull requests while review time rose 91%. Content has the same shape and worse consequences, because a bad merge in a repository fails in staging while a bad merge on a public blog is read by your clients.

So we bounded the volume rather than the generation. The pipeline is capped at what one editor can genuinely read in a week, and the editor's job is not to tidy sentences. It is to answer three questions: is every statistic in here actually in the research file, is every client reference one we are allowed to make, and would I say this out loud to a client. Posts that fail the third question get deleted rather than fixed. That happens more often than the volume-obsessed version of this pipeline would tolerate, which is the point.

The prompt also carries a list of things the model must never write: client names that are not cleared for publication, any claim we cannot source, any statistic without a link. The editor checks the list on every branch. Prompts are guidance, not controls, and treating them as controls is how organisations end up apologising publicly.

Labelling what the machine wrote

Beau Cronin's TAIL scale, published on 18 August 2026, gives a 0 to 5 rating for how much of a piece AI wrote. TAIL 4 is "AI substantially authored, with my comprehension, review, and editing", and Cronin describes it as the last level that is meaningfully human in the loop. That is where this post sits, and where the series sits. It is a more honest label than a generic "AI-assisted" footer, because it tells you exactly which half of the work a person did.

Publishing the mechanics is becoming its own small genre, including technical blogs disclosing their own generation process. We would rather be in that group than pretend the drafts are typed by hand.

One thing the humanising step is not for: beating detectors. AI writing detectors such as Pangram are now deployed everywhere and, as the Washington Post reported on 25 August 2026, they aren't always accurate. Their reporting notes a papal encyclical was flagged as partly AI-generated. A detector score is not a quality signal, and optimising a draft to pass one produces text that is neither honest nor better. We run the humanising pass because unedited model prose is full of sales phrasing and stock triads that no engineer would say in a meeting.

What it costs

Per post, the machine spend is search API calls plus model tokens, and it rounds to nothing against an hour of senior time. The real cost is the editor: roughly an hour per post to read the draft against the research file, cut what cannot be supported, and fix the parts where the model wrote something plausible and wrong.

Compare that with an agency retainer, where you pay for both the writing and the research, and where the person writing about your systems has never seen them. The pipeline flips the ratio. We pay almost nothing for the drafting and spend the money on the judgement, which is the part an agency cannot do for us anyway because it depends on knowing what we have actually built.

Volume is not the win, and it is worth being sceptical of pipelines sold on volume. A twelve-month practitioner review of AI content platforms starts from a B2B SaaS blog with 12 published articles and basically zero traffic, which is a fair reminder that publishing more of nothing produces more nothing.

Honest limitations

The pipeline cannot interview anyone. It cannot visit a client site, sit through a workshop, or notice the thing a client said offhand that turns into a good post. Everything original in this series comes from a person putting it in the backlog notes first, and the quality of a post is capped by the quality of those notes.

It also gets things wrong in a specific way: it will produce a confident sentence built on a source that does not quite say that. That is exactly what the research file and the branch review exist to catch, and it is why the editor's hour is not negotiable.

If you want to build something similar, start with the review step. Decide who reads every piece, how long they get, and what they are allowed to reject. Then automate backwards from there. The generation was never the hard part.

PicNet builds production AI systems for Australian organisations. Talk to us about what a first project could look like.

Originally published at picnet.com.au.

The robot that chases people: automated reminders, stale quotes and follow-ups

Guido Tapia — Mon, 24 Aug 2026 00:51:03 +0000

Most operations teams have a version of the same list. Quotes sent three weeks ago with no answer. A job waiting on a client document since before the last public holiday. A renewal that needed a conversation in June and is now awkward. Nobody decided to drop these. They fell through a crack, and nobody was paid to watch the cracks.

The usual fix is a person: an ops coordinator with a spreadsheet and a Monday morning ritual. That works until they take leave. The better fix is a scheduled task with an opinionated data model behind it, plus a language model doing the one part that genuinely needs judgment, which is writing a chase message that does not read like it came from a machine.

This is part of our Practical AI in Business Operations series, and it is deliberately the least glamorous entry. There is very little AI in it. That is the point: the value comes from the scheduler, and the model earns its place at the last step.

The table is most of the trick

A working reminder system is smaller than people expect. An open design discussion on a tender tracking project sketches a reminders table keyed on the record id plus days_before, sent and sent_at, swept by a daily cron. The state columns are what stop the same nag firing twice. That is the whole mechanism.

Ours look roughly like this:

create table reminders (
  id                uuid primary key,
  entity_type       text not null,          -- quote | job | invoice | renewal
  entity_id         uuid not null,
  milestone         text not null,          -- first_follow_up | stale_quote | decision_chase
  days_before       int,                    -- exactly one of these two is set
  remind_on         date,
  due_date          date not null,          -- computed; the only thing the sweep looks at
  assignee_id       uuid not null,
  snoozed_until     date,
  suppressed_reason text,                   -- an honest reason to stop chasing
  sent              boolean not null default false,
  sent_at           timestamptz
);

The sweep is one query, run daily:

select * from reminders
where due_date <= current_date
  and not sent
  and (snoozed_until is null or snoozed_until <= current_date)
  and suppressed_reason is null;

Note due_date <= current_date, not = current_date. Exact date equality means one bad deploy day silently drops every reminder that fell on it. The same ticket works through that and two related traps, and lands on catch-up rather than skip, with reminders whose milestone has already passed suppressed in favour of a louder "this was missed" state. It also resets sent when a deadline moves: a write-once boolean means pushing a due date back permanently silences the record.

Timezones matter more here than they look. An organisation with staff in Perth and Sydney has a two or three hour window where "today" is two different dates. Pick an org-level timezone, compute due_date in it, and stop treating server time as the answer.

Deciding when it fires is the same decision as deciding how someone hears about it

The tender ticket treats notification delivery and the reminder model as one design question, not two, and that is right. A reminder that broadcasts to a shared channel has different semantics from one that reaches an individual. If your only channel posts to a group, then every reminder is public, and that constrains the content: no pricing, no margin, and a hard look at whether client names belong in a channel that may include contractors. Under the Australian Privacy Principles, "who can see this notification" is a design input, not a compliance review item at the end.

Targeting also decides whether the system survives. Reminders that ping people who have already done the work train the whole team to mute the robot within weeks. Chase the assignees with nothing recorded against the milestone, not everyone attached to the record.

Escalation is a schedule, not a mood

Escalation should be boring and configurable. For a client submission deadline, defaults of seven days, three days, one day and the morning of the due date work well. Two rules make it useful rather than noisy:

A missed item does not drop off the list. It stays on the dashboard in a loud state until someone resolves it, and it posts once to the team channel when it is missed.
Escalation changes audience, not just frequency. The third reminder goes to the assignee's manager. If your escalation only shouts louder at the same person, it is not escalation.

For stale quotes, the trigger is different but the shape is the same: days since sent with no client response, days until the quote's validity expires, and a hard stop at 30 June for anything priced against a financial year.

Snooze, and the re-arm problem

Snooze is where these systems go wrong, and the failure is not the trigger. It is when the system decides to re-arm after a human acts.

There is a neat illustration of this from an unrelated field. A robot's heading-hold controller was counter-steering after a manual turn: the operator would turn, recentre, and the robot would swing back the other way. The hold released during the turn and re-latched at the wrong point, against a target the machine had already rotated past. The fix was to shorten the hold's memory so the target caught up with reality faster.

Reminders have the identical bug. Someone rings the client on Tuesday and snoozes for a week. On Wednesday the client replies and the quote status changes. If the snooze re-arms against the old due_date, the reminder fires next Tuesday about a conversation that is already finished, and the person who did the right thing gets nagged for it. Recompute due_date on every state change to the underlying record, and let the snooze expire against the recomputed value. The rule is simple: the reminder chases the current state of the record, never the state it had when the row was written.

Give people an honest way to stop the nag

The most valuable column in that table is suppressed_reason. Without it, "hasn't got to it" and "this cannot be actioned" look identical to the scheduler, and it chases people forever for work that is impossible. The tender ticket adds an explicit "no supplier found" record for exactly this reason. Ours are things like "client asked us to hold until the new financial year" or "waiting on their legal, no date". Each one silences the reminder and leaves a written trace of why, which is far more useful in a Monday meeting than a snoozed row.

Where the model actually goes

Two places, both narrow.

The first is classification. A rule that decides whether a quote is stale by looking at a status field alone will get it wrong, because status fields lag reality. Reading the actual email thread and the file notes is a better signal, and that is a job a model does well. One caution: read the whole record, not a sample. A scheduled job in another project fetched a partial version of its source text for years, and the correction was explicit, to pull all of the text rather than a sample of it. Reminder rules that peek at only the last two emails will confidently call a dead quote fresh.

The second is drafting the chase message. The model gets the client name, what was sent, when, what has happened since, and two or three examples of how that account manager actually writes. It produces a draft. A person reads it, changes a line, and sends it from their own mailbox.

We do not auto-send these, for two reasons. Clients can tell, and an unsupervised bot emailing your customers about commercial matters is a Spam Act 2003 problem waiting to happen rather than a productivity gain. The point of the automation was never to write the email. It was to make sure someone was asked to.

Design every state before you build

Alert preferences deserve real screens, not a config file. One mobile build we looked at gives alerts a dedicated /settings/alerts route alongside profile and saved items, and enumerates every screen at every lifecycle state for review. Copy that discipline. Pending, due, sent, snoozed, escalated, missed, suppressed and dismissed are eight states somebody has to design, and the ones teams skip are always suppressed and missed, which are the two that decide whether people trust the thing.

Costs and honest limits

The model calls are the cheapest part. A few hundred short drafts a month is a rounding error next to the engineering time, and most of that time goes into failure semantics rather than prompts. Budget for a small build with a fortnight of tuning the thresholds afterwards, because the first set of defaults will be wrong.

The real limit is consumption. Scheduled automation whose output nobody acts on is worse than none, because it costs attention and buys nothing. That same project found an extraction job that had run against roughly twelve articles a year, was compared against a model's reading, "and then used for nothing else". The author's verdict was blunt: that is backwards. Before you build the chaser, decide who reads its output and what they do next. If the answer is thin, build the daily digest instead and stop there.

PicNet builds production AI systems for Australian organisations. Talk to us about what a first project could look like.

Originally published at picnet.com.au.

Cutting vulnerability noise: using AI and public data to demote CVEs that don't matter

Guido Tapia — Fri, 21 Aug 2026 00:56:05 +0000

A container image scan finishes and hands back a few hundred findings. A dozen are critical. Most of the rest are base image packages with a CVE ID, a CVSS score, and no patched version anywhere in the world. The team has one afternoon a fortnight for this work. Where do they start?

Usually they start by scrolling, and that is the whole problem. When a report is 90% unactionable, people stop reading it, and the one finding that did have a fix available gets the same amount of attention as the 200 that did not. Noise is not just annoying. It is the mechanism by which real vulnerabilities get missed.

This post is part of our Practical AI in Cyber Security series, and it covers something we run for clients rather than something we think would be nice: cross-referencing scanner output against public vulnerability data so that findings with no available fix get demoted automatically, and findings with a released fix stay loud.

What "no fix available" actually means

Scanners report what a package version is vulnerable to. They are much less consistent about whether anyone has shipped a repaired version.

The distinction matters because the two cases need completely different responses. If Debian, Alpine, Ubuntu or Red Hat has published a fixed package, the answer is to bump the version and rebuild, which is an hour of work. If the upstream maintainer has marked the issue as won't fix, disputed, or minor, or the affected code path is not compiled into the distro build at all, there is nothing to install. The only available responses are compensating controls, removing the package, or accepting the risk and writing it down.

Both arrive in the report looking identical. A CVSS 7.5 with a fix and a CVSS 7.5 without one sit next to each other, sorted by severity, as though they are the same kind of work item.

The pipeline

The approach is deliberately boring. Deterministic data does the demotion, and AI is confined to the parts that are genuinely unstructured.

Normalise scanner output into one record per finding: CVE ID, package name, installed version, ecosystem, image or host, and where it runs.
Query OSV.dev for each package and version. OSV aggregates advisories across language ecosystems and Linux distributions and returns affected version ranges, including whether a fixed version exists for that specific distro build.
Apply the rule. A finding with a fixed version in the ecosystem the package actually came from stays at its original priority. A finding with no fixed version anywhere, and no runtime exposure flag, drops to a demoted queue that gets reviewed monthly rather than fortnightly.
Everything demoted keeps its full record, with the reason, the advisory that justified it, and the timestamp of the data the decision was made on.
Feed the promoted findings into the ticketing system the team already uses, one ticket per rebuild rather than one per CVE.

For clients with several environments, the fetch and reconciliation runs as a scheduled integration job. We use Centazio, our open source data integration platform, for this sort of thing, because the hard part is not the API call, it is keeping the state consistent across repeated runs and knowing which records are stale.

Where AI helps, and where we keep it out

The demotion decision itself is not an AI decision. Version comparison is a solved problem and it should stay deterministic, because a language model that occasionally hallucinates a fixed version is worse than no enrichment at all.

Three things in this pipeline are messy enough that a model earns its place. The first is advisory prose. Distro advisories and upstream issue trackers explain in free text why something is not being fixed, and a model summarising twenty of those into "these six are all the same unfixed glibc locale issue" saves real reading time. The second is clustering: grouping 180 findings into the eight base image rebuilds that would resolve them. The third is drafting the risk acceptance note, which a human then edits and signs.

Note the pattern. The model reads, groups and drafts. It never decides on its own that something is safe to stop looking at. Practitioners are still asking each other what tooling exists for reviewing AI-assisted work in engineering pipelines, with no settled answer as of August 2026, so designing around a verification step rather than trust is the sensible default. It is also worth being explicit with stakeholders about where the model sits, because audiences in 2026 scrutinise disclosed AI use closely, and a security control nobody trusts does not get used.

Fail noisy, not fail quiet

The dangerous failure mode is obvious once you name it: the enrichment source is unreachable, every lookup returns nothing, "no fixed version found" is indistinguishable from "no answer received", and the whole report quietly demotes itself to zero. The dashboard goes green on the day it should go red.

So the pipeline distinguishes the two cases at the type level and treats an unavailable source as a check failure:

def triage(finding, osv):
    try:
        advisory = osv.lookup(finding.cve, finding.package, finding.ecosystem)
    except (TimeoutError, HTTPError) as e:
        raise EnrichmentUnavailable(finding.cve) from e   # alerts, does not demote

    if advisory.age > MAX_CACHE_AGE:
        raise EnrichmentStale(finding.cve, advisory.age)

    if advisory.fixed_versions:
        return Priority.ORIGINAL
    if finding.runtime_exposed:
        return Priority.REVIEW
    return Priority.DEMOTED

Two rules follow from that. Findings that could not be enriched keep their original severity and are flagged as unenriched, so a broken run produces a louder report than a healthy one. And the cached advisory data carries a maximum age, because silently triaging today's images against a three week old snapshot is the same bug wearing a disguise.

The economics for a lean team

Most Australian organisations we work with have two or three people covering infrastructure, patching and everything else. Their real constraint is rebuild cycles, not knowledge. They know the base image is old.

The value of demotion is that it makes the remaining queue small enough to finish, which is what turns patching into a habit instead of a quarterly panic. The ACSC's Essential Eight expects patches for applications and operating systems to be applied within defined windows and expects you to demonstrate it. A queue of fifteen fixable findings can be closed and evidenced. A queue of three hundred, mostly unfixable, cannot, and the auditor sees the same wall of red the engineers gave up on.

Honest limitations. Version matching across distros is fiddly, because backported patches leave the version string looking unfixed when the vulnerability is already gone, so expect a tuning period and some manual overrides. Demoted is not deleted: the monthly review exists because a package with no fix today may get one next Tuesday, and a re-query catches that. Runtime reachability, which would let you demote far more aggressively, needs instrumentation most teams do not have, so we treat exposure as a coarse flag rather than a proof. And the LLM step costs a few dollars a month at typical volumes, which is nothing next to the engineering time to set the pipeline up and keep the version comparison honest. That build effort is the real cost, and it is worth it only if someone is going to work the shortened queue.

PicNet builds production AI systems for Australian organisations. Talk to us about what a first project could look like.

Originally published at picnet.com.au.