DEV Community: Hamze Zare

Phishing That Survives MFA: What Microsoft's 2026 Digital Defense Report Means for Company Email

Hamze Zare — Sat, 03 Oct 2026 01:11:29 +0000

Microsoft published its 2026 Digital Defense Report on 1 October. Most of the coverage went to AI-run attacks and ransomware. The part that matters most for anyone who runs company email is quieter, and it sits on pages 33 and 45 of the PDF: phishing is back as a main way in, and the typical phishing page is no longer a fake login form that just steals a password.

This piece pulls out the email findings, explains what they mean in practice, and is honest about what a mail platform (ours included) can and cannot do about them.

What the report actually says about phishing

The report covers July 2025 to June 2026. In Microsoft's incident response cases, phishing was the initial access vector in 23% of intrusions, up from 7% the year before. Over the same period, the share of cases with no identified entry point fell from 25% to 14%, so part of that jump may be better visibility, but the direction is clear.

The bigger change is the kind of phishing. Adversary-in-the-middle (AiTM) kits now make up 44.6% of identified phishing techniques, against 33.6% for standard URL phishing and 12.9% for attachments. An AiTM page is a reverse proxy. The victim sees the real sign-in page, types a real password, approves a real MFA prompt, and the proxy keeps the session cookie that comes back. On the same page Microsoft reports that 87.7% of phishing intrusions involved credential or session harvesting.

And once an attacker has one working account, they go looking for more. Of intrusions that started with valid accounts, 52.2% involved follow-on credential theft.

Why a six-digit code no longer saves you

Here is the uncomfortable part. SMS codes, authenticator-app codes and push approvals all stop the old attack, where someone buys a leaked password and logs in from another country. None of them stop AiTM. The proxy simply forwards whatever the user types, code included, to the real server in real time. The user did everything right and still handed over a live session.

What does stop it is authentication that is tied to the website's address: FIDO2 security keys and passkeys. The browser will not sign a challenge for a look-alike domain, so there is nothing useful for a proxy to relay. Microsoft's own recommendations in the report say the same thing in plainer words: move beyond traditional MFA and prioritise phishing-resistant methods.

We think that is right, and it applies to us. FanMail offers SMS one-time passwords and TOTP authenticator codes as a second factor. They are a big improvement over passwords alone, and for many organisations they are the only second factor people will actually enrol in. They are not phishing-resistant, and we would rather say so here than have a customer learn it from an incident report.

The delivery tricks that get past filters

Getting the victim to the proxy page is the other half, and the report shows attackers rotating formats faster than filters adapt:

QR codes in PDFs. Microsoft Defender for Office 365 detected more than 145 million QR code phishing attacks, and by April 2026 PDFs carried 79% of them. The user opens the PDF on a work laptop and scans the code with a personal phone, which sits outside every corporate control.
CAPTCHA gates. On the same page, more than 100 million phishing attacks put a CAPTCHA in front of the credential page, which keeps automated scanners from ever seeing it.
Device code phishing. The lure sends people to the real Microsoft device sign-in page and the attacker collects OAuth tokens afterwards. No fake page at all, so there is nothing to block by URL.
Fake help desk. A burst of junk email, then a chat message from a new account posing as IT support offering to fix it. That one is aimed at your people's trust in their own support team.

We wrote earlier about how a PDF can carry script into a help desk or file share, in our note on hardening PDF attachment downloads. The QR-in-PDF trend is a different attack through the same door: a file format everyone treats as safe.

Five things to do this quarter

Start passkeys or FIDO2 keys with the accounts that matter most. Admins, finance, HR and anyone who can approve payments. You do not need the whole company on day one.
Shorten and watch sessions. AiTM steals a session, not a password, so long-lived sessions and silent token refresh are what make it pay. Alert on the same session appearing from a new network.
Treat QR codes in attachments as links. If your mail filter cannot read a QR code inside a PDF, assume users will scan it with a phone that has no protection.
Tell staff how the real help desk contacts them. One sentence in onboarding, repeated every few months: support never asks you to install a remote tool from a chat message.
Keep SPF, DKIM and DMARC strict on your own domains. It will not stop a look-alike domain, but it stops the cheapest version of the attack, a spoof of your exact address.

Where FanMail fits, and where it does not

FanMail is a self-hosted mail platform built on Stalwart Mail Server. It can help with parts of the list above: outgoing mail is signed with DKIM and checked against SPF and DMARC, the message reader sanitises HTML and can block external images, and users can turn on SMS or authenticator-app two-factor login. Because it runs on your own servers, the server logs are yours to query when you want to check where a session has been used.

What it does not do is make SMS or TOTP codes phishing-resistant, and no mail server can. If AiTM is your main worry, put phishing-resistant authentication in front of email, whatever mail product you use. For more on why we built SMS login in the first place, see our notes on SMS OTP, and if you are comparing hosted options, our FanMail, Google Workspace and Zoho Mail comparison covers data control and cost.

Originally published on fanpino.com.

The UAE Wants Half Its Government Services on AI Agents. What Should a Company Help Desk Copy?

Hamze Zare — Fri, 02 Oct 2026 00:03:52 +0000

In April 2026 the UAE said it wants half of its federal government services, sectors and operations running on agentic AI within two years. In August it published the design guide that tells federal entities how to get there. Most of the coverage focused on the AI part. The part we found more useful for anyone running a service desk in the Gulf is a much older idea, stated very plainly: the customer should not have to work out who is responsible for their problem.

That idea does not need an AI agent to work. It needs a help desk that is set up properly. This piece goes through what was announced, what the latest Gartner customer service numbers say, and what a private company serving Gulf customers can copy now, with or without AI.

What the UAE actually announced

Three announcements matter here, in order.

23 April 2026. Sheikh Mohammed bin Rashid announced that half of federal government services will be powered by AI within two years, with Sheikh Mansour bin Zayed overseeing the programme and a task force led by the Minister of Cabinet Affairs. His line was: "AI is no longer a tool. It analyses, decides, executes and improves in real time."
20 May 2026. The first batch of AI agents went into use in tax auditing, procurement, customer happiness and technical support.
21 August 2026. The National Committee for the Agentic AI Project released a unified guide for designing government services around AI assistants. The idea is that the assistant understands the request, works out which entity handles it, coordinates between entities and does the steps, so the customer stops repeating the same information to different offices.

One sentence from the August guide is worth reading twice: customers are "ultimately interested in obtaining a result rather than navigating a series of procedures." Success is measured by completed outcomes, not by how many transactions were processed. That is a change in what gets counted, and it is the bit most companies skip.

Early numbers from one ministry

The Ministry of Finance published its call centre figures for the first half of 2026 after adding generative AI, sentiment analysis on calls and a "chat with documents" feature. It reported 97.11% first contact resolution against a 90% target, an average speed of answer of 8 seconds against 20, an abandonment rate of 1.88% and average handling time of 4 minutes 35 seconds.

These are the ministry's own numbers and a call centre answering questions about one ministry's services is a narrower job than a company help desk that also handles faults, refunds and complaints. Still, they show where the bar is moving for anyone whose customers deal with UAE government services every week. People compare your support to the last good experience they had, not to your competitors.

What customers say they want

Two Gartner surveys from August 2026 put the hype in context.

The first, of 3,566 B2B and B2C customers surveyed in February and March 2026, found that 87% say a company using GenAI for service must give them a way to reach a human. Half said GenAI made their interactions easier. Gartner's advice was blunt: do not make the AI a mandatory first step for every issue, because customers forced through several failed bot turns are less likely to use it again. Eric Keller, a senior director analyst at Gartner, told CX Dive that customers split roughly in half between those who like self-service and those who want a person.

The second, of 199 service and support leaders surveyed in April and May 2026, found AI spending up 38% while total service budgets grew only 2%, with about 13% of the service budget now going to AI. The money is coming out of labour and overhead.

Put those together and you get the risk. Budgets are moving from people to bots at the same moment customers are saying the human exit is non-negotiable. We wrote about the gap between "deflected" and "resolved" in an earlier piece on AI ticket deflection. The UAE guide, read carefully, sides with resolution.

What a Gulf service desk can copy now

None of this requires an autonomous agent. It requires the plumbing an agent would need anyway. If the plumbing is wrong, an AI on top will just be wrong faster.

1. One front door

The customer should not choose between "billing", "technical" and "sales" before they can ask anything. One intake form or channel, then routing by category, branch or keyword to the team that owns it. If you still run separate inboxes per department, that is the first thing to fix. Departments outside IT are often the worst at this, as we covered in why those departments still run on phone calls.

2. Context travels with the ticket

When a request moves from first line to a specialist, the full thread, attachments and internal notes move with it. The customer is never asked for their contract number twice. This is the "no repeated submissions" principle from the August guide, and it is a data model question before it is an AI question.

3. A visible way to a person

If you add a bot, the escalation to a named team has to be one step, not a hidden option after three failed answers. Log every handoff, so you can see which topics the bot keeps failing on.

4. SLAs on your real calendar

The UAE federal government moved to a Saturday and Sunday weekend in 2022. Saudi Arabia still works Sunday to Thursday. A company with customers in both, plus Ramadan hours and public holidays, gets wrong due dates from any SLA timer that counts clock hours. Due dates should follow each branch's working calendar.

5. Count outcomes, not tickets closed

Track reopen rate and time to actual resolution next to volume. A ticket closed by a bot and reopened two days later is not a win, and a dashboard that counts it as one will push you in the wrong direction.

6. Arabic and English in the same ticket

In the Gulf a customer writes in Arabic and the specialist reads English, or the other way round. System messages, statuses and notifications should show in each reader's own language, with right-to-left layout that actually works, not a mirrored afterthought.

Where AI fits after that

Once the intake, routing and history are clean, AI has something solid to stand on: summarising long threads, suggesting a category, drafting a reply from the knowledge base. Give it narrow permissions and keep a human approving anything that changes money or account state. Our guide to agentic AI governance in help desks goes through those controls in more detail.

The order matters. The UAE did not start with a chatbot. It started by deciding that the customer's result is the unit of work and that the government, not the citizen, carries the job of finding the right office. A company can make the same decision this quarter.

Where FanDesk fits

FanDesk is our help desk, and it covers the layer underneath: one intake with branch, department and team routing, threaded tickets with internal notes and attachments, SLA due dates calculated against business hours and holidays, a knowledge base, and notifications by email, SMS and Telegram. The interface runs in Arabic, English and Persian with proper RTL, and ticket history renders in each reader's language. It is not an autonomous AI agent and we do not sell it as one. It runs as a cloud subscription or on your own servers. You can see the details on the FanDesk product page.

Our honest view: most Gulf companies we talk to are closer to step one than step six. Fix the front door first. The AI can wait a month.

Originally published on fanpino.com.

EU CBAM for Gulf Steel Fabricators: The Data You Owe Before September 2027

Hamze Zare — Thu, 01 Oct 2026 00:04:24 +0000

If you fabricate structural steel in the Gulf and any of it ends up in the EU, this September matters more than it looks. The European Parliament is voting on extending the Carbon Border Adjustment Mechanism (CBAM) to roughly 180 to 450 more steel- and aluminium-heavy products, depending on whose list wins. And 12 months from now, on 30 September 2027, EU importers file the first CBAM declaration that comes with a bill attached.

Most of the Gulf commentary on CBAM is about aluminium smelters and steel mills, because that is where the tonnage is. Coral's analysis puts UAE exports of CBAM-covered goods at about US$2.7 billion in 2023, with aluminium making up 68 to 75 percent of covered exports for the UAE, Oman and Saudi Arabia. Fabricators get less attention. They shouldn't, because one of the product codes already inside CBAM is exactly what a fabrication shop ships.

Fabricated structures are already in scope

CN code 7308 covers structures and parts of structures of iron or steel: bridge sections, towers, lattice masts, roof frameworks, columns, and "plates, rods, angles, shapes, sections, tubes and the like, prepared for use in structures". It is listed in Annex I of the CBAM Regulation (EU) 2023/956 alongside the mill products of chapter 72, and guides to the chapter 73 codes point out that for structural assemblies the upstream steel emissions flow through mass-balance rules, which is where it gets complicated.

So this isn't a 2028 problem for a steel structure fabricator. The transitional reporting period ran from October 2023 to the end of 2025. Since 1 January 2026 the definitive period applies, and EU importers of your 7308 goods are building up a certificate obligation for every tonne they bring in this year.

What changed in the last twelve months

The simplification package. Regulation (EU) 2025/2083 replaced the old per-consignment exemption with a single 50-tonne annual threshold per importer, required importers above it to hold "authorised CBAM declarant" status, and moved the first annual declaration and certificate surrender to 30 September 2027, covering 2026 imports.
The money is small this year, then it isn't. Because about 97.5 percent of EU free allocation still applies in 2026, only around 2.5 percent of the gross carbon cost is payable on 2026 imports, according to Coral's calculation. The same analysis notes that the mark-up on default values rises to 30 percent from 2028, and free allocation is scheduled to reach zero by 2034.
The downstream extension. On 17 December 2025 the Commission proposed adding 180 downstream products with an average steel or aluminium content of 79 percent, from 1 January 2028. The Council's general approach on 12 June 2026 went to about 200 goods; Parliament's environment committee voted on 9 July for a list of 457, according to the Parliament's research service. Trilogue follows the plenary vote.

For the new downstream goods, the Commission's text is explicit: only the emissions embedded in the steel and aluminium precursors count, not the emissions from downstream processing or assembly. Their example is a car door: CBAM applies to the steel plate inside it, not to stamping it. For iron and steel goods generally, CBAM counts direct emissions only; indirect emissions from electricity apply to cement and fertilisers.

Why the fabricator ends up holding the data problem

Read those rules from a fabrication shop's point of view and the conclusion is uncomfortable. Your own welding, cutting and painting barely register. Almost all the embedded carbon in a truss is the carbon in the plate and sections you bought. That means the number your EU customer needs from you is mostly a number from your mills, attributed to the right shipment.

The EU importer can always fall back on default values. The problem is that defaults are set high on purpose and the mark-up grows each year. An importer who can get verified, installation-level data from you pays less. An importer who can't may start preferring a fabricator who can, or pushing the cost difference back onto your price.

To give them actual values, you need to answer four questions for every shipment:

Which mills and installations produced the steel that went into these assemblies?
How many tonnes from each of them?
How much steel was consumed to make the shipped tonnage, not just how much was shipped? Offcuts and scrap are part of the precursor mass.
Does each mill have an emissions figure for that product, and is it verified?

The fourth question is between you and your mill. The first three are yours, and they are traceability questions. They are the same questions a quality auditor asks when a heat is quarantined, and the same ones behind the EU digital product passport for construction steel.

Where shops lose the thread

In most fabrication shops we have seen, the mill certificate is filed when the steel arrives and the heat number is painted on the bar. Then the bar gets cut. Six parts from one bar end up in four assemblies across two shipments, the offcut goes back to the rack with no marking, and three weeks later someone uses it on a different project.

At that point nobody can say, for a given shipment, how many tonnes came from which mill. You can estimate it from purchase orders for a whole project. You can't defend that estimate to a verifier, and it falls apart on any project that draws from mixed stock, which is most of them. We described the recall version of this problem in our piece on traceability and recall risk. CBAM is the same gap with a price per tonne on it.

What to put in place before the 2027 filing

Carry the heat number through cutting. Every cut part inherits the heat of the bar or plate it came from, and so does every offcut you keep. If this happens on paper today, it will fail the first time a job gets busy.
Record weights at shipment, per assembly. A shipment record that lists assemblies, their weights and the heats inside them lets you roll tonnage up by mill without re-measuring anything.
Track yield. Keep the ratio of steel consumed to steel shipped per job. Your importer's precursor mass depends on it, and it is also the cheapest way to see where offcuts disappear.
Ask your mills now. Find out which of them can give you installation-specific emissions for the products you buy, and in what format. Some Gulf mills are well ahead on this. Others will send you a sustainability report and hope that's enough.
Agree the handover with each EU customer. Who compiles the emissions communication, in what template, and by when. 30 September 2027 is a deadline for the importer; they will want your data months earlier.

Where Fidar MES fits, and where it doesn't

Fidar MES was built for structural steel shops, and the chain it keeps is heat number to mill certificate to material to part. A cut part inherits its heat. Offcuts above a set length stay in stock as usable remnants with their heat, and the nesting step records yield for each bar it plans. Shipments snapshot the weight of what is loaded, and each assembly still points back to its parts and their heats. Getting tonnage per mill for a shipment out of that is a query over records you already have, not a new data collection exercise.

What Fidar does not do: it doesn't calculate embedded emissions, it doesn't store your mills' emission factors, and it doesn't file anything with the EU CBAM Registry. Those belong to your mill, your importer and whoever they use for the declaration. Our part is making sure the tonnage-by-source answer exists and holds up when someone checks it.

If you're still working out what an MES should cover in a fabrication shop, start with what an MES is for steel fabrication. The regulation will keep moving through trilogue this autumn. The data you need for 2026 imports is being created on your shop floor right now, whether you record it or not.

Originally published on fanpino.com.

UAE E-Invoicing: What to Fix Before the 30 October ASP Deadline

Hamze Zare — Wed, 30 Sep 2026 04:51:23 +0000

If your UAE business turns over AED 50 million or more a year, 30 October 2026 is the date you must have an Accredited Service Provider (ASP) appointed for e-invoicing. Go-live is 1 January 2027. The Ministry of Finance already moved the ASP date once, from 31 July, and it said clearly that the January go-live did not move with it.

Most of the project plans we have seen treat the next four weeks as a procurement exercise: compare a few ASPs, sign one, done. Picking the ASP is the easy part. The ASP validates and routes what your systems send it. If your systems send the wrong thing, send it twice, or never find out it was rejected, the ASP won't fix that for you. This article is about the part that sits on your side of the connection.

Where things stand in September 2026

The legal basis is Ministerial Decisions No. 243 and No. 244 of 2025. Decision 243 sets the framework (scope, ASP role, data model, archiving). Decision 244 sets the phases and deadlines.
On 10 May 2026 the Ministry of Finance extended the ASP appointment deadline for businesses above AED 50 million from 31 July to 30 October 2026, kept full implementation at 1 January 2027, and reported 32 approved service providers at that point.
Businesses under AED 50 million appoint an ASP by 31 March 2027 and go live on 1 July 2027. Government entities follow on 1 October 2027. Voluntary adoption opened on 1 July 2026 (Andersen summary).
Scope is B2B and B2G. B2C is excluded for now, as are some financial services and airline passenger tickets (KPMG).
Penalties under Cabinet Decision No. 106 of 2025 start from each phase's go-live: AED 5,000 per month for not implementing, AED 100 per invoice sent late (capped at AED 5,000 a month), and AED 1,000 per day for not reporting a system failure or data change (VATupdate).

What actually changes

A PDF attached to an email stops being the invoice. The invoice becomes a structured XML document in the PINT AE format (the UAE profile of Peppol's international invoice), and it travels through a five-corner model. Boru Consulting describes the corners as: the supplier's source system, the supplier's ASP (validates against PINT AE and routes over Peppol), the buyer's ASP, the buyer's receiving system, and the Federal Tax Authority, which gets the tax data from the supplier's ASP.

Two consequences get missed. First, buyers need an ASP as well, so your accounts payable team changes how it receives invoices, not just how sales issues them. Second, the whole thing is only as good as the data your source system produces. A missing buyer TRN or a wrong VAT category no longer gets fixed quietly by someone editing a PDF. It gets rejected, and a rejected invoice that nobody notices becomes a late invoice.

List every system that issues an invoice, not just the ERP

The ERP vendor will usually have a connector. The trouble is the other places that bill customers. In the companies we work with, these are the usual ones:

a custom billing or subscription app that sends its own invoices,
an e-commerce or portal checkout that issues tax invoices for B2B buyers,
a service or maintenance module that bills contract work,
credit notes done by hand in a spreadsheet and then emailed.

Credit notes are covered by the same rules as invoices, and the Andersen summary notes that both must be issued within 14 days of the transaction. If credit notes live in a spreadsheet today, that is a process to rebuild, not a field to map. This inventory is also the moment to find the point-to-point links between systems that nobody has documented, the same problem we wrote about in why institutional systems don't integrate.

What our own payment integration taught us

We built the checkout on fanpino.com ourselves and connected it to an external payment gateway. It is not an e-invoicing ASP, but the failure modes are the same kind: your system sends a document to a third party, gets an ID back, and has to reconcile later. Two bugs we hit are worth repeating because an ASP integration will produce the same ones.

The gateway returned its reference ID as a JSON number. Our code expected a string. So every call failed on our side after the gateway had already created the payment. The user saw an error, and a real transaction existed on the other end. In e-invoicing terms, that is an invoice the ASP accepted while your ERP thinks the send failed. If the retry logic simply sends again, you have a duplicate invoice with the tax authority.

The second bug came from the same number. The callback looked the payment up using the ID as a string from the query string, while the database stored a number. That lookup would never match, so every paid order would have failed at confirmation. It showed up in testing only because we traced one request end to end.

What we changed, and what we would ask of any ASP integration:

Normalise IDs at the boundary, once, in the client that talks to the provider. Don't cast in five places.
Use your own invoice number as the idempotency key. Before any retry, ask the ASP for the status of that number instead of resending.
Treat "the ASP acknowledged it" and "the invoice was validated and delivered" as different states. Store the ASP message ID and each status change. A redirect or a 200 response is not proof of anything.
Have one screen or query that answers "which invoices from last week are not in a final state?" If that takes a developer, it won't get checked.

You have to notice failures to report them

The rules require technical failures to be reported to the FTA within 2 business days, and changes to registered data to be notified to your ASP within 5 business days (Andersen). Both assume you know something went wrong. A silent integration is the expensive case: the penalty for not reporting a failure runs daily.

So the integration needs logs you can still read after a restart, and an alert when rejections pile up. We learned the log part the hard way on an unrelated incident, where a container rebuild wiped the only copy of our access logs. The fix was small, and it's described in send container logs to stdout, not files.

Master data and where the data lives

Most early rejections will come from master data rather than from code. Check counterparty TRNs, item codes and the VAT category on each GL line before go-live, not after. Boru's checklist puts master-data cleansing next to ASP selection for exactly this reason.

Decision 243 also includes archiving and data sovereignty rules, and Andersen's summary states that e-invoice data must be stored within the UAE. Ask where every copy actually sits: the ERP if it is SaaS, the ASP, the email archive that still holds PDF copies, and the backups. This is the same question we get about mail servers, and the reasoning in why companies still self-host email in 2026 applies here too. Confirm the exact archiving period and format with your tax adviser; we are not giving tax advice here.

A four-week plan before 30 October

Week 1: list every system that issues invoices or credit notes, and every system that receives supplier invoices. Name an owner for each.
Week 2: shortlist ASPs on API quality as well as price. Ask for a sandbox, the status model, how duplicates are handled, and the error codes you will receive. Ask where their data is hosted.
Week 3: clean master data (buyer TRNs, item codes, VAT categories) and map one real invoice from each source system to PINT AE fields.
Week 4: sign the ASP, then send test invoices from each source through the sandbox, including one forced failure and one retry, and confirm you can see both in your own records.

After 30 October the work continues until 1 January, but by then the question should be how well the integration works, not which vendor to use.

Where Fanpino fits

We are not an ASP and we don't file anything with the FTA. What we do is the part on your side: connecting in-house billing apps, portals and service systems to the ASP you choose, with idempotent sending, status tracking and logs your finance team can read. If one of your invoice sources is a custom system nobody wants to touch, that is usually where we start.

Sources

Originally published on fanpino.com.

Malicious MCP Servers: What Deadbugz Taught Us About Auditing Our AI Agent Setup

Hamze Zare — Tue, 29 Sep 2026 03:43:37 +0000

On the evening of August 10, 2026, a single GitHub account opened 23 pull requests against unrelated AI and developer-tool projects in 74 minutes. Each one added an MCP server called productivity-suite to the project's config. It offered text formatting and summarization, and for the first three tool calls that is all it did. After the third call it changed the instructions it sent back to the AI agent, telling it to look for SSH keys, AWS credentials, shell history and Kubernetes config, and to keep quiet about it. Pillar Security published the details on August 12 and named the campaign Deadbugz.

We run AI coding agents with a stack of MCP servers every day, so we did the obvious thing and audited our own setup. It did not pass. This post covers why this kind of attack is different from the npm and PyPI incidents most teams already plan for, what our audit found, and the checklist we are now working through.

Why an MCP server is a different kind of dependency

A normal library does what its code says. You can read it, pin it, scan it. An MCP server does two things: it runs code on your machine or on someone else's, and it hands your AI agent text that the agent treats as instructions. Tool names, tool descriptions and prompt templates all come from the server, and the model reads them the same way it reads your request.

That second channel is what Deadbugz abused. According to Pillar, the server kept a per-client counter. Once a client had made three tools/call requests, later tools/list and prompts/get responses carried the new instructions. No new tool appeared, and nothing in the tool list looked wrong at install time. Someone trying the server for a minute would see a harmless text formatter. The payload arrived through metadata the agent was already trusting.

Pillar's advice to people building MCP clients is the part worth repeating: a change in the tool definitions of a server you already approved should be treated as a security event and need approval again. Most clients today do not do that. You approve a server once, and whatever it says afterwards goes straight into the model's context.

How exposed is everyone else?

This is not only a problem on developer laptops. Censys counted 12,520 MCP services reachable from the internet on April 28, 2026, spread across 8,758 IP addresses, and more than 21,000 by May 6. The servers in their report were reachable without authentication. The largest groups were data and knowledge tools (1,776, many of them direct database query interfaces) and infrastructure tools (1,549). Censys did not call the tools, so these counts show exposure, not confirmed compromise.

The vendors are reacting. On August 14 Cloudflare added MCP traffic detection to its Gateway, keyed on the MCP-Protocol-Version header, so a company can at least see which machines talk MCP and block direct connections that skip an approved portal. The protocol's own security best practices page is blunt about local servers: they run with the same privileges as the client, and a client that offers one-click setup must show the exact command, without truncation, before running it. It also recommends sandboxing servers with minimal default access to the file system and network.

Network visibility helps a security team. It does nothing for the question a developer should ask first, which is: what did I actually install, and what can it reach?

What we found in our own setup

Our development host runs Claude Code with seven MCP servers configured globally and two more scoped to single projects. We wrote about how we split work between agents in our RPI workflow post; this audit was about the plumbing under that workflow. Four findings mattered.

Three servers had no version pin. One was started with npx -y and a bare package name, one with @latest, and one straight from a Git repository's default branch through uvx. Each of those downloads and runs whatever is newest every time a session starts. For a Deadbugz-style change to reach us, nobody would need to send us a pull request. The upstream package would only need to change, through a hijacked maintainer account like the September 2025 npm phishing attack, or through a maintainer who changes their mind.

Four servers were remote HTTP endpoints. Their tool definitions live on someone else's server and can change at any moment without any update on our side. Pinning does not help there. The only defense is to notice when the definitions change, and we had no way to notice.

The biggest blast radius was not the shell. Our browser-automation server connects over the Chrome DevTools Protocol to a long-running browser profile that stays logged in to several of our business accounts. Anything that can steer that server can act as us on all of them. We had been thinking about SSH keys. The browser session was the more valuable target.

We had already been bitten by tool-name confusion. Earlier this month two Playwright MCP servers were active at once with overlapping tool names, one attached to that shared browser and one launching its own sandboxed Chromium. The agent kept picking the wrong one and we spent a long debugging session assuming the browser had crashed. Nothing malicious happened, but it showed us that the model chooses tools by name and description, and nothing in the client warned us that two servers were claiming the same names. A hostile server could do the same on purpose.

A practical MCP audit checklist

This is the checklist we are working through on every machine where an agent has MCP servers. Our own config failed items 2 and 3 when we first checked it.

Inventory everything. Global config, per-project config files, plugin-provided servers, and anything a teammate added in a pull request. Look at the full command and arguments for each one. If you cannot say why a server is there, remove it.
Pin every locally launched server. Use an exact version number for npm packages (such as 1.4.2, never @latest) and a commit hash for Git sources. Update on purpose, after reading the changelog.
Snapshot tool definitions. Save the output of tools/list and prompts/list for each server and compare it on a schedule, including after several calls in one session, since Deadbugz only changed after the third call. Any diff in a description should be read by a person.
Treat remote servers as third parties. Only connect ones run by a vendor you would give the same data to directly, and check what scopes the token they get actually has.
Keep secrets out of reach. Run agents as a user that cannot read ~/.ssh, cloud credential files or production kubeconfigs. If an agent needs a logged-in browser, give it a separate profile with only the accounts that task needs.
Scope servers to projects. A database server that one project needs should not be loaded in every session on the machine.
Restrict outbound traffic where you can. An allow-listed egress proxy turns quiet exfiltration into a blocked request you can see in a log.
Check the published indicators. Pillar lists the remote endpoint productivity-suite-mcp.onrender.com and a hidden local file at ~/.config/.cache/.sys/.deadbug-mcp.py. A quick grep over your MCP config files and a find for that path take seconds.

For the first pass, one line finds the unpinned launches in a Claude Code config: grep -nE '@latest|"npx"|git\+https' ~/.claude.json .mcp.json. It will flag some safe entries too. That is fine, the point is to look at each one.

The same rule applies to AI features you build

Deadbugz is a supply-chain story, but the lesson underneath is older: anything that lands in a model's context can act as an instruction, so the model should never hold more power than the person it is working for. That is why we design in-house assistants to read and cite rather than act, and why access checks belong in the retrieval layer instead of in the prompt. We covered the access side in scoping an internal AI assistant to the person asking, and the approval side in governing agentic AI in a helpdesk.

In FanMind, our document assistant, every outbound call goes through one allow-listed tunnel, web search is limited to domains an admin approves, and confidential documents are excluded from AI indexing by default. None of that makes prompt injection impossible. It limits what a successful injection can reach, and after an audit like ours that is the question that matters most.

If you only do one thing this week, open your MCP config and read every command in it. Ours turned up four problems we had not thought about.

Originally published on fanpino.com.

Exchange 2016/2019 Updates End in October 2026: A Self-Hosted Exit Plan

Hamze Zare — Mon, 28 Sep 2026 11:10:26 +0000

If you still run Exchange Server 2016 or 2019 on your own hardware, the last safety net goes away at the end of October 2026. Mainstream and extended support already ended on October 14, 2025. What kept some servers patched since then was a paid Extended Security Update (ESU) program, and Microsoft has said plainly that it will not be extended again.

This piece is for the admin who has to pick a route in the next few weeks. It covers what the dates actually mean, the three realistic options, and what a move to self-hosted open-source mail looks like in practice. For that last part we lean on a migration we ran ourselves. One honest caveat up front: our project moved a university off IceWarp, not Exchange. Most of what we learned carries over. Some of it doesn't, and we'll say where.

The dates, straight from Microsoft

October 14, 2025: end of support for Exchange 2016 and 2019. No more bug fixes, security fixes, time zone updates or technical support for anyone without an ESU contract (Microsoft Learn).
ESU Period 1: October 2025 through April 2026.
ESU Period 2: the start of May 2026 through the end of October 2026. It is a separate contract. If you bought Period 1, you have to buy Period 2 again. It is sold through your Microsoft account team, requires an Enterprise Agreement, and is not included in Volume Licensing or Software Assurance (Exchange Team blog).
After October 2026: "Once October 2026 ends, there will be no further updates for Exchange 2016/2019, even if you currently have a Period 2 ESU," and "there will be no further extension of Exchange 2016/2019 ESU program timeline" (Exchange Team reminder).

Two details in the Period 2 FAQ are easy to miss. ESU updates are shared privately with paying customers only, and Microsoft says it is not committing to release any security update during Period 2 at all. So even the paid bridge only gives you fixes if Microsoft decides a vulnerability is serious enough to ship one.

An Exchange server that faces the internet and gets no patches is a serious liability. Exchange has been one of the most heavily targeted server products of the last several years. Plan as if the server has to be off the internet, or off Exchange 2016/2019, by November.

The three routes that actually exist

1. Upgrade in place to Exchange Server Subscription Edition (SE)

Microsoft's own on-premises path. You stay on Exchange, your admins keep their skills, Outlook keeps every feature it has today. The trade is licensing: SE is a subscription product, so the "buy it once, run it for ten years" model is gone. If your problem is only the deadline, this is the lowest-risk move and nothing below should talk you out of it.

2. Move to Microsoft 365 / Exchange Online

What most of the search results for this topic recommend, and for many organizations the right answer. Someone else patches the servers. You pay per user, forever, and your mail lives in Microsoft's cloud.

3. Leave Microsoft's mail stack for self-hosted open source

This is the option most guides mention in one line and then drop. It makes sense for a specific profile:

You have enough mailboxes that per-user pricing turns into a large annual number.
You have a hard data-residency requirement, meaning mail has to stay in-country or on hardware you control.
For organizations in Iran and similar markets, a USD-billed subscription carries currency and sanctions risk on top of the price.

If none of those apply to you, route 1 or 2 is probably simpler. The rest of this article is about route 3, because that is where we have first-hand experience.

What we actually did: 10,000 mailboxes off a commercial suite

Lorestan University ran its email on IceWarp, a commercial on-premises mail and collaboration suite, and was paying roughly $70,000 a year in licensing for 10,000 mailboxes. We moved all of it to Stalwart Mail Server, which is open source. Infrastructure cost at the same scale is now under $4,000 a year. The full cost breakdown is in our cost case study, and the reasons organizations still choose self-hosting in 2026 are in this earlier piece. We won't repeat either here.

The deployment is deliberately boring. It runs on one server: Stalwart Community (pinned to the v0.16 line) with PostgreSQL as the datastore, MinIO for blob storage and NATS, all in Docker, with a daily backup that gets test-restored. Accounts come from the university's existing LDAP/Active Directory instead of being created one by one. That last point matters a lot if you are leaving Exchange, because your directory is already in AD.

What carries over from an Exchange world

Directory. Stalwart can read accounts from LDAP/AD. Your users, groups and passwords stay where they are.
Mail clients over standard protocols. IMAP, SMTP and POP3 work with Outlook, Thunderbird, Apple Mail and phone clients. Compatibility with Outlook, Thunderbird and mobile was one of our explicit acceptance criteria.
Calendars and contacts. CalDAV and CardDAV are supported, so calendars and address books have a standard home.
A fast webmail. Stalwart speaks JMAP, a modern protocol built for web clients. We built our webmail on it, and it is noticeably quicker than IMAP-based webmail. We wrote about why in this piece on JMAP.

What does not carry over (read this part twice)

Exchange ActiveSync. Stalwart Community does not implement EAS. In our own project plan, ActiveSync was listed as out of scope, with a note that if a pilot showed users depended on it, we would reconsider the mail engine. If your phones and your Outlook setup rely on ActiveSync today, test this first. Don't discover it on cutover day.
The Outlook features that only exist on MAPI/EWS. Outlook over IMAP is a different experience from Outlook against Exchange. Shared mailbox delegation, public folders, and the way free/busy and meeting rooms behave are the usual gaps. Collect the features your users actually rely on before you promise anyone a like-for-like move.
Some admin comforts. On the Community edition, some settings management and native metrics sit behind Stalwart's Enterprise license. We replaced the metrics side with Prometheus and Grafana. Budget time for that kind of glue work.
Somebody's pager. Self-hosting means you own deliverability, spam filtering, TLS certificates and backups. Nobody else patches it at 3 a.m.

The migration mechanics that saved us

Treat mail history as sacred. Our rule was that no mailbox is touched without a backup, and the source is never modified. History was copied with file-based export tools, followed by a weekly incremental sync that only appends and never deletes. That way a message deleted by mistake on the old system could not wipe the copy on the new one.

Verify by Message-ID, not by counting files. A folder that has the same number of files on both sides can still be missing messages and have duplicates of others. Unique Message-IDs are what tell you the history arrived intact.

Pilot one department first. This was the lesson we'd push hardest. Moving a small group first surfaced DNS and DKIM mistakes while only a handful of mailboxes were affected. None of the problems were exotic. They were the kind you only notice once real mail flows through new records, and it's far better to find them with twenty users than with ten thousand.

Get DNS and reputation right before the big cutover. SPF, DKIM, DMARC and a correct PTR record for the sending IP all have to be in place. If your new server sends from an IP address with no history, expect to warm it up gradually. A cold IP that suddenly sends a university's worth of mail gets throttled or junked by the big providers.

Think about login recovery. On your own mail server, "we'll email you a reset link" is circular. We added SMS one-time-password login with 2FA, and at this scale it mattered more for day-one adoption than any webmail feature. We also added an opt-in email-to-SMS alert for staff who don't sit in their inbox all day.

If you have five weeks, not five months

A cross-platform migration of a few thousand mailboxes is not something to rush before October 31. If you can't finish in time, the honest order of moves is:

Get the Exchange servers covered first. That means an in-place upgrade to Exchange SE, or Period 2 ESU if you already have it, or at minimum taking OWA and other Exchange endpoints off the public internet.
Take inventory: mailbox count, total size, who uses ActiveSync, which shared mailboxes and public folders matter, and which applications relay mail through Exchange.
Stand up the new platform alongside the old one and move one pilot department.
Fix what the pilot finds, then migrate department by department with incremental sync running the whole time.
Switch MX records only when the pilot has run cleanly for a while.

Whether you land on Exchange SE, Microsoft 365, or your own servers, the deadline itself is not negotiable. Leaving an unpatched Exchange server on the internet in November 2026 is the one option that is clearly wrong.

The Lorestan side of this, including what 10,000 real users noticed and what they didn't, is written up here.

Originally published on fanpino.com.

Mailcow vs Stalwart vs Mailu at 10,000 Mailboxes: What Holds Up in Production

Hamze Zare — Sat, 26 Sep 2026 20:47:49 +0000

Most "mailcow vs Stalwart vs Mailu" posts are written from a homelab: one domain, a handful of mailboxes, a feature checklist. That's a fine way to pick a server for your family. It tells you very little about what happens at 10,000 mailboxes, where the questions change from "does it have a spam filter" to "what happens when one node dies" and "how do I provision ten thousand accounts without clicking ten thousand times".

We run Stalwart for a university with 10,000 mailboxes (the cost side of that migration is in a separate write-up). So one of the three columns below comes from production. The other two don't. We have not run mailcow or Mailu at this scale, and where this article describes them, it's describing their official documentation, linked inline. Treat those parts as a reading of the docs, not as field notes.

The short version

	mailcow	Mailu	Stalwart
Shape	Docker Compose stack: Postfix, Dovecot, SOGo, Rspamd, ClamAV, MariaDB, Redis, Nginx	Set of containers (Postfix, Dovecot, Rspamd, admin, webmail)	One Rust binary doing SMTP, IMAP, POP3, JMAP, CalDAV, CardDAV, WebDAV
Minimum RAM (per docs)	6 GiB + 1 GiB swap	1 GB without antivirus, 3 GB with ClamAV (+1 GB swap)	Depends on the stores you attach
Webmail	SOGo (with calendar, contacts, ActiveSync)	Roundcube or SnappyMail	None bundled for users; JMAP for building one
Storage	Local disk + MariaDB	Local volumes; external DB possible	Pluggable: RocksDB, PostgreSQL, MySQL, SQLite, FoundationDB; blobs in S3/Azure
Multi-node	No documented active-active mode; cold-standby via rsync	Community Helm chart, front-end replicas, needs RWX storage	Clustering in Community; read replicas and sharded stores in Enterprise
License	GPL-3.0	MIT	AGPL-3.0, plus a commercial Enterprise license

mailcow: the most complete box, and it is one box

mailcow is what most people mean when they say "self-hosted email that just works". According to its system requirements page, the default install wants at least 6 GiB of RAM plus 1 GiB of swap. The same page is refreshingly honest about why: a single SOGo worker can use around 350 MiB, and a company with 15 phones on ActiveSync and about 50 concurrent IMAP connections should plan for 16 GiB. ClamAV and the full-text search engine are the other big consumers, and both can be switched off.

What you get for that memory is a lot. SOGo gives users webmail, calendars, contacts and Exchange ActiveSync out of the box, which matters if you're replacing Exchange for people who live in Outlook on their phone. Mailbox migration is built into the admin UI as sync jobs, which run imapsync under the hood. The community is large, and most problems you'll hit have a forum thread already.

The limit for a large deployment is the shape. mailcow is a Compose stack meant to live on one host. We couldn't find a supported active-active or multi-node mode in its documentation; the redundancy story it does document is a cold standby, a consistent rsync copy to a second machine that you switch to when the first one dies. That's a reasonable design for a few hundred users. At 10,000, "restore to the standby" means an outage you have to schedule people around. The docs also rule out LXC, OpenVZ and Virtuozzo, so check your hosting before you start.

Mailu: small, tidy, and friendly to orchestration

Mailu is the lightest of the three on paper. Its requirements page asks for 1 GB of RAM and 1 GB of swap without antivirus, and 3 GB with ClamAV. Webmail is either Roundcube or SnappyMail, there's a REST API for the admin side, and the code is MIT-licensed, the most permissive of the three.

It's also the one that takes Kubernetes seriously as a deployment target, through a Helm chart. Read that chart's README before you plan around it. The front end can run as several replicas or as a DaemonSet, but running across nodes with the default single volume requires a storage class with ReadWriteMany access. Mailu's own docs point to the chart and note that they're looking for maintainers for it. Neither is a reason to avoid it. Both mean HA with Mailu depends on your storage layer at least as much as on Mailu itself.

Stalwart: what we actually ran into

Stalwart is a single binary written in Rust that speaks every mail and groupware protocol itself, instead of wiring Postfix to Dovecot. It's dual-licensed: AGPL-3.0 for the Community edition, and a commercial license for Enterprise. The reasons we picked it for a 10,000-mailbox migration off IceWarp were practical: JMAP (so we could build a webmail that doesn't feel like 2005, more on that in our JMAP piece), external storage, and a clustering path that doesn't require an enterprise contract.

Here's what running it looked like, including the parts that cost us time.

Storage is split, and that's the point

We keep metadata and settings in PostgreSQL and message blobs in S3-compatible object storage (MinIO). That split is what makes Stalwart workable at this size: PostgreSQL gets backed up and replicated like any other database, the blob store scales on its own, and no mailbox lives on one machine's local disk. It also means a PostgreSQL backup is a backup of the server's whole configuration, because since v0.16 the settings live in the database too.

Old guides will mislead you

Version 0.16 changed the configuration model. There's no big config.toml anymore. A small JSON file on disk describes only how to reach the datastore, and everything else (listeners, domains, DKIM, spam settings) is stored as objects in the database and managed through the web admin or the management API. Most tutorials you'll find online predate this, and following them produces errors that don't obviously point back to the cause. The management API is OAuth-protected with no password grant, so the very first setup is interactive, through a browser or a device-code flow.

Some settings weren't reachable on Community

We wanted a server-side send rate limit per user, so a compromised account talking raw SMTP couldn't blast out mail. On the Community build we ran (v0.16.8), we couldn't find a working way to apply it: --config only accepted the datastore JSON, the settings endpoints returned 404 even with a valid token, and editing the binary config rows in PostgreSQL directly wasn't a risk worth taking on a live mail server. We tested all of this on an isolated instance, not production. Our reading is that this part of settings management sits in Enterprise on that version, though Stalwart's public edition comparison doesn't spell it out line by line. We ended up rate-limiting in the webmail layer and at the reverse proxy in front of JMAP instead. If per-user sending limits are a hard requirement for you, test that exact setting on the Community edition before committing.

For the record, the comparison page does list what is Enterprise-only: multi-tenancy with per-tenant quotas, the AI/LLM spam classifier, account archiving and undelete, SCIM provisioning, read replicas and sharded stores, and the live dashboards and metric alerts. Basic clustering is in Community. Published Enterprise pricing is per mailbox per year, starting at €2.00 and dropping to €0.89 at higher volumes, so at 10,000 mailboxes it's a real line item but not an IceWarp-sized one.

Small things that bite on a restricted network

On first start, the admin web UI is downloaded from GitHub. If your server can't reach GitHub, serve that file from somewhere it can reach.
The submission port, 587, wasn't open by default in our install. Mail clients configured for 587 will fail to send until you add that listener.
If you want a log file for your own tooling, configure a file tracer. In our Community build that worked through the JMAP management objects, unlike the rate-limit settings above.

Ten thousand accounts, and getting mail across

Creating accounts one by one isn't realistic at this size. The sane option is pointing Stalwart at the existing directory (LDAP/AD) so accounts come from the source of truth. For moving mail off IceWarp we used imapsync for messages and notes, and a small script of our own to copy calendars, contacts and files DAV-to-DAV, since both sides speak CalDAV/CardDAV/WebDAV. The awkward part wasn't Stalwart at all: IceWarp stores passwords hashed, so we couldn't just replay them. The workable approach was a temporary password per account for the duration of the copy, then restoring the original hash afterwards. Stalwart now also ships its own migration tool, Vandelay, which we didn't use, but it's worth a look before you script your own.

HA, as designed

Stalwart's Community clustering runs several independent nodes on the same shared stores, coordinated through a message bus. Our notes favored NATS over Redis pub/sub for coordination at this size, with HAProxy in front and the shared stores made redundant in their own right: PostgreSQL with a replica and failover, MinIO in distributed mode. That last part is where the actual work is. Stalwart clustering doesn't make a single PostgreSQL instance any less of a single point of failure.

So which one?

If you're under a few hundred users, want calendars and ActiveSync for phones without building anything, and a cold standby is an acceptable recovery plan, mailcow is the easy recommendation. It's mature and it's complete.

If you want the smallest footprint, an MIT license, or you already run Kubernetes and have ReadWriteMany storage you trust, Mailu fits well.

If you're at thousands of mailboxes, need the service to survive losing a node, want mail stored in systems you already know how to back up (PostgreSQL and S3), or want JMAP to build a proper web client on, Stalwart is the one we'd pick again. Go in knowing that v0.16's configuration model is new, older guides are wrong about it, and some admin features sit in the paid edition. Budget time to learn it on a test instance before your cutover weekend, not during it.

Whichever you pick, the server is usually not the hard part at this scale. Deliverability, DNS, directory sync and moving years of mail without losing flags are where the calendar goes. Why self-host at all in 2026 covers the other side of that decision.

Originally published on fanpino.com.

RPI in Practice: Research, Plan, Implement with Claude Code Subagents

Hamze Zare — Wed, 23 Sep 2026 21:50:41 +0000

Our first article on the RPI framework covered the why: coding agents do their best work when you split a task into Research, Plan and Implement, and keep each phase's context clean. This one is the how. It is the working setup we use day to day with Claude Code subagents, including the parts that went wrong.

Nothing here is a benchmark. It is one small team running a monorepo of about a dozen products (helpdesk, MES, email hosting, a university AI assistant) with an AI coding agent doing a large share of the work. The failure stories are the useful part.

The short version

Research runs in a subagent, never in the main conversation. It returns a summary or writes a file.
Plan happens in the main conversation, in writing, before any code changes.
Implement runs in one or more subagents, each with a narrow scope and clear ownership of files and tools.
Verify against something that can't lie to you: a file on disk, a fresh page load, a length check. Never against the agent's own echo of what it did.

Why research belongs in a subagent

Research is the phase that eats context. Reading logs, grepping a codebase, pulling a Search Console export, paging through a mailbox: most of that output is looked at once and never needed again. If it lands in the main conversation, it stays there for every later turn, pushing the model out of what Dex Horthy calls the "smart zone" and into the "dumb zone", where it starts forgetting instructions and repeating mistakes.

Anthropic's subagent documentation states the goal plainly: "Preserve context by keeping exploration and implementation out of your main conversation." Each subagent gets its own context window and hands back only a summary.

In practice, our research subagents do two things differently from a quick question:

They write their findings to a file. When we researched topics for this blog, the research agent wrote a structured topics file with sources and target queries, and returned a 200-word summary. The writers that came after read the file, not the summary.
They are told what not to do. "Research only, no publishing, no browser" is part of the directive. A research agent that starts fixing things mid-investigation is how you end up with changes nobody planned.

One example of why the research phase matters: Search Console flagged 79 pages on our site as "alternate page with proper canonical tag." The easy move was to start tweaking canonical tags. Drilling into the example URLs first showed that none of them were on our site at all. They belonged to a forgotten subdomain pointing at a server we don't run. The fix was deleting one DNS record, not touching a single template. Research first saved a pointless code change.

Planning: the one phase that stays in the main conversation

The plan is where your judgement goes in, so it stays where you can see it. Our plans are short and concrete: which files change, what "done" looks like, what is explicitly out of scope, and how the result will be checked.

The most useful habit has been writing each subagent's directive as if for a capable colleague who just walked in: what to do, what not to touch, what to report and in how many words. Directives that say "fix the bug based on your findings" push the thinking onto the subagent, and the results show it. Directives that name the file, the function and the check come back right far more often.

Implementation: parallel agents need ownership rules

Running several implementation agents at once is where the time savings are. It is also where most of our incidents happened. Three rules came out of them.

1. One agent owns each shared resource

We drive a real browser over a remote debugging port for tasks that have no API: Reddit, a Telegram chat, the Search Console UI. Two agents driving the same browser at once means tabs switching under each other and actions landing on the wrong page. So every directive now states it: "You are the only agent allowed to drive the browser," or "No browser, another agent owns it." The same applies to a mailbox, a database migration, or a deploy.

We also learned that "the browser" is not always one thing. At one point the browser tool the agent had been using was connected to a different, isolated browser instance than the one on the debugging port. It was getting blocked by a site while the real browser was not. Checking the target list directly (curl localhost:9222/json) and connecting to it explicitly fixed it.

2. Commit fast, and never trust an uncommitted worktree

When several sessions share a repository, uncommitted work is fragile. We lost edits once when another session ran git reset --hard in the same working tree. The rule since then: small commits, pushed immediately, and git add by explicit path so an agent never sweeps up someone else's half-finished changes.

The opposite problem showed up this month. Our production site had been built from a working tree that contained files nobody had ever committed. Tracked code imported them, so the site built fine on that one machine. When the site moved to a new server, a clean checkout of main failed to build. The fix was committing the missing files and proving it with a build from a fresh worktree. "It builds here" is not the same as "it builds."

3. Scope each agent to the smallest thing that can be checked

"Update the SEO titles" is too loose. "Change only meta_title and meta_description on these three documents, preserve every other field, then curl the live page and confirm the new title" is the kind of task an agent finishes correctly. When we once let an agent update whole documents, it dropped the Arabic translations on the way; the narrower instruction would have prevented it.

Verification: check the world, not the transcript

This is the lesson that cost the most. An agent reporting "done" means the agent believes it is done. Tool output shown back to the model is not always the ground truth either.

Tool output can be quietly wrong

In long sessions we saw displayed tool results with words silently dropped. The underlying data was fine; the text the model saw was not. Twice that nearly led to a wrong conclusion, including a false diagnosis that a website was corrupting form input. The fix is boring: write results to a file and read the file, or compare a length or checksum instead of eyeballing text. Before we publish a post through a web form now, the agent checks that the field's character count matches the source exactly.

"The button was clicked" is not "the message was sent"

Our worst incident was in a Telegram web chat with a real business contact. The agent inserted text into the message box with a DOM editing command. The text showed on screen, but the app's internal draft state never updated. Every send attempt read stale, partial state, and repeated retries sent seven garbled fragments to the other person before anyone noticed. We deleted them through the app's own API and sent one clean message.

Two changes came out of it. Input now goes through real input events that the app actually listens to. And after sending, the agent reads the message back from the app's own data store and compares it to the intended text before reporting success. A green checkmark in the UI is not evidence.

Verify from where the user stands

After a deploy, check the live URL from outside, not the container you just restarted. We deployed an SEO fix, confirmed it on our server, and only later noticed that the domain's DNS now pointed to a different host still serving the old build. Checking both, with curl --resolve for the origin and a plain request for what the public sees, would have caught it straight away.

A directive template that works for us

Task: one sentence.
Scope: the exact files, records or pages. What is out of scope.
Ownership: which shared resources you may use (browser, mailbox, deploy).
Rules: no bypassing bot checks, no paid actions, commit by path only.
Verify: the specific check that proves it worked.
Report: under N words, with commit hashes and anything that needs a human.

It looks like overhead. In practice it is the difference between one pass and three.

What we would tell a team starting with RPI and subagents

Put research in subagents first. It is the cheapest win and the lowest risk.
Keep the plan in the main conversation and keep it written down.
Add parallel implementation only when you have ownership rules for shared resources.
Make every agent prove its result against the real system. Trust the file, the fresh page load and the checksum, not the summary.

For the reasoning behind the three phases, see part one. For a broader view of the discipline, Sourcegraph's practical guide to context engineering is a good companion read.

AI helped draft this write-up; the workflow, the incidents and the fixes are our own.

Originally published on fanpino.com. AI helped draft this write-up; the workflow, the incidents and the fixes are our own.

Search Console flagged 79 pages that weren't on our site

Hamze Zare — Wed, 23 Sep 2026 06:46:44 +0000

We verify our main domain in Google Search Console as a domain property. Last week that choice showed its downside.

The bucket we'd stopped reading

"Alternate page with proper canonical tag" is usually boring for us: query-string versions of the pricing and contact pages, each canonicalizing back to the clean URL. Working as intended. So when it jumped to 79 URLs with a failed validation, the first instinct was to shrug.

Clicking into the example URLs changed that. Not one of them was on the main site. All 79 were on an old subdomain, each with a different cache-busting query parameter, so every crawl looked like a new page to Google.

What was actually there

The subdomain stopped being part of our stack a long time ago. Nobody removed the DNS record. It still pointed at a server we don't operate, and that server's TLS certificate had expired. Google kept resolving the name, failing the fetch, and filing the result under our property.

Why a domain property does this

A domain property covers every subdomain under the zone, automatically. There's no opt-in per hostname. That's handy when the whole zone is yours and maintained. It also means a forgotten record anywhere in DNS shows up as your site's problem.

The fix

One DNS record deleted. We didn't renew a cert on a box we don't manage, and we didn't set up redirects for a hostname nothing should link to. With the name gone there's nothing left to crawl, and the bucket should drain over the next few recrawls.

Two habits worth keeping

Open the example URLs before you decide a Search Console category is fine. The aggregate number looked exactly like the harmless version.
Read your DNS zone top to bottom now and then. For every record: is this still ours, and does it point at something we still look after?

AI helped draft this write-up; the investigation and the fix are our own.

A stored-XSS report we couldn't quite reproduce, and hardened anyway

Hamze Zare — Thu, 17 Sep 2026 00:45:34 +0000

A researcher named Dhruv emailed us to report a stored XSS path: someone attaches a PDF with embedded JavaScript to a support ticket in our helpdesk product, and when a support agent opens it, the script runs "in the context of the application."

What we found when we checked

The download route has forced Content-Disposition: attachment and application/octet-stream since December 2025 — confirmed with git blame. The frontend attachment viewer has zero iframe, embed, or object anywhere in the code; every non-image attachment goes through a Blob download, never an inline render. Uploads are already validated with python-magic, reading the real file bytes instead of trusting the browser's claimed content type, and image/svg+xml is explicitly excluded from the allow-list for exactly this reason — an SVG can carry a <script> tag.

So the exact path Dhruv described likely doesn't reproduce as written. That's still not a reason to leave the code as it was.

What we changed anyway

Upload-time validation only tells you what a file was when someone uploaded it. Nothing stops the stored bytes from being served differently later if some other code path changes. We closed that gap by re-detecting the real MIME type at download time too, from the same bytes on disk, not the type stored in the database:

detected_type = magic.from_file(str(file_path), mime=True)
is_image = FileStorageService.renders_inline_safely(detected_type)

return FileResponse(
    path=str(file_path),
    filename=file_path.name,
    media_type=detected_type if is_image else "application/octet-stream",
    content_disposition_type="inline" if is_image else "attachment",
    headers={
        "X-Content-Type-Options": "nosniff",
        "Content-Security-Policy": "default-src 'none'; sandbox",
    },
)

Only an allow-listed image type is ever served as inline. Every PDF, and everything we don't explicitly recognize, is forced to download — now with nosniff and a sandboxed CSP on the response, so a browser can't decide otherwise even if some future code path opens the file directly. Four tests were added around this logic and it shipped to production the same day.

The actual lesson

A bug report that doesn't reproduce exactly as written is still worth reading carefully. Dhruv's email pointed at a real gap — not an exploitable one today, but a gap between what we validate at upload and what we trust at download. Closing that gap cost an afternoon. Finding out we needed to cost someone else's careful attention, and earned a reply thanking them for it.

AI helped draft this write-up; the investigation, the code, and the fix are our own.