DEV Community: SeaOtter ( Kentaro Mori )

You Don't Need to Be a Security Specialist to Verify Your Defenses: Learning with Web Security Control Lab

SeaOtter ( Kentaro Mori ) — Tue, 27 Oct 2026 09:06:51 +0000

Introduction: security is no longer "the specialists' job"

If you build web applications, scenes like these are probably becoming familiar:

You wrote the login flow and the admin pages yourself. But security checks keep slipping, because "there's nobody here who really knows that stuff."
You let AI write more of your code now. Working software arrives faster, but you have no way to judge whether that code is safe.
You promote AI adoption or digitalization inside your company, and when someone asks "is the security OK?", you are not sure what "OK" would even be based on.

Few organizations have a dedicated security person. Yet when an incident happens, the people who respond are the ones who built the system or promoted it. For many engineers today, the first problem is not learning advanced attack techniques. It is having a way to verify that the defenses you put in actually work.

That verification step is where things get fuzzy. You can say "I added an authorization check" or "I set attributes on the cookie," but if the condition for "checked" is vague, you end up with "the scan passed, but I can't explain what it guaranteed." Done in name only — I meant to have it covered.

Web Security Control Lab (below, "the lab") is a small learning system for practicing that verification step. It is not a diagnostic tool for security professionals. It is teaching material for engineers who build web apps or drive technology adoption in their organizations, so they can try things safely on their own machine and learn the basic pattern of defense plus verification.

This article first explains what the lab is for and how I would like you to use it, and then follows the implementation on both the application side and the scanner side, in code. The technical content is not watered down. Terms are explained where they first appear, so if security is not your specialty, read it while mapping the ideas onto your own application.

Repository: https://github.com/moruku36/web-security-control-lab

Safety premise: This lab is for learning only. The app binds to 127.0.0.1 only; never expose it to the internet or a shared network. As described later, the scanner refuses non-local targets. Do not use real data or real passwords, and never scan systems that belong to someone else. SECURITY.md

This article covers commit 4dc4d9d (main as of October 22, 2026). I verified the runs myself on October 24, 2026, in an isolated Python 3.12.3 environment. Links are pinned to this commit unless noted otherwise.

What this lab is for

The lab's basic loop is observe → detect (scanner) → analyze the cause → fix → re-verify. Instead of memorizing terms in a classroom style, you touch an app that has weaknesses, find them with a check, think about why they happen, and compare against the fixed state. Experiencing that whole loop on a small subject is the point.

The setup comes in pairs:

A FastAPI app that deliberately leaves weaknesses in (VULNERABLE mode), and the same app with defenses applied (HARDENED mode)
A scanner (CLI) that applies the same six conditions to both modes and returns PASS / FAIL / ERROR

Why bundle "a weak app," "a fixed app," and "a checking tool"? Noticing weaknesses takes practice, and a fixed app alone does not show you what changed. With the checking tool in the middle, "which conditions must hold before we may call it fixed" is decided up front, and the learning does not end at "I sort of get it."

I imagine three ways of using it:

Usage	What you do	What you get
1. Compare first	Start both modes, run the same checks, compare the results	The overall picture of what changes with and without defenses
2. Go deep on one	Follow the LAB-01 worksheet: predict → observe → explain the cause → fix → re-verify	One theme — authorization — understood to the point where you can explain it
3. Apply it to your own app	Use the six check conditions as a checklist and review your own app's login, admin pages, and logs	The habit of thinking about defense and verification as a pair

Just as important is what the lab is not. It is not a tool for assessing production services, and its results are not proof that a production system is safe (see "Limitations"). It is not for probing other people's systems either: the scanner is designed to refuse non-local targets in the first place.

The whole picture: only three parts

Figure 1: Overall architecture. Colors: blue = learner, green = scanner, purple = target app, orange = launcher, yellow = data. Rendered from the repository's Mermaid source with Pele 1.8.0.

Element	What it is	Notes
Target app	FastAPI (`app/`), one process	Takes exactly one `LAB_MODE` per start
Data	SQLite (inside the app process)	In-memory by default; changeable via `LAB_DB_PATH`
Scanner	Host-side CLI `security-lab` (`scanner/`)	Checks over HTTP only; never opens SQLite directly

The key point: the scanner does not peek inside the app (its database). It accesses the app from outside, over HTTP — the same vantage point an attacker or a user has — which makes the perspective close to real-world practice: "how does this look from the outside?"

Start the app with uvicorn on the host, or with Docker Compose's single web service (VULNERABLE, 127.0.0.1:8000). There is no database container and no scanner container. The mode is read from the LAB_MODE environment variable at startup and never changes during the process's life. To compare both modes, start separate processes on separate ports.

Try it: the smallest run

So this does not stay read-only, here is the minimal procedure. You need Python 3.11+ and Git (commands below are for Bash; on Windows PowerShell, only the way you set environment variables differs — noted below).

git clone https://github.com/moruku36/web-security-control-lab.git
cd web-security-control-lab
git checkout 4dc4d9dc1adfdf9cfcf07abf3593f04a5722d886
python -m venv .venv
source .venv/bin/activate
pip install -e '.[dev]'

The commit is pinned so everyone works on the version this article covers. This state (detached HEAD) is fine for reading and running. If you want to practice fixing things yourself, first create a working branch, e.g. git switch -c my-lab-practice.

Use two terminals to start both modes:

# Terminal 1: the mode that leaves weaknesses in
LAB_MODE=VULNERABLE uvicorn app.main:app --host 127.0.0.1 --port 8000

# Terminal 2: the mode with defenses applied
LAB_MODE=HARDENED uvicorn app.main:app --host 127.0.0.1 --port 8001

On PowerShell, set the variable before starting uvicorn, like $env:LAB_MODE="HARDENED". The mode is read at startup, so changing it requires a server restart.

In a third terminal, confirm startup and mode, then scan:

curl -s http://127.0.0.1:8000/health
curl -s http://127.0.0.1:8001/health

security-lab scan http://127.0.0.1:8000 --format json --expected-mode VULNERABLE
security-lab scan http://127.0.0.1:8001 --format json --expected-mode HARDENED

Check that the mode in the /health JSON is what you expect before going on. --expected-mode tells the scanner to confirm the app is in the intended mode; if it disagrees, every check becomes ERROR. Pass the base URL — do not append a path like /admin.

Trying it in a browser is also worthwhile. Log in as the regular user alice / user and open the admin page /admin: in VULNERABLE it is displayed; in HARDENED it is rejected with 403. But when comparing both modes in the same browser, use separate profiles, or test one mode at a time and clear cookies. Cookies are not separated by port, so separate tabs alone can make sessions interfere (RFC 6265 §8.5).

How to read the results (just this, to start)

Results are three-valued:

PASS: the check's condition was met
FAIL: a weakness was observed
ERROR: a prerequisite could not be confirmed, so the check did not complete. This is not success

When the checks run properly, the expected results are: VULNERABLE → 6 FAIL (HIGH 1 / MEDIUM 4 / LOW 1), HARDENED → 6 PASS (findings 0). Watch the exit codes, too. 0 means "the report completed" — it is 0 even when the report contains FAILs. "Exit code 0 = no problems" is a misreading we will come back to later.

The six checks: all of them show up in "ordinary apps"

None of the six targets is exotic. Any web app with a login has all of these elements in daily use. First, where each one appears in your own app:

ID	Target	Everyday scene	If it is weak (summary)
LAB-01	Authorization on the admin page	Admin screens, approval features	Someone who is not an admin can perform admin operations
LAB-02	`HttpOnly` on the session cookie	Anything with a login	A login state becomes easier to abuse via scripts
LAB-03	`SameSite` on the session cookie	Logins, integrations with external sites	A login state becomes easier to ride on unintended requests
LAB-04	Security headers	Every screen and API	Browser-side defenses do not engage
LAB-05	Information exposure in error responses	API error handling	Clues for inferring the internal structure
LAB-06	Recording failed logins	Logins, operational monitoring	Malicious attempts happen and nobody notices

Two terms, briefly. Authentication establishes who you are (logging in). Authorization decides whether you may do this operation. Being logged in and being allowed to see the admin page are different things, and LAB-01 targets exactly that confusion. A cookie is a small piece of data stored by the browser and automatically attached to requests when its conditions match. Carrying the login-state identifier in a cookie is a widespread pattern, and its "read restriction (HttpOnly)" and "sending conditions (SameSite)" are what LAB-02 and LAB-03 examine.

Figure 2: Each control guards a different boundary and has different check points. Colors: blue = browser side, purple = server side, green = scanner, red = range not verified by these checks.

The app side: where do the defenses live?

How the mode branches

The mode is read from the environment in app/config.py, and each feature branches on that value. The weaknesses are not "misconfigurations": they are written into the code explicitly, with a marker comment INTENTIONAL_VULNERABILITY_FOR_LOCAL_LAB, so that reviewers can distinguish "intentional weakness" from "accident." Marking places that are deliberately so is an idea that transfers to production code in reviews, too.

The endpoints:

Path	Role	What changes by mode
`GET /health`	Returns status and mode	Nothing (used for the scanner's prerequisite check)
`GET/POST /login`, `GET /logout`	Authentication; issue and discard sessions	Whether failures are logged (LAB-06)
`GET /profile`, `GET /api/profile`	Login-required page / the same content as JSON	Nothing
`GET /admin`	Admin page	Whether the role is checked (LAB-01)
`GET /api/diagnostic`	Diagnostic error response	Whether internal information is exposed (LAB-05)
`GET /audit-logs`	Returns the audit log	Nothing (unauthenticated in both modes — a limitation, below)

The practice accounts are fixed plaintext fixtures (regular user alice / user, administrator admin / admin). They are teaching data, unrelated to how credentials should be handled in production.

LAB-01: check authorization at the endpoint itself

In the /admin implementation (app/main.py), an unauthenticated visitor is sent back to login with a 303 in both modes. The difference comes next: in HARDENED, anyone whose role is not admin gets 403; in VULNERABLE, that check simply does not exist.

if LAB_MODE == LabMode.HARDENED.value:
    if user.get("role") != "admin":
        raise HTTPException(
            status_code=status.HTTP_403_FORBIDDEN,
            detail="Access denied: Administrator role required.",
        )

A common real-world failure is "regular users never see a link to the admin screen, so it's fine." Even if the link is invisible, a direct request to the URL still arrives. What this implementation shows is the principle: the check belongs at the endpoint itself, and that is where you refuse. When you build admin features in your own app, check this first.

The expected observations:

Observation	VULNERABLE	HARDENED
`/admin` while logged out	303 back to login	303 back to login
`/admin` as a regular user	200, page displayed	403, refused
`/admin` as an administrator	200, displayed	200, displayed
`/admin` with the old token after logout	303 back to login	303 back to login

Note that not everything differs between the modes. The only difference is regular-user access; invalidating the old token after logout is correct behavior shared by both modes, placed there on purpose. It is practice in distinguishing "the state after a defense" from "the parts that were already right."

Sessions and cookies

On a successful login, a token is issued with secrets.token_hex(24), a record is created in the sessions table, and the token is placed in the lab_session cookie (app/auth.py). Logout deletes the record from the sessions table, so the old token becomes invalid in both modes.

The cookie attributes branch by mode: HARDENED sets httponly=True and samesite="lax"; VULNERABLE sets neither (LAB-02, LAB-03). Note that secure=False is a deliberate choice for HTTP on localhost, shared by both modes (see "Limitations"). Rather than leaving everything to framework defaults, check once which attributes your session cookie actually carries — in many apps, that alone surfaces review points.

Headers, errors, logs

LAB-04 (security headers): a mechanism for attaching instructions to responses — "browser, behave like this." Only in HARDENED, HTTP middleware adds Content-Security-Policy: default-src 'self', X-Content-Type-Options: nosniff, Referrer-Policy: strict-origin-when-cross-origin, and X-Frame-Options: DENY. Whether they are configured shows up directly in the check result.
LAB-05 (error responses): /api/diagnostic returns, in VULNERABLE, a 500 JSON containing a mock traceback and runtime-environment information; in HARDENED, it is narrowed down to a generic message and an error_code. Detailed errors are convenient during development, but returning them to users in production gives clues about the internal structure. Every exposed value is mock teaching data, not a real secret.
LAB-06 (audit log): successful authentication is always recorded as AUTH_SUCCESS, but AUTH_FAILURE on failure is recorded only in HARDENED. If you record only successful operations and not failures, attempts such as password spraying cannot be detected when they happen. Not recording failures is itself treated as the weakness here.

The scanner side: what does a PASS actually prove?

Now the checking side — especially relevant if you want to add automated checks to your own app, or if you have to read a vendor's assessment results.

The scanner's six checks (scanner/checks.py) judge by these conditions:

ID	Target	PASS condition	Example FAIL reason codes	Severity
LAB-01	Authorization on `/admin`	Anonymous → login; regular user → 403; admin → displayed; old token after logout → invalid	`USER_ADMIN_ACCESS`, etc.	HIGH
LAB-02	Session cookie	Has the `HttpOnly` attribute	`COOKIE_ATTRIBUTE_MISSING`	MEDIUM
LAB-03	Session cookie	Exactly one `SameSite` attribute, with value `Lax` or `Strict`	`COOKIE_ATTRIBUTE_MISSING` / `COOKIE_SAMESITE_VALUE_INVALID`	MEDIUM
LAB-04	Response headers of `/health`	CSP, X-Content-Type-Options, and Referrer-Policy exist and are non-empty	`HEADER_MISSING`	MEDIUM
LAB-05	`/api/diagnostic`	Returns a generic error without internal information	`DIAGNOSTIC_EXPOSURE`	LOW
LAB-06	Audit log	The `AUTH_FAILURE` count increases across a failed login	`AUTH_FAILURE_NOT_LOGGED`	MEDIUM

Never let ERROR become success

Each check's result is three-valued: PASS (condition met), FAIL (weakness observed), ERROR (a prerequisite could not be confirmed; incomplete). The important rule: when a prerequisite collapses, do not pronounce on the weakness — fall to ERROR. A common failure of automated checks is a check that "didn't really run" turning into "no problem found." This lab does the opposite, thoroughly.

For example, LAB-01 does not start by hitting /admin. It first logs in with the synthetic accounts and confirms via /api/profile that the identity and role are as expected (that confirmation stays in the evidence as verified), and only then makes its four observations: anonymous, regular user, administrator, and the old token after logout. If the login prerequisite cannot be established, the check is ERROR. When several weaknesses are observed, the primary reason code is chosen by priority: anonymous access > regular-user access > session retained.

LAB-03's condition is strict, too. It looks only at the Set-Cookie header for lab_session (other cookies' SameSite is ignored), requires exactly one attribute, and — ignoring letter case and surrounding whitespace — accepts only Lax / Strict. If attributes are duplicated, or there are multiple target headers so the effective value is indeterminate, the result is ERROR, not FAIL. Note that SameSite=None means "does not meet this lab's condition"; it is not a general judgment that it is dangerous. What browsers do when the attribute is unspecified is also outside this check's scope.

Conversely, some checks are deliberately narrow. LAB-04 looks only at the presence and non-emptiness of headers; it does not evaluate how strong the CSP's content is. LAB-06 looks only at whether the count increased; a strict correspondence between a specific attempt and a log entry, or alerting, is out of scope. A PASS means only "that check's condition was met" — nothing more is claimed. That is the design policy. If you read results with the same discipline, you will neither over-read nor under-read vendor reports and automated assessments.

Minimize the evidence

What remains in a report's evidence (schema 2.0) is restricted by an allowlist to actor (anonymous / user / admin / user_old_token), endpoint, status code, and permitted redirect targets — at most 16 observations — plus fixed reason codes. Messages are fixed templates per reason code; response bodies and exception strings never leak into a report. The validation in scanner/models.py enforces this. It is a design so that the checking tool itself does not take in the target's internal information as "evidence." When you share check results internally or paste them into tickets, whether this care exists changes how easy they are to handle.

The scanner's own safety design: if the target is wrong, it stops before sending

Figure 3: Verification flow. If the prerequisites cannot be established, it ends in ERROR and is never treated as success. Colors: green = scanner processing and the HARDENED expectation, orange = the VULNERABLE expectation, red = abnormal termination (refusal / ERROR).

"It's a checking tool, so it may check anything" is not true. Target validation is fail-closed — when in doubt, stop (scanner/target.py, scanner/cli.py).

Permitted hosts are only localhost, loopback IPs (judged with ipaddress), and the Compose service name. Anything else is refused before any request is sent
Target URLs are accepted only in base-URL form (scheme + host [+ port]); URLs with credentials, paths, queries, or fragments are refused, and a refused URL is not echoed into the error message
The HTTP client ignores proxy environment variables (trust_env=False), does not follow redirects, and uses a 5-second timeout
The only credentials used are the synthetic fixtures. At scan start, /health confirms the service name and mode; if that disagrees with --expected-mode, every check becomes ERROR (MODE_MISMATCH)

As more work is delegated to AI agents and automation tools, designs like "if the target is wrong, stop before sending" and "targets outside your authority are simply unreachable" apply to automation in general, not only to checking tools.

Exit codes — the key point when wiring this into automation (CI):

Exit code	Meaning
0	Report completed (0 even when it contains FAILs)
1	No subcommand
2	Target refused, or argument error
3	Inspection incomplete (contains ERROR), or runtime error

In other words, "exit code 0" does not mean "no weaknesses." Judge pass/fail from the report's status_summary and findings. If you build your own checks into CI, you need the same separation between "the tool ran successfully" and "the checks passed."

Verified results

Results I confirmed in my environment (Python 3.12.3, commit 4dc4d9d, October 24, 2026). I started both modes exactly as in this article's procedure, and the scan results matched the expectations.

Tests: all 126 passed (unit / integration / vulnerabilities)
VULNERABLE (port 8000): 6 FAIL, findings HIGH 1 / MEDIUM 4 / LOW 1, exit code 0
HARDENED (port 8001): 6 PASS, findings 0, exit code 0
I also confirmed that deliberately mismatching --expected-mode makes every check ERROR (MODE_MISMATCH) with exit code 3, and that targeting https://example.com is refused before any request, with exit code 2

The text-format report for VULNERABLE looks like this (excerpt):

Web Security Control Lab
Target: http://127.0.0.1:8000 (mode reported by target: VULNERABLE)
Scope: six local educational checks only; not a production or compliance assessment.
[HIGH] LAB-01 Broken Access Control
  /admin is accessible by a non-admin user
[MEDIUM] LAB-02 Session Cookie
  HttpOnly flag is missing
[MEDIUM] LAB-03 Session Cookie
  SameSite policy is missing
[MEDIUM] LAB-04 Security Header
  Content-Security-Policy is missing
[LOW] LAB-05 Information Disclosure
  Detailed error information is exposed
[MEDIUM] LAB-06 Security Logging
  Failed authentication attempt was not recorded

The repository's CI includes a job that runs live checks against both modes, and the run for the target commit succeeded (CI run, confirmed October 22, 2026).

Limitations: HARDENED does not mean "safe in production"

This is easy to misunderstand, so let me be explicit. HARDENED means only "the state with the six lab controls applied." As the README states, the following constraints remain in both modes:

Fixed plaintext test credentials, sessions with no expiry, logout via GET, and an unauthenticated /audit-logs
No Secure on cookies (premised on HTTP on localhost)
The checks are observations by an HTTP client, not verification of real browser behavior. Confirming cookie attributes does not prove safety against cross-site operations
CSRF defenses themselves (tokens, etc.) are not among the lab's controls, and the scanner does not drive a browser to attempt CSRF (MDN's CSRF explainer)
Passing the six checks proves neither production safety nor this material's educational effectiveness (educational effectiveness is unevaluated)

For a real service you would additionally need HTTPS enforcement, hashed password storage, session expiry and revocation design, CSRF tokens, access control and retention for logs, CSP content design, and more. The lab's scope ends at the entrance: putting "the basic six" into a checkable form.

How to use this in your own work

Finally, how to apply it, by reader:

If you develop web applications. When you add a feature, use the six conditions as review viewpoints as they are. Added an admin feature? Try the same URL as logged-out / regular user / administrator. Added a login? Look at the session cookie's attributes and whether failures are logged. When a review says "we've handled it," being able to ask "confirmed by which observation?" is this material's practical value.

If you have AI write code, or you promote AI adoption. For generated code, "it works" and "the defenses work" are separate claims. The lab's approach — define the defense and its check condition as a pair, and never pass a check whose prerequisites collapsed — is a yardstick for reviewing AI-generated authentication and authorization code. The automation side's fail-closed design (target allowlists, minimized evidence) is also a reference for deciding what scope to delegate to agents.

If you develop in a team. The six check conditions work as a shared language for security reviews. Being able to say "by LAB-01's condition, this screen is open to regular users" instead of "this somehow feels risky" moves the discussion forward even without a specialist in the room.

Summary

What I most want to convey with this lab is not the names of individual defenses, but the design of verification:

Define the defense and its check condition as a pair: decide what to fix, and by which observation you will call it fixed, at the same time
Never let ERROR become success: a check whose prerequisites are unconfirmed is "incomplete," not "passed" — and distinguish that in the exit code, too
Do not widen what PASS means: do not speak of things you did not check (CSP strength, real browser behavior, CSRF as a whole) as if they had passed
Make the checking tool fail-closed, too: target allowlists, minimized evidence, no exception strings recorded

Start by launching both modes and looking at how the results differ; then go deep on just one, with the LAB-01 worksheet (predict → observe → explain the cause → fix → re-verify). The repository is under the MIT license.

References

Repository: https://github.com/moruku36/web-security-control-lab
README — startup, the six checks, limitations
SECURITY.md — the local-only scope
docs/SECURITY_CONTROLS.md — each control's theory, detection logic, and OWASP mapping
MDN: CSRF
RFC 6265 §8.5 — cookies and port boundaries

What Is a Forward Deployed Engineer (FDE)? Origins, Business Models, and How AI Labs and Japanese IT Firms Are Using It

SeaOtter ( Kentaro Mori ) — Fri, 23 Oct 2026 12:13:01 +0000

Introduction

On October 22, 2026, OpenAI held a live event called "Working at the Frontier: Forward Deployed Engineering at OpenAI" and published a page describing the program.

https://openai.com/business/forward-deployed-engineering/

The term FDE (Forward Deployed Engineer) has become very common over the past year. At the same time, people often ask questions like these:

If the job is to go to a customer, listen to what they need, and build a system, how is that different from a solutions engineer or a contract developer?
The role apparently started at a company called Palantir. Why are AI companies adopting it one after another right now?
In the end, how does this model make money?

This article tries to answer those questions by organizing public information (company announcements, earnings materials, news reports, and job postings) in chronological order. Everything here reflects what was known as of October 23, 2026. FDE is a way of working that is still taking shape, so treat this as a first edition that will be updated as things change.

Key points

An FDE is not just "a developer stationed at a customer." Two things set it apart from traditional contract development: taking responsibility until the system is used in production and delivers results, and feeding what was learned in the field back into the company's own product or AI models.

It started at Palantir. Things built in the field were cleaned up by the product team, turned into product features, and monetized through software licenses.

In 2025, a widely discussed study reported that most enterprise generative AI pilots were not producing results. In 2026, OpenAI, Anthropic, Microsoft, AWS, and others committed close to $10 billion in total to FDE organizations.

There are roughly three ways to make money with FDEs: recover the cost through product licenses, recover it through deployment fees plus AI usage fees, and charge for outcomes or a fixed price instead of hours worked. Large Japanese IT companies are adopting the third model.

Terms used in this article

Term	Meaning
Production	The state where a system is used in everyday work, not just tried out
PoC / pilot	A small trial to check whether something works before full adoption
Person-month billing (ningetsu)	Pricing based on "one person working for one month." It has long been the standard in Japanese system integration
Usage-based pricing	Fees that depend on how much you use. For AI, this is usually the amount of text processed (tokens)
Gross margin	Revenue minus the direct cost of delivering the product or service. Software companies tend to be high; services companies that rely on people tend to be low
Private equity (PE)	Investment firms that buy stakes in companies, improve their operations, and profit from the increase in value

1. What did OpenAI announce?

First, here is what the OpenAI page says.

Item	Details
Role	FDEs join the customer's team, observe how work gets done, and bring AI into production
Process	Define the outcome → Discover → Prototype → Validate → Go to production → Scale
Tools	OpenAI Presence (an AI agent platform), Context Layer (a shared base for company knowledge), process mapping, and code modernization with Codex
Examples	T-Mobile (55% of customer chats handled without handoff to a person), BBVA (80% accuracy in credit risk assessment), modernization of 15,000 systems
Organization	OpenAI's own FDEs work together with its subsidiary, DeployCo (the OpenAI Deployment Company)

Figure 1. The flow of an OpenAI FDE project (based on OpenAI's page)

The FAQ on that page contains two answers that show what kind of role this is.

To the question "Do FDEs provide professional services?", the answer is "No." FDEs are described as engineers who work with professional services partners to solve problems at the frontier of AI.
It also says: "If something can be achieved with existing AI capabilities, an FDE is usually not needed."

In other words, OpenAI's FDEs do not take on just any request. They go into cases where current AI is not yet good enough to run in production, and they report the model weaknesses they find to the product and research teams. Broad rollout to many companies is handled by DeployCo and partners.

2. History: from Palantir to the AI labs

Figure 2. A short history of FDE (compiled from various sources)

2-1. Origins: Palantir (late 2000s to 2010s)

The FDE role was created by Palantir, the US data analytics software company. It is credited to Shyam Sankar, who joined in 2006 as the company's 13th employee and is now its CTO (profile). Palantir originally built software for government and intelligence agencies. Those customers handle a lot of classified information, and their work is highly specialized. The usual approach of "listen to the requirements, take them back to the office, and build" did not solve their problems well.

Palantir split the people who went out to customers into two roles (Palantir blog):

Delta: Forward Deployed Software Engineer. The person who writes code and builds things on site
Echo: Deployment Strategist. The person who owns the problem and the outcome, and moves the customer organization

The official blog explains that "in theory, Echos are more product-manager-like and Deltas are more technical, but in reality both roles are a mix of product management, engineering, and strategy."

Nabeel Qureshi, who worked at Palantir for about eight years, wrote the essay "Reflections on Palantir" (2024), a valuable inside account of how this worked. The main points:

FDEs spent three to four days a week at the customer's office, learning the work of demanding industries such as manufacturing, healthcare, intelligence, and aerospace.
The first major commercial project was with Airbus in Toulouse, France. The team built software that removed bottlenecks in A350 production and significantly increased the production rate.
FDEs prioritized speed and accepted rough, hacky solutions to get something working. The product development team then cleaned these up into stable software that many customers could use.
Many features of Palantir's main product, Foundry, came from the product team automating what FDEs had been doing by hand in the field.

Figure 3. Palantir's "field → product → next field" loop (based on Qureshi's essay)

2-2. "Isn't Palantir just a services company?"

The model of employing many FDEs was criticized for a long time: "That's not a software company, it's a consulting firm." This criticism was loud around the company's 2020 listing as well. Qureshi himself writes that around 2016 it was "partly fair."

Since then, however, the numbers have looked like those of a software company.

Gross margin in 2023 was about 80% (from Qureshi's essay, which compares it with about 32% at Accenture).
In Q2 2026, revenue grew 93% year over year to $1.935 billion, and US commercial revenue grew 149% to $764 million (Palantir earnings release). Gross margin is reported at about 85% (secondary source).

Because improvements made in the field keep getting absorbed into the product, the share of manual services work does not grow much as the customer base grows. That is Palantir's explanation of why it did not become a services company.

2-3. AIP Bootcamp (2023 onward)

In 2023, as generative AI spread, Palantir launched the AIP Bootcamp. Its FY2024 annual report (10-K) describes it as a way to "deliver real workflows on actual customer data in days." It is usually described as an intensive workshop of about five days, in which FDEs and the customer build AI use cases on the customer's own data. Since this leads into full contracts, FDEs also act as the front door for sales.

2-4. Renewed attention in the generative AI era (2025)

There are two main reasons FDEs suddenly drew attention in 2025.

The problem of pilots not turning into results became clear. In its report "The GenAI Divide: State of AI in Business 2025," MIT's NANDA initiative found that about 95% of enterprise generative AI pilots had no measurable impact on profit and loss (Fortune). The cause was attributed not to model quality but to how organizations adopt the technology. Note that this figure comes from early-stage research, and the reported sample sizes differ between news outlets.
Job postings grew sharply. There are reports that FDE postings grew more than 800% from January to September 2025 (Financial Times, via Fast Company) and an analysis showing 1,165% year-over-year growth (Bloomberry). The methods differ, so the numbers are not directly comparable, but they agree on the scale: roughly 8 to 12 times in one year.

OpenAI's FDE team also started with two people around this time and had grown to about 140 by August 2026, according to Colin Jarvis, OpenAI's head of FDE (Engineering Leadership).

2-5. The year of big commitments (2026)

In 2026, companies began creating dedicated organizations and companies and committing large amounts of money.

Date	Company	What happened	Scale
May 4, 2026	Anthropic	Announced an enterprise AI services joint venture with Blackstone, Hellman & Friedman, and Goldman Sachs. Launched in July as Ode with Anthropic	$1.5B ($300M each from Anthropic, Blackstone, and H&F), about 100 engineers
May 11, 2026	OpenAI	Launched the subsidiary OpenAI Deployment Company (DeployCo) and agreed to acquire Tomoro in the UK	Over $4B initial investment, about 150 FDEs and specialists from Tomoro
June 30, 2026	AWS	Created a new FDE organization that sends teams to customers for 45-day engagements	$1B
July 2026	Microsoft	Created the Frontier Company (an internal unit; some see it as a reorganization of its consulting services)	$2.5B, about 6,000 people
Sept 8, 2026	Accenture and Google Cloud	Set up a dedicated Gemini Enterprise unit	Target of 1,000 FDEs
Oct 2, 2026	Anthropic	Launched the Claude Frontier Academy to train partner engineers in a medical-residency style program	$100M, 10,000 engineers by the end of 2027

Sources: OpenAI, CNBC, TechCrunch, The Next Web (AWS), GeekWire (Microsoft), Directions on Microsoft, Anthropic

Investor Tomasz Tunguz estimates the money committed to FDE efforts over the past year at about $9.75 billion across five companies (The FDE Arms Race). OpenAI's October 22 event is part of this trend.

3. What kind of work is it?

3-1. What gap does an FDE fill?

Figure 4. The deployment gap and the role of FDEs (based on OpenAI's explanation)

OpenAI calls this the "deployment gap." No matter how capable AI becomes, it does not produce results unless it is connected to data scattered across the company, unwritten business rules, access control, and the habits of people on the ground. FDEs go inside the customer to close that gap.

3-2. How is it different from existing roles?

To answer "How is this different from a solutions engineer or contract development?", here is a comparison with similar roles.

	Pre-sales solutions engineer	Management consultant	Contract development (person-month)	FDE
Main output	Demos and proposals	A recommendations deck	A system built to spec	A system used in production that produced results
Who defines the problem	The customer	Worked out together	The customer (requirements)	Found together, including whether it is worth solving
Writes code?	A little	No	Yes (with a large team)	Yes (with a small team)
Where responsibility ends	The sale	The report	Delivery and acceptance	Production use and adoption
Where learnings go	Sales materials	Firm know-how	Often stay inside the project	Into the company's product or AI
Basis for fees	Product sales	Time	Headcount × duration	Product or AI usage, fixed price, outcomes

Figure 5. Where FDE sits relative to similar roles (positions are approximate)

The differences come down to two points.

Responsibility until results show up in production Traditional contract development usually ends at "build what was asked for and deliver it," and consulting ends at "make a recommendation." FDEs own the work until the business numbers actually improve.
A path back into the product This is the biggest difference. In contract development, what is learned on a project often stays within that project. FDEs pass what they learn back to the product team at Palantir, or to the research team at OpenAI. For example, at OpenAI a field report that "the model keeps making simple mistakes when converting COBOL to Java" was passed to the research team as a request for more training data in that area (Engineering Leadership).

Put the other way around, an FDE with no path back into the product is just on-site contract development with a new name. This is a useful lens when looking at the Japanese market later.

3-3. What the work looks like day to day

Job postings and interviews paint the following picture.

A lot of time on site: At OpenAI, about half of working time is spent with customers. It varies by region: about 80% in the UAE and Japan, and two to three weeks a month in Europe and the US (Engineering Leadership). Anthropic's postings list 25–50% travel.
Example work: One OpenAI example from Japan is modernizing the code of a customer with many legacy systems. Days alternate between meetings with teams such as payments, writing tests, and building migration code and progress dashboards.
Skills in demand: According to an industry site that aggregates postings, the most requested skills are customer communication, travel and on-site work, and Python. Only 32% of postings ask for LLM expertise (Synon FDE daily, Japanese). What matters more than AI knowledge is the ability to build something that actually works in the customer's environment.
Pay: Anthropic's US postings list $280K–$320K, and one analysis puts senior FDEs at AI labs at $350K–$550K (Tunguz).

3-4. Does this mean "AI writes the code and engineers just talk to customers"?

Some people read FDE that way. It is half right and half not quite.

What's right: Thanks to generative AI and coding agents, the effort needed to build has dropped sharply. As a result, a small team of three to five people can take a single use case all the way to production (see Fujitsu below). With less need for large teams, deciding what to build matters more.
What's not quite right: FDEs are not people who only talk. Palantir's Deltas and OpenAI's FDEs are engineers who write code and ship it to production. The planning and coordination role is often split into a separate role, such as Palantir's Echo or Salesforce's Deployment Strategist.

In short, the job is not "focus on talking to customers." It is to go into the customer's workplace and, together with them, use AI as a tool to build something all the way through.

4. Business models: how does it make money?

Looking at the companies adopting FDEs, there are roughly three ways they make money (this grouping follows an overview by a Japanese FDE consultancy).

Figure 6. Three FDE business models

Model A: Product adoption (Palantir, Salesforce)

FDE work is treated as an upfront investment. Bootcamps and pilots start small and short, and the company does not try to recover FDE costs at the start.
The cost is recovered through software licenses once the product is embedded in the business. Reports describe services as "a means to drive product adoption, not the primary revenue stream" (Yahoo Finance).
For this model to work, field improvements must keep flowing into the product. Otherwise it turns into an unprofitable on-site service.

Salesforce follows the same model, using FDEs to drive adoption of its AI agent product Agentforce. Its postings pair FDEs with Deployment Strategists, closely mirroring Palantir's Delta and Echo split.

Model B: AI lab and cloud deployment companies (OpenAI, Anthropic, Microsoft, AWS, Google)

Figure 7. Money flow in the deployment company model (based on company announcements and reports)

In this model, FDE work generates two kinds of revenue.

Deployment fees: DeployCo and Ode charge customers. Pricing has not been disclosed.
AI and cloud usage fees: Once AI agents run in production, they are used every day, and the usage fees flow continuously to the AI lab or cloud provider. This is most likely the real goal. A pilot consumes very little, but once AI is used across a whole company, usage grows by orders of magnitude.

Another notable feature is the partnership with private equity firms.

DeployCo has 19 investors led by TPG, including consultancies and integrators such as Bain & Company, McKinsey, and Capgemini. According to OpenAI, its partners sponsor more than 2,000 businesses worldwide (OpenAI).
At Ode, Blackstone and H&F refer their portfolio companies as potential customers, but Ode also sells to other companies (CNBC).

For PE firms, this is a way to improve operations at portfolio companies with AI and increase their value. For AI labs, it means access to more than 2,000 potential customers from day one. There are also reports that DeployCo's investors have a guaranteed minimum return of 17.5% a year (Tunguz). This could not be confirmed in official announcements, but if true, it suggests the business is expected to have high margins.

Microsoft, AWS, and Google have their own clouds, so their model is simpler. Workloads that FDEs bring into production translate directly into Azure, AWS, and Google Cloud consumption.

Model C: IT firms and consultancies selling outcomes (Hitachi, Fujitsu, NTT DATA, Accenture)

For IT companies and consultancies that do not have their own AI models, FDE is a way to move away from person-month billing.

Small teams, short cycles: At Fujitsu, three to five FDEs work on a single use case (Weekly BCN, Japanese).
Fixed or outcome-based pricing: Fujitsu says "the price is fixed and we don't show a breakdown. The deal happens if the customer judges that it's worth that amount." NTT DATA describes "a service package priced based on the customer's problem and expected results, not on person-month rates" (same article).
Reselling partner products: Fujitsu combines its services with Palantir's products, and Accenture with products from Anthropic, OpenAI, Google, Workday, and others.

Figure 8. Person-month contracting vs. the FDE model (based on Weekly BCN and other sources)

When generative AI reduces the effort needed to write software, pricing by "hours × rate" runs into a contradiction: the more you use AI, the less you earn. This is likely the main reason large Japanese IT companies are investing in FDE.

Comparing the three models

	A. Product adoption	B. Deployment company	C. Outcome-based
Examples	Palantir, Salesforce	OpenAI DeployCo, Ode, Microsoft, AWS, Google	Hitachi, Fujitsu, NTT DATA, Accenture
FDE cost	Upfront investment (mostly unrecovered)	Partly recovered through deployment fees	Recovered through fixed or outcome-based fees
Real revenue source	Software licenses	AI and cloud usage	Service fees and resale margin
Where learnings go	The company's product	The company's models and product	The customer and partner products
Main risk	Productization falls behind and it becomes a services business	Customer lock-in concerns, PE return demands	It becomes person-month billing with a new name

5. What each company is doing

5-1. Outside Japan

Company	Initiative	Notes
Palantir	FDEs (Delta) and Deployment Strategists (Echo), AIP Bootcamp	The original. Estimated at about 400–500 FDEs, about 12% of all employees (Tunguz)
OpenAI	Its own FDEs (about 140) focus on feeding insights into product and research, while DeployCo (about 200) scales delivery	Headcounts as of August 2026 (Engineering Leadership). Tomoro's customers include Tesco, Virgin Atlantic, and Supercell
Anthropic	FDEs in the Applied AI team, the Ode joint venture (CEO Chris Taylor, co-founder of Fractional AI), and the Claude Frontier Academy	Ode is "Claude-first" but uses other models when needed. The Academy trains engineers at partners such as Accenture, Deloitte, and McKinsey rather than hiring them into Anthropic
Microsoft	Frontier Company ($2.5B, about 6,000 people, led by Rodrigo Kede Lima, former president of Microsoft Asia)	Most staff come from existing teams. Some see it as a reorganization of its consulting services
AWS	FDE organization ($1B) sending teams for 45-day engagements	Customers include the Allen Institute, Cox Automotive, the NBA, the NFL, Ricoh, and Southwest Airlines
Google Cloud	Hiring its own FDEs, plus a 1,000-FDE Gemini Enterprise unit with Accenture	Also partnered with Hitachi on FDE-driven "physical AI" (June 2026)
Salesforce	Pairs Agentforce FDEs with Deployment Strategists	Plans to roughly double its UK FDE headcount to about 95 (ITPro)
Accenture	Anthropic Business Group (about 30,000 people trained on Claude, Dec 2025), a Gemini Enterprise unit with Google, and an FDE program with Workday (Oct 2026)	Acts less as a company promoting "FDE" as its own job title and more as the delivery partner for each AI vendor's FDE motion

Because Accenture does not have its own AI models or a flagship software product, it fits Model C rather than A or B: it builds a dedicated unit for each AI vendor and takes on deployment work. In 2026 it has also started using the term "FDE" in its work with Google and Workday.

5-2. Japan

Company	Initiative	Notes
Hitachi	Plans to build an FDE organization of about 1,000 people in Japan during fiscal 2026. Launched the "Physical AI FDE Team" service on Sept 3, 2026	Uses AI-driven interviews to draw out the know-how of skilled workers and structure it so AI can use it. Partnered with Google Cloud (Nikkei, Japanese, MONOist, Japanese, Google Cloud blog, Japanese)
Fujitsu	Set up an FDE division in April 2026. Became a Palantir Global FDE Partner on Sept 10	Has worked with Palantir since 2020. Senior Palantir FDEs join Fujitsu's teams for set periods. Examples include about ¥500 million a year in savings in maintenance parts management, and about 98% less effort to draft personnel transfer plans at Trusco Nakayama (Fujitsu, Japanese, Weekly BCN, Japanese)
NTT DATA	"DataEnablement Suite," covering everything from finding the problem to implementation and adoption	Sold as an outcome-based service package rather than by person-month. In one case it identified about 130 issues at a regional power company
NEC and others	Among the "four major IT companies" Nikkei reported as expanding FDE	At their July–August 2026 earnings briefings, the companies all spoke of moving away from person-month billing (Nikkei xTECH, Japanese). Details are behind a paywall and could not be confirmed
Dentsu Group	Began developing about 3,000 FDEs and FDMs (Forward Deployed Managers) across four Japanese group companies (Oct 14, 2026)	Grows FDE capacity by training existing staff rather than hiring
AI and SaaS companies	LayerX, Loglass, AI Shift, JAPAN AI, SB OAI Japan, and others have created or are hiring for FDE roles	76 FDE postings at 47 companies in Japan as of Oct 10, 2026 (unofficial count from an industry site)

What stands out in Japan is that large IT companies are adopting FDE as a way to move away from person-month billing (Model C). Outside Japan, AI labs and software companies lead; in Japan, large IT companies are becoming the main players. As with Fujitsu partnering with Palantir and Hitachi with Google Cloud, they also serve as local delivery partners for overseas vendors' FDE motions.

6. What comes next?

6-1. Why this is likely to become a lasting trend

The bottleneck has moved from "model capability" to "how AI is adopted." As the gap between models narrows, what differentiates is how deeply AI is embedded in the business. Ode's chief technologist put it this way: "model selection matters, but it's not where the majority of calories are spent" (TechCrunch).
Building got cheaper, so small teams can reach production. Integration work that used to take a large team months can now be done by a small team that knows how to use AI, in a much shorter time. That makes Palantir's approach reproducible at other companies.
It matches how money flows. AI labs earn from usage, cloud providers from consumption, and IT firms from moving beyond person-month billing. Whatever their position, "being used in production, continuously" is what drives everyone's revenue.

6-2. Things to watch out for

Drifting into a services business Software companies have gross margins in the 70–80% range, while people-based services companies are said to be around 30–40%. If FDE learnings are not fed back into the product, the company becomes a low-margin consultancy. There is persistent criticism that "FDE is just a new name for implementation consultants" (n47).
Lock-in The more FDEs build business processes on a specific vendor's AI, the harder it becomes to switch. For the vendor that is a moat; for the customer it can feel like a toll booth (Tunguz). Customers should check whether what is built and learned stays with them, or only improves the vendor's product.
Competition for talent People who combine strong engineering, business understanding, and interpersonal skills are rare. Anthropic choosing to train 10,000 partner engineers instead of hiring them reflects this shortage.
A new name on old work As discussed in section 3, what matters about FDE is "owning production results" and "feeding learnings back into the product." Simply renaming on-site contract development as FDE meets neither. As the Fujitsu example of "fixed price, no breakdown" shows, whether the contract itself has changed is a good way to tell.

6-3. What this means for engineers

Three things stand out as expectations for engineers.

Being evaluated on outcomes, not output As AI takes over more of the code writing, the value of deciding "what to build and why" and of "keeping it running in production" goes up.
Domain knowledge OpenAI says it deliberately builds FDE teams that mix industry experts with people from other industries.
Turning field lessons into something reusable Whether you can generalize a one-off solution so the next customer can use it is what separates FDE from traditional contract work.

What FDE is ultimately aiming for is this: go into the customer's workplace, use AI to build until results show up, and leave that experience in a form that can be reused next time.

Summary

FDE is a role Palantir created in the late 2000s. It worked by absorbing field solutions into the product and earning through software licenses.
In 2025 it became clear that many generative AI pilots were not producing results, and in 2026 OpenAI, Anthropic, Microsoft, AWS, Google, and others set up dedicated organizations and companies and made major investments. OpenAI's October 22 event is part of that trend.
What sets it apart from traditional contract development is owning production results and having a path to feed learnings back into the product or AI.
The money comes from product licenses (Model A), deployment fees plus AI and cloud usage (Model B), or fixed and outcome-based pricing (Model C). Large Japanese IT companies are using Model C to move away from person-month billing.
Whether something labeled FDE really is FDE becomes easier to judge by looking at the shape of the contract and where the learnings go.

FDE is still taking shape. This article is a first edition and will be updated as the companies involved make their next moves.

References

Company announcements and official materials

OpenAI, "Forward deployed engineering for enterprise AI" https://openai.com/business/forward-deployed-engineering/
OpenAI, "OpenAI launches DeployCo to help businesses build around intelligence" (May 11, 2026) https://openai.com/index/openai-launches-the-deployment-company/
Anthropic, "Claude Frontier Academy" https://anthropic.com/news/claude-frontier-academy
Anthropic, "Anthropic and Accenture partnership" https://anthropic.com/news/anthropic-accenture-partnership
Blackstone press release https://www.blackstone.com/news/press/anthropic-partners-with-blackstone-hellman-friedman-and-goldman-sachs-to-launch-enterprise-ai-services-firm/
Palantir Q2 2026 earnings release https://www.sec.gov/Archives/edgar/data/0001321655/000132165526000039/a2026q2ex991pressrelease.htm
Palantir FY2024 10-K https://www.sec.gov/Archives/edgar/data/1321655/000132165525000022/pltr-20241231.htm
Palantir Blog, "A Day in the Life of a Palantir Deployment Strategist" https://blog.palantir.com/a-day-in-the-life-of-a-palantir-deployment-strategist-951cb59a5a96
Fujitsu press release (Sept 10, 2026, Japanese) https://global.fujitsu/ja-JP/pr/news/2026/09/10-01
Google Cloud blog, Hitachi and Google Cloud on physical AI with FDE (Japanese) https://cloud.google.com/blog/ja/products/gcp/hitachi-and-google-cloud-collaborate-on-the-social-implementation-of-physical-ai-using-fde?hl=ja

News and analysis

Nabeel S. Qureshi, "Reflections on Palantir" https://nabeelqu.substack.com/p/reflections-on-palantir
Engineering Leadership, "Inside OpenAI's Forward Deployed Engineer" https://newsletter.eng-leadership.com/p/inside-openais-forward-deployed-engineer
Tomasz Tunguz, "The FDE Arms Race" https://tomtunguz.com/fde-arms-race/
TechCrunch (Ode with Anthropic) https://techcrunch.com/2026/07/15/anthropic-blackstone-bet-the-next-trillion-dollar-ai-business-is-implementation-not-models/
CNBC (Anthropic joint venture) https://www.cnbc.com/2026/05/04/anthropic-goldman-blackstone-ai-venture.html
GeekWire (Microsoft Frontier Company) https://geekwire.com/2026/microsoft-announces-2-5b-frontier-company-to-embed-ai-engineers-inside-customers
Directions on Microsoft https://www.directionsonmicrosoft.com/microsoft-launches-its-own-forward-deployed-engineering-unit-the-frontier-company/
The Next Web (AWS) https://thenextweb.com/news/aws-1bn-forward-deployed-engineers-enterprise-ai/
ITPro (Salesforce) https://itpro.com/security/salesforce-doubles-down-on-forward-deployed-engineers-in-uk-agentforce-push
Fortune (MIT NANDA, "The GenAI Divide") https://fortune.com/2025/08/18/mit-report-95-percent-generative-ai-pilots-at-companies-failing-cfo
Bloomberry, "I analyzed 1,000 forward deployed engineer jobs" https://bloomberry.com/blog/i-analyzed-1000-forward-deployed-engineer-jobs/
n47, "The Forward Deployed Engineer dilemma" https://www.n47.com/insights/the-forward-deployed-engineer-dilemma
Nikkei, "IT大手4社、『FDE』拡充急ぐ" (Japanese) https://www.nikkei.com/article/DGXZQOUC183EE0Y6A810C2000000/
Nikkei xTECH, "IT大手4社、脱・人月へFDE拡充に意欲" (Japanese) https://xtech.nikkei.com/atcl/nxt/column/18/00001/11956/
Weekly BCN, on FDE at Fujitsu and NTT DATA (Japanese) https://www.weeklybcn.com/journal/feature/detail/20260907_217086.html
MONOist, Hitachi HMAX solutions (Japanese) https://monoist.itmedia.co.jp/mn/articles/2609/07/news035_3.html
Synon, "FDE industry daily" (Oct 15, 2026, Japanese) https://synon.co.jp/article/news/2026-10-15-fde-daily
FDE Consulting, "Companies adopting FDE in 2026" (Japanese) https://fde-consulting.jp/blog/fde-companies-japan

Note: Some figures in this article, namely DeployCo's valuation and investors' minimum return, Palantir's FDE headcount, and FDE salaries, come from investor blogs and job posting aggregates and could not be confirmed in official announcements. FDE job growth figures also differ in period and method depending on who compiled them.

Never Hand the Agent the Keys: Reading Meta Muse's Tech Stack from Public Sources

SeaOtter ( Kentaro Mori ) — Fri, 23 Oct 2026 07:22:34 +0000

This is an English version of an article originally written in Japanese for Zenn.

Give an AI agent access to your email, calendar, and shopping accounts, and you immediately run into a problem: the agent needs credentials to do its work — but if a credentialed agent gets tricked, the credentials go with it.

Security researcher Simon Willison calls the combination that makes this dangerous the "lethal trifecta":

Access to private data
Exposure to untrusted content (web pages, email, files)
A channel to communicate externally

Muse, the personal AI agent Meta launched in the US on September 8, 2026, satisfies all three by design. It operates a browser on your behalf, reads and writes email, and even shops for you. Fittingly, the safety engineering blog post Meta published the same day is devoted almost entirely not to model performance, but to a different question: how do you build boundaries outside the model that hold even when the model is fooled?

In this article, I organize Muse's tech stack into four layers — infrastructure, application, AI architecture, and secret handling — based on Meta's public materials and the analyses and verification pieces written about them.

Infrastructure: one Linux VM per user (the Muse Secure VM), split inside into two security domains
Application: the agent itself and its tools are confined to a "runtime cell"; the actual connector work happens in privilege-separated workers outside the cell
AI architecture: Muse Spark at the core, an agent loop that assembles context and executes tools, and memory stored as files
Secret handling: the agent only ever holds surrogate tokens with no privileges; the real credential is swapped in only at the moment of sending

A note on sources. This article is based on public materials. I am a Muse user, but I have not seen Meta's internal implementation. Where Meta has not disclosed something, I say so explicitly.

How to read this article: separating fact, observation, and speculation

The statements below rest on public materials as of October 23, 2026, plus observations from the Muse session used to write this article. Because these sources are of different natures, I keep three categories distinct throughout:

Category	Meaning	Example
Fact (public sources)	Stated in Meta's official announcements or engineering blog, or in analyses quoting them directly	The Secure VM's two-domain split; Sentinel's role
Observation (my environment)	Values actually measured on the VM assigned to the writing session	2 CPU cores, ~7.7 GiB of memory (details below)
Speculation	My interpretation from public materials and product behavior; not confirmed by Meta	The internal module split of the application layer

The "observations" in particular are one example, from one session, at one point in time — not a published free-plan spec. I found no material in which Meta publishes VM sizes per plan.

The four main sources are listed in the References section at the end: Meta's safety engineering post "How We Built Safety Into Muse," the Muse Spark Safety & Preparedness Report, a Qiita article analyzing 198 posts on X together with the technical blog, and technical explainers by explainx.ai and Forkast.

The big picture

The organizing idea is to separate "the smart part" from "the part that holds power," even inside a single VM (Figure 1).

Figure 1: Overview of the Muse Secure VM. The agent "proposes" actions; only Sentinel, on the host side, can permit execution. Colors: blue = user, green = runtime cell, red/orange = host-side safety components, purple = external services (created with Pele)

Layer	Main components	Role
Client	iOS / Android / Web (muse.ai) / Mac / WhatsApp	Request input, approval dialogs, presenting results
AI	Muse Spark 1.3	Reads context and picks the next step (text or a tool call)
Application	Agent harness, tools, skills, connector CLIs, browser component	Does the work — but holds no privileges
Permissions & secrets	Sentinel, hatch-authd, privsep workers, hatch-safety	Permission decisions, credential storage and swapping, input/output inspection
Infrastructure	Muse Secure VM (Linux), runtime cell (systemd-nspawn), eBPF / LSM	Cell vs. host separation, kernel-level tracking of data flow

1. Infrastructure: one Linux VM per user, two domains inside

The Secure VM

Muse assigns each user a dedicated cloud VM. Meta describes it as an isolated Linux computer with a browser, CPU, memory, and storage — the "system of record" for the agent's activity.

Measuring the VM assigned to my writing session gave the following (observation):

Item	Observed value (2026-10-23)
OS	Linux x86_64 (kernel 7.0 series)
CPU	2 cores (host: AMD EPYC 9D25)
Memory	~7.7 GiB
Home disk	100 GB
Swap	none

To repeat: this is a single example. You cannot conclude "the free plan has these specs." What you can say is that, at least in this session, the environment was a usable working Linux machine, sufficient for running shell scripts and small builds.

The runtime cell and the host-side domain

The inside of the VM is not trusted uniformly. According to the public materials, it is split into two domains with different trust levels.

The runtime cell is where the agent itself, the browser, and the tools run, implemented as a systemd-nspawn container:

Root inside the cell is mapped to an unprivileged user on the host — even if an attacker gains root inside the cell, they remain an ordinary user on the host
It has its own root filesystem, separated from the host side
System calls are filtered; dangerous ones such as io_uring are blocked
Capabilities such as CAP_SYS_PTRACE (inspecting other processes) and CAP_NET_ADMIN (reconfiguring the network) are stripped

The host-side domain holds the safety-relevant components as separate systemd units, outside the cell:

Component	Role
Sentinel	The sole permission authority for connector actions and outbound traffic
hatch-authd	Stores credentials (OAuth tokens, etc.) and performs surrogation, described below
Privsep workers	Run the actual connector work outside the cell, with narrow privileges
hatch-safety	An independent classifier ensemble that inspects model inputs and outputs
Postgres	Persists application state, separated from both the cell and the credential store

Communication between the cell and the host side is limited to Unix domain sockets, with the sending process verified at kernel level via SO_PEERCRED. The characteristic move at this layer is to replace "we trust the peer by convention" with "the kernel verifies who the peer is."

2. Application: the agent proposes; execution happens outside the cell

The agent harness

The program that assembles context around the model and mediates tool calls is called the harness here. Meta's internal codename is "Hatch," traces of which remain in service names like hatch-authd. User reports describe system files such as AGENTS.md being visible from inside the product, and a roughly 350 MB Rust harness binary having been observed. That is user reporting, not a Meta announcement, so I treat it in this article as an external report.

In my own environment, I can also confirm that Markdown files recording persona and memory (IDENTITY.md, MEMORY.md, and so on) exist as system files and that the agent reads them as it works (observation). Having persona and memory live outside the model weights, as files, is one of this product's most legible design traits.

Tools, skills, connectors

The application layer seems to divide into three kinds of components (this is my organization from product behavior, not Meta's official taxonomy):

Tools: basic operations — shell execution, file read/write, web search, browser control, creating events and documents
Skills: playbooks for tasks such as reservations, shopping, and email management, so the model follows a procedure instead of inventing one from scratch each time
Connectors: connections to external services such as Gmail and Google Calendar; OAuth grants are stored separately from the agent proper

The key design point: the connector CLIs inside the runtime cell only parse arguments; they never touch credentials. Typed arguments and file descriptors are passed over a Unix socket to a privsep worker outside the cell, and only there does the real work that touches credentials happen. Even if a prompt injection rewrites the tool-calling code inside the cell, there is simply no path from inside the cell to the credentials.

The browser component

Web operations are handled by a separate component running a real Chromium. What the public materials reveal about how it "sees" is interesting:

Pages are given to the model only as accessibility-tree snapshots, not as raw DOM
JavaScript cannot be executed in the page's own context, and developer tools are disabled
Separate classifiers watch for prompt injections arriving via the DOM, images, and files
The agent pauses while the user is operating the browser manually, or while credentials are being autofilled

Instructions planted in page scripts are, by construction, unlikely to reach the model at all.

3. AI architecture: the model, the agent loop, memory as files

The core model

Muse's core is Muse Spark 1.3, released on September 2, 2026. Reported pricing: a free tier up to 100M tokens per week, Power at $20/month, and Maximum at $100/month. A higher-end model, Muse Wonder, was announced on October 12, 2026, with Muse integration rolling out to subscribers first — but the model running in my writing session was Muse Spark (observation).

The agent loop

The basic behavior is the loop common to many agent products (Figure 2).

Figure 2: The agent loop and memory. Memory files outside the model, plus background processing, are what make "it keeps working after you close the app" possible (created with Pele)

Assemble context from the request, system instructions, retrieved memory, and conversation history
The model chooses: answer in text, or use a tool
Feed the tool result back into context and repeat from step 2

Long conversations are summarized along the way, with only the important parts carried forward in compressed form. Continuity is maintained not by "remembering everything verbatim," but by summaries plus memory written to files.

Background processing

The "keeps working after you close the app" promise is carried by this layer. On top of scheduled jobs (cron), event-driven hooks, and resident services, there is processing that organizes memory between conversations and does proactive research. The Qiita article relays a user's story of being notified, unprompted, about a recall on a car they had just bought — plausibly an example of this proactive processing at work (speculation).

4. Secret handling: the agent never holds the real keys

This is the layer I most want to introduce.

Credential surrogation

What the agent receives is not a real OAuth token or password, but a surrogate token that carries no privileges by itself. Only when Sentinel permits a connector action is the real credential swapped in, at the moment of sending (just-in-time insertion). Figure 3 shows the flow.

Figure 3: Credential surrogation. Even if a prompt injection tricks the agent into "sending the password," the agent never had the real thing to begin with (created with Pele)

This is a decisive difference from the "instruct the model not to leak" design. Instructions are advice; a fooled model breaks them. But "you cannot leak what you do not hold" survives the model's judgment failing.

Email gets additional care: one-time codes, password reset links, and magic links are removed before the agent ever sees the message, so that email access cannot become a stepping stone to taking over other accounts.

Sentinel and taint tracking

For every connector action and every piece of outbound traffic, Sentinel decides: permit, deny, or ask the user. Decisions are made at both L4 (destination host, resolved IP, port, protocol) and L7 (HTTP method, path, content), and are said to include defenses against SSRF techniques such as DNS rebinding.

The second mechanism tracks data flow (Figure 4).

Figure 4: Tainted egress. The taint marking is done by the kernel, via eBPF cgroup programs and LSM hooks — it does not depend on the model's self-report (created with Pele)

A tool execution starts in an untainted state
Reading the user's data marks the process as tainted
The propagation of the mark is tracked by eBPF cgroup programs (used to capture traffic) and eBPF programs attached to LSM hooks
Narrow untainted traffic may be permitted automatically, but tainted traffic never passes automatically — it goes to the user for confirmation

The point is that "is this action safe?" is judged not by the type of action, but by whether the user's data was touched along the way.

Approval happens on a separate screen, not in the chat

User approval is likewise obtained not as chat text, but in a dedicated client-side dialog, and the answer returns directly to Sentinel without passing through the model. Approval scopes come in five kinds: once only, for the session, for the task, time-boxed, and always.

Keeping the approval UI out of the stream of model-generated text matters, in my reading. If "Do you approve?" were rendered inside the same chat that attacker content also flows into, that rendering itself could become a target.

Shopping payments

Payments don't hand over a real card number either. For each purchase, the user approves the total amount, and for a new store, Stripe Link issues a single-use virtual card scoped to that store, amount, and expiry. If the card number leaks from the store, it cannot be used for anything else.

5. Five layers against prompt injection

The public materials describe the prompt-injection defenses not as one mechanism but as five layers:

#	Layer	Content
1	Model training	Muse Spark 1.3 itself is trained not to follow instructions smuggled into external content
2	Harness labeling	Content from external sources (web, email, files) is explicitly labeled "untrusted" when placed into context
3	Classifier ensemble	Independently trained injection-detection classifiers run in parallel over external data (hatch-safety)
4	Human approval	Any operation that moves data out of the VM goes to the user for approval
5	Kernel-level boundary	The runtime cell, privilege separation, hatch-authd access control, Sentinel — designed to hold even if layers 1–3 all fail

Layer 5 expresses the philosophy: do not bet safety on the model being right. The model is treated as something that will make mistakes, and the last line of defense is placed in the kernel and in processes the model cannot control.

6. Safety evaluation of the model itself

Separate from the product's defenses, an evaluation of the model alone has also been published. The Muse Spark Safety & Preparedness Report is a 159-page report assessing the chemical/biological, cybersecurity, and loss-of-control domains under Meta's Advanced AI Scaling Framework.

Key points:

In pre-mitigation evaluation, chemical and biological capabilities were judged likely to reach "high risk"
With layered mitigations in place, the published judgment is that residual risk is acceptable for deployment inside Meta AI
A third-party evaluation (Apollo Research) also reported high "evaluation awareness" — the model's ability to notice it is being evaluated

Note carefully: this report's primary subject is "Muse Spark as the core model of Meta AI," not "the operation of the Muse personal-agent product as a whole." A model evaluation does not by itself guarantee product safety. This article therefore treats the model's story (this section) separately from the product's defenses (sections 1–5).

7. Open questions that remain

Even a carefully built design leaves open questions, visible in both the public materials and user reports.

It is not hidden from Meta itself

The current standard VM is isolated from other users and from the internet, but it is not concealed from Meta. Confidential VMs — where the user holds the keys and Meta cannot access the contents — have been described as planned for the second half of 2026. "Isolated" and "invisible even to the provider" are different claims, and for this product, only the former currently holds.

No independent audit yet

Most of this article's "facts" ultimately trace back to Meta's own engineering blog and launch-day announcements. The explainers are detailed, but the primary source is Meta. As of this writing, I could not confirm any published independent audit by a third party. The bug bounty — up to $300k, and up to $130k for a successful prompt injection affecting a single user — is evidence of how seriously Meta takes this attack class, and at the same time, pricing that assumes such events can happen.

Blocked at the door

According to the Qiita article, investment bank Jefferies tested several hundred major US sites and found that about a third refused Muse and about a third imposed verification or restrictions. Amazon began blocking Muse on September 21, 2026. However cleanly the agent side separates credentials, whether the other party's site accepts agents at all is a separate problem — the bottleneck is shifting from model quality to "can it get in the door at all," as the same article puts it.

Misreading what a permission means

The same Qiita article relays a Marketplace case: a user who had chosen "always allow" expected to be asked each time, while the agent accepted an offer and sent the user's address to a buyer. Not a credential leak, but an accident of misjudged permission scope. The more finely approvals are divided, the easier they are to misjudge; the narrower they are, the more approval fatigue grows. That tension remains no matter how strong the defenses are.

Limitations

Finally, what this article cannot claim:

The diagrams were drawn by me from descriptions in the public materials; they are not Meta's internal design documents. Where sources describe things differently, I prioritized the more concrete descriptions (the Qiita article and explainx.ai).
I could not fetch Meta's original engineering blog (research.meta.ai) directly from my writing environment, so its content was verified through quotations and summaries in the Qiita article and overseas explainers. For the primary source, follow the link in the References.
The VM spec observation is one example from one session, and may differ by time, plan, and region.
The reported user voices may skew toward early adopters and newsworthy cases. The Qiita article itself notes that posts on X do not represent all users, and that its sentiment classification has methodological limits.
Announced-but-unreleased items such as Muse Wonder integration and Confidential VMs are treated as "planned," distinct from shipped features.

Conclusion

What stays with me about Muse's design is not the model's cleverness, but the unglamorous components around it, built on the premise that the model is not to be trusted: systemd-nspawn, eBPF, SO_PEERCRED on Unix sockets, privilege separation — a combination of well-worn Linux parts, solving the problem of not handing real keys to an agent that might be fooled, and not letting it decide for itself what may leave.

Agent safety cannot be measured by model refusal rates alone. "On the day the model is completely fooled, what still holds?" Muse's public materials are, to my knowledge, one of the most concrete attempts yet to answer that question from the infrastructure side.

References

Meta, "Introducing Muse: The World's First Personal AI Agent Built for Everyone," 2026-09-08. https://about.fb.com/news/2026/09/introducing-muse-personal-ai-agent/
Tarek Sheasha (Meta Superintelligence Labs), "How We Built Safety Into Muse," 2026-09-08. https://research.meta.ai/blog/security-and-safety-for-ai-agents-our-approach-with-muse
Cristina Menghini et al., "Muse Spark Safety & Preparedness Report," arXiv:2606.12429, 2026. https://arxiv.org/abs/2606.12429v1
Hisashi Ito, "Xの投稿198件と技術ブログで見る、MetaのAIエージェント Muse の向き不向きと安全設計" (in Japanese), Qiita, 2026-10-11. https://qiita.com/hisashi-ito/items/ffe164ae4aa88abb293e
explainx.ai, "Meta Muse: Personal Agent + Sentinel VM Security (Sept 2026)." https://www.explainx.ai/blog/meta-muse-personal-agent-launch-sentinel-vm-security-2026
Blair Hayes, "Meta's Muse Agent Lives Behind a Kernel-Level Sentinel," Forkast, 2026-09-24. https://forkast.news/metas-muse-agent-lives-behind-a-kernel-level-sentinel-the-architecture-reveals-where-agent-security-is-heading/
Fello AI, "Meta Muse AI Agent: What It Does and What It Costs." https://felloai.com/meta-muse-ai-agent/
Stripe, "Stripe helps Muse, Meta's new personal AI agent, shop across the internet with Link," 2026-09-08. https://stripe.com/gb/newsroom/news/stripe-helps-meta-muse-shop-with-link

All figures in this article were written in Mermaid, converted to SVG with Pele 1.8.0, and then to PNG.

What I Could and Couldn't Verify Self-Hosting Open Models: Colab, RunPod, and an API Direction for Final Reviews

SeaOtter ( Kentaro Mori ) — Sat, 17 Oct 2026 10:39:07 +0000

Summary

I started from a Qwen Gradio app that runs in Google Colab, then rented GPUs on RunPod to connect open-weight models to Open WebUI. That happened in three phases:

Phase 1 (2026-10-09): a small Qwen2.5-1.5B-Instruct model returned one reply through Open WebUI.
Phase 2 (2026-10-15): Qwen3.8-27B in BF16 returned one reply the same way. I took no quality or speed benchmarks.
Phase 3 (2026-10-15): I tried to run a GGUF build of GLM-5.3-Flash on an H200 SXM. The container image pull completed, and I stopped the run while the model weights were still downloading. That produced zero model outputs and no A/B evaluation.

The same day, my published final retrospective ended the RunPod evaluation. For ordinary review of code, documents and final deliverables, it records the MiMo-V2.6-Pro API as an unverified operational candidate (no API account, key, call or result has been verified).

My own account today, which the public repository doesn't contain: ordinary code, prose and final-artifact review became the recurring use case, and my usage frequency and the operational burden of running GPUs led me to take the MiMo-V2.6-Pro API as my practical direction. I have not called it, integrated it, benchmarked it, or shown it to be cheaper or better. I also don't consider the self-hosting work wasted. It remains useful technical learning for me.

The rest of this post covers the setup, the order of the work, what the records actually support, the obstacles and how I triaged them, the work that never shows up on a GPU bill, the questions that separate an API from self-hosting, and where self-hosting still makes sense.

How to read the numbers

I separate four kinds of statement in the text: measured values from the records, estimates, my own decisions, and general inference.

Dollar figures are estimates. Each is a measurement window multiplied by the rate shown at the time. The window runs from just before Deploy until the $0/hr display after Stop. It is not response latency and not GPU-only time, and none of the figures is a final invoice. The Phase 1 figure covers only its final trial. Because the windows differ and no invoice has been reconciled, I don't add the phases into a total.

Links point to the public docs on main.

The setup

This is a simplified view of Phases 1 and 2. Phase 3 had the same pod-side shape with llama.cpp as the server, but it never got as far as a model response.

Open WebUI
   │  authenticated HTTPS, through RunPod's HTTPS proxy
   ▼
GPU pod on RunPod
   └─ inference server (Phases 1-2: vLLM / Phase 3: llama.cpp server)
        └─ open-weight model (Qwen, then a GLM-5.3-Flash GGUF)

Phase 1: an authentication guard sat in front, and only GET /v1/models and POST /v1/chat/completions were allowed through. vLLM listened behind it on loopback. (Phase 1 record; this record is in Japanese only.)
Phase 2: the vLLM API key was injected from a RunPod Secret reference, and a local Open WebUI used an OpenAI-compatible connection. An unauthenticated /v1/models request returned 401 before I sent anything. (Phase 2 record)
Phase 3: a llama.cpp server image pinned by digest, told to fetch GGUF weights itself. (Phase 3 trial record)

For Phases 2 and 3, I set pre-approved limits before starting (Phase 2: 45 minutes and $3; Phase 3: 75 minutes, $7, and six stop conditions). In Phase 2 I finished the UI-side setup before the GPU started, then ran Deploy, one request, Stop, a $0 spend-rate check, and Terminate in one sequence. In Phase 3 I stopped and terminated under those limits.

I also designed a job API and runner that would save results before stopping the pod, but never built it (README). The RunPod lifecycle code lives in a separate repo, qwen-runpod-operations, according to the Phase 2 record. I haven't re-read that repo for this post.

The order of the work

A Qwen Gradio app on Colab. The repository began as a Qwen Gradio app and then tried on-demand RunPod inference (README). The README lists two Colab notebooks, one chat-only and one multimodal. The app is still in the repository, but I did not deploy and verify the whole app or its multimodal workflow on RunPod (final retrospective). The public records don't say why I moved to RunPod, so this post doesn't either.
Phase 1 (2026-10-09). A minimal connectivity test with a small model.
Phase 2 (2026-10-15, PR #44). The same path with a 27B-class model, as one short connection test.
Phase 3 (2026-10-15, PR #45). A planned comparison of three GLM variants (A: a GGUF of the official model; B: an abliterated GGUF variant; C: a security-tuned FP8 build based on full GLM-5.3 rather than Flash). I finished the GPU-free preparation and started track A. The README treats the whole plan as historical.
Closing (2026-10-15, PR #46). The final retrospective ended the evaluation and recorded the API direction for review work.

What the records actually support

Phase	Setup	What was verified	Window and estimate	Not verified
1 (Oct 9)	RTX PRO 4000 24 GB, Qwen2.5-1.5B-Instruct (vLLM)	One reply displayed in Open WebUI. I asked for Japanese and got the English `Connected successfully.`	Measurement start to first stop confirmation: 319.8 s, about $0.051 (final trial only)	Answer quality, instruction following, speed
2 (Oct 15)	A100 SXM 80 GB, Qwen3.8-27B BF16 (vLLM)	One reply through Open WebUI's OpenAI-compatible path, in Japanese as instructed: 「接続確認できました」	Just before Deploy to the $0/hr display after Stop: 875.1 s, about $0.44 (at $1.80/hr)	Quality, speed, long context, image input, the manual-hook route, lifecycle automation
3 (Oct 15)	H200 SXM 141 GB, GLM-5.3-Flash GGUF (llama.cpp)	Image pull completed (digest matched). Weights are 108.7 GB in total; container-disk usage reached about 69 GB at 23:29 JST, when the fetch rate had fallen to about 1.6 GB/min, and I stopped the run two minutes later	Just before Deploy to the $0/hr display: 1381.7 s, about $2.04 (at $5.31/hr)	Zero model outputs, no A/B evaluation. B not run; C cost-estimated only

Sources: Phase 1, Phase 2, Phase 3 trial, and the final retrospective. Each run ended with a $0/hr display after Stop and a $0 spend rate after the pod was deleted. The Phase 2 and 3 records add a pod count of 0, and Phase 3 also records the endpoint returning 404.

Two cautions about reading this table:

Phases 1 and 2 are connection tests. "One reply" in Phase 2 means one reply came back through the documented A100 and Qwen3.8-27B BF16 path. It says nothing general about quality or speed.
The Phase 3 preparation included a scorer that I checked against a mock server. Its scores are test-fixture values, not GLM measurements, and I don't use them here.

Obstacles and how I triaged them

The final retrospective is explicit that these were operational faults, distinct from reasoning or answer capability, and that platform or security restrictions shouldn't be read as model inability.

The GPU didn't come back after Stop (Phase 1, earlier trial)

An earlier trial got as far as a ready model but never confirmed a reply in Open WebUI. After I stopped it, I couldn't get a GPU back on the same pod. According to the record, RunPod releases the GPU on Stop, so a stopped pod can't resume if another user has taken the GPU on its original host. The container disk is also wiped on Stop. For the final trial I prepared the UI first and only updated the connection target once a fresh pod was up (Phase 1 record).

Stripped quotes and a restart loop (Phase 2)

I put an argument with a JSON value in the start command. RunPod stripped its quotes, vLLM exited on the argument error, and the pod restarted in a loop. /health returning 404 or 502 can't tell a restart loop from a slow load, so the lesson I recorded is to read the System and Container logs in the first minute. I fixed it by removing the argument and editing the pod. No weights had been fetched yet, so only time was lost (Phase 2 record).

The first request was rejected (Phases 1 and 2)

In Phase 2, Open WebUI 0.12.2 defaulted Function Calling to Native. It sent enabled tools with tool_choice: "auto", and vLLM rejected the request because the server had no tool-call settings. Switching the chat to Legacy and regenerating fixed it. The same switch worked in Phase 1, but there I never confirmed the rejection's cause from the actual payload.

A thinking model and a small max_tokens (Phase 2)

Qwen3.8 starts by thinking, so 64 tokens can run out before any answer appears. Sending chat_template_kwargs = {"enable_thinking": false} as an Open WebUI custom parameter returned only the answer.

Template defaults (Phase 2)

RunPod's "vLLM Latest" template defaulted to a guessable API key and to a HF_HOME that assumed a persistent volume. I replaced the first with a Secret reference and moved the second onto the container disk.

A `/health` that stayed at 502 (Phase 3)

In Phase 3 /health returned 502 the whole time. llama-server doesn't start listening until the -hf fetch finishes, so this wasn't an error signal, and there were no server errors or restart loops. I followed progress through container-disk usage (JST):

Time	Container disk used
23:13	about 16 GB
23:15	about 35 GB (about 9 GB/min)
23:18	about 51 GB (about 5.5 GB/min)
23:29	about 69 GB (about 1.6 GB/min)

The measurement started at 23:08:35.5, and the image pull and container creation finished at 23:10:45. The image pull completed; I stopped the run while the weights were downloading. My stop condition 3 required readiness within 25 minutes of model start, and the projection at the time put fetch completion near 23:54 and readiness near 23:58. I gave the abort order at 23:31:12, the $0.00/hr display after Stop appeared at 23:31:37, and I terminated the pod after that (Phase 3 trial record). The record doesn't establish why the fetch slowed, and I won't guess.

The final retrospective frames Phase 3 as not an image-download failure, not evidence that the model is poor, and not evidence that RunPod is generally unusable. Phases 1 and 2 each reached one reply on RunPod.

GPU supply (Phase 3)

My planned H200 NVL went out of stock right after approval, and switching to the H200 SXM, at about 1.4 times the rate, needed a fresh approval. The record treats GPU supply as one more variable.

Work that never appears on the GPU bill

The public records show these non-GPU items:

GPU-free preparation. Phase 3's P3-0 covered a model registry with pinned revisions (via Hugging Face's public API, with no downloads), a check of the first 8 MB of the GGUF headers, a check that the inference image contained the target model's implementation, an evaluation set, and a scorer validated against a mock server. The first version of the scorer counted HTTP errors as refusals, and I fixed that during validation.
UI setup before the GPU starts. Function Calling, max_tokens, thinking off and Model IDs were finished beforehand, which the Phase 2 record says kept GPU time minimal.
Debugging. About 4 of Phase 2's 875 seconds (about $0.12) went to the argument error. The window includes that time.
Proving shutdown. I checked the $0/hr display after Stop, the $0 spend rate after Terminate, the pod count, and, in Phase 3, that the endpoint returned 404.
Reconciling the bill. In Phase 3 the displayed balance dropped $3.17 against an estimate of about $2.04. The roughly $1.10 gap may be billing lag, but that is unconfirmed. The final invoice is unreconciled, and storage or volume deletion beyond what the reports cover is unverified (final retrospective).
Everything around the model. The retrospective lists environment and dependency compatibility, authentication setup, quote and argument handling, Open WebUI settings, GPU supply, large weight transfer and storage, and shutdown and billing confirmation.

The retrospective also records "about two weeks" as my experienced calendar and preparation effort, explicitly not measured wall time, GPU hours, or benchmark time. I don't break it down further here. Of the items above, the only durations recorded are Phase 2's roughly 4 minutes on the argument error and each run's measurement window. No public record measures the rest of the preparation and checking time.

API or self-hosting: the questions I would ask

Read this in three layers:

Public record (PR #46's retrospective): consider the MiMo-V2.6-Pro API (mimo-v2.6-pro) as an operational candidate for ordinary code, document and final-artifact review. It is unverified: no account, key, call or result, no integration, and no measured same-workload comparison with self-hosting. That status stays as recorded (final retrospective).
My present account (not in the public record): the recurring review use case, my usage frequency and the GPU operating burden led me to choose the API as a practical direction. It is my decision, not a measured result.
Measured results: none for the API. No calls, integration, benchmark, or cost or quality comparison, and no claim that it is cheapest, best, or ranked anywhere.

The retrospective also notes that the model's weights are published as open weights, so it isn't API-only.

The questions below are general inference, not measurements from my setup. I did not check MiMo's current pricing or the individual clauses of its provider's privacy policy against the official pages while writing this, so I leave them out.

How will you compare costs? Compare against an assumed usage volume. A pod bills per hour and an API per token, so the unit prices can't be compared directly. My Phase 1 to 3 figures come from connection-test windows and can't be converted into a per-token cost. Check the provider's current pricing page when you decide.
What happens to your data? Whether code or documents may go to an external API is a question to settle before sending anything. Check the provider's official pages for how data is retained and used, and check the applicable contract, permissions and region. I did not verify these while writing this post. An API isn't inherently private, and self-hosting isn't inherently secure.
How much will you run it? A pod's cost grows with the time you hold it, and an API's cost grows with the tokens you use. Which is cheaper depends on your usage, and I haven't measured it.
How much operating time can you spare? The previous section's non-GPU work is yours to do when you self-host. The records also show cases where I couldn't get the GPU I wanted.
How exposed do you want to be to change? An API's prices and behavior can change on the provider's schedule. Self-hosting means following model and serving-stack updates yourself.

Where self-hosting still earns its place

The retrospective names pinned reproducibility, control over weights, custom requirements, privacy architecture and sustained high utilization as cases where self-hosting may fit. It adds that you still need verified data handling and a measured, same-workload cost comparison. My own additions, as inference:

You need a specific set of weights or a specific quantization. The Phase 3 trial record notes that tracks A and B share a quantization and size but use different weights, so they can be compared fairly (Phase 3 trial record). A comparison that fixes the quantization and size and varies only the weights is awkward to build on a single provider's API.
The input can't leave your environment. Self-hosting is a strong candidate, though you still have to verify how your own environment handles the data.
You want to adjust the serving stack itself. Server settings and launch arguments are only yours to change when you run the server.
Usage is high and steady. If utilization is continuously high, the cost picture can shift. That still needs its own measurement.

RunPod did work for me: Phases 1 and 2 each reached one reply. Ending Phase 3 says nothing about whether RunPod is usable.

Lessons recorded in the repo

These come from the final retrospective and the phase records. They are lessons, not scheduled work.

Check how start-command arguments split locally, and keep JSON-valued arguments out of the start command (use environment variables or a config file).
Read the container logs in the first minute, because /health 404 or 502 can't separate a restart loop from a slow load.
Finish all UI settings before the GPU starts.
Fetching large weights is the main risk. The recorded ideas are to pre-fetch onto a network volume in the same region from a cheap CPU pod, to use a parallel downloader and verify SHA-256 against the registry, and to keep the GPU and the volume in the same region.
Split the readiness stop condition into fetch and load. Mine was written for startup failures and ended up firing on download speed.
Approve a fallback GPU and its limit up front, so a stock change doesn't stall the run.
Define shutdown proof: a $0 spend rate after Stop, zero pods after Terminate, and the endpoint unreachable (404).
Inject API keys from Secret references, always override a template's default key, and keep keys, endpoint URLs and pod IDs out of public docs.
Report operational faults separately from model capability.

What this post doesn't claim

A total cost across phases. Each figure is a window-based estimate, not a final invoice.
A quality, speed, long-context or image-input benchmark for any model.
Any result for the GLM models (Phase 3 produced zero model outputs).
A cause for the slow weight download in Phase 3.
A reason for ending the evaluation, or for moving from Colab to RunPod. The public records give neither.
Any measurement of the MiMo-V2.6-Pro API's behavior, quality, latency or cost, any check of its data handling, or any integration status.
MiMo's current pricing or any individual clause of its privacy policy. I didn't verify those on the official pages.
That RunPod or any other service is unusable, or that an API is always cheaper or easier.

Sources

Repository: moruku36/open-llm-multimodal-lab (README)
Phase 1 verification record (Japanese only)
Phase 2 verification record (PR #44)
Phase 3 trial record (PR #45)
Final retrospective (PR #46)

How I use scheduled actions in Gemini to keep up with technology

SeaOtter ( Kentaro Mori ) — Wed, 14 Oct 2026 09:05:19 +0000

When I read about AI, cloud services, and security, I use several kinds of sources. Papers explain new methods, release notes describe product changes, and engineering posts describe what people tried. I look for different things in each source. One combined list can make it hard to decide what to read more closely.

I use eight information-gathering routines with scheduled actions in Gemini. For papers, I ask what is new and how it might help my development work. For cloud updates, I ask whether the change could affect my systems. I treat the schedule and the questions I want a summary to answer as separate choices. Here is how I set up those requests.

These are my settings as of October 13, 2026. All times are Japan Standard Time. The schedules below describe the configured frequency, rather than a log confirming every run.

Find papers and practical engineering advice

1 Select ten papers and take a closer look at one

I schedule the paper request for 7–8 a.m. every day. I ask for ten papers, preprints, or technical reports from research institutions and company research teams. Preprints are papers shared before formal journal publication. For each item, I ask what is new and how the proposed method or system works.

To help choose relevant papers, I provide context from my public GitHub work and my professional interests in AI, security, and cloud services. I then ask for a closer look at one especially relevant item, including ideas for using it in development. The aim is to help me choose which paper to read in full.

2 See what developers in overseas communities are discussing

Every day at 7:30 a.m., I ask Gemini to check Hacker News, a site for technology discussions, and the Reddit communities r/programming, r/MachineLearning, and r/devops. The request focuses on new open-source software, technical debates, and opinions about different technologies. Open-source software makes its source code available for use and modification under its license.

I ask for a short explanation of each topic, why it is getting attention, and what people like or criticize about it. I also ask for links to the original posts so I can read the discussions myself.

3 Collect lessons from real development work

At 8 a.m. every day, I ask for ten practical articles or resources about AI, cloud services, and security. Sources include Japanese and international engineering blogs, knowledge-sharing sites, and conference presentations. I look for accounts of using a technology, fixing problems, and designing systems.

Alongside the official documentation, I want to know what went wrong for people who used the technology and how they solved it. When selecting papers, I focus on new methods. For practical articles, I ask what to watch out for when using a technology.

4 Check how cloud updates could affect my systems

I have a separate request to check official release information and blogs for AWS, Azure, and Google Cloud in the morning or periodically. It covers new features and previews, which let users try features before their full release. It also covers features that are no longer recommended and changes that could stop existing setups from working.

I ask for both a summary of the update and its possible effect on my infrastructure: the servers and cloud services I use together. I want to find features worth trying and updates that may need a closer check, such as a possible configuration change.

Choose a schedule that fits the information

5 Follow updates on a specific topic

I also have a request to follow a topic over time. One example is reporting about a possible OpenAI stock market listing, or IPO, along with partnerships and changes in how the company is run. I ask it to check financial media and official sources periodically and alert me to changes. The IPO is a topic I follow, not a confirmed event.

6 Read short news summaries in the morning and evening

My general and technology news request is scheduled twice a day, in the morning and evening. I ask for short summaries of current affairs, economic news, and technology from major news outlets. This helps me follow broader news alongside my specialist interests.

7 Find technical videos with explanations and demonstrations

I ask for technical video recommendations every day from YouTube and similar sources. Topics include testing AI models, ways of organizing development work, and security. These give me options when I want to learn by watching an explanation or demonstration.

8 Choose books on the first day of each month

On the first day of each month, I ask for technical books, including new releases, that match my interests and recent technology trends. I use a different schedule for books that take time to study than for news I want to check each day.

Use the information in development work

Choose sources and a schedule → Ask for details that fit the purpose → Read the original sources → Check test conditions and my environment → Choose what to read or try next

I ask for different details in each of the eight routines. For papers, I want a candidate for deeper reading and possible uses. For community discussions, I want to understand the interest and disagreement. For practical articles, I want things to watch out for during implementation. For cloud updates, I want the possible effect on my setup. I request these details to help me decide what to read or try next.

For example, suppose I wanted to improve an app that searches documents and uses them to answer questions. A paper might suggest a new method. An engineering post might explain problems to watch out for when using it. A cloud update might introduce a useful feature. This is only an example of how the information could fit together. It does not describe a change I made or an improvement I measured.

Before using an idea, I would still need to read the original paper or official documentation. I would check how it was tested, which versions it supports, and whether it fits my environment. A popular technology may be worth a look, but those details matter when deciding whether to use it.

An example request you can adapt

The example below was written for this article using the routines described above. It is an example you can adapt, not the original prompt from my setup.

“My interests are AI, cloud services, and security. Use the public GitHub context I share to select ten relevant papers, preprints, or technical reports from research institutions and company research teams. For each item, explain what is new and how the method or system works. Choose one especially relevant item and explain it in more detail. Suggest possible uses in development and list what I should check before trying it. Include links to the sources. Separate what you verified from what you inferred. Say if you could not read the full text of a source.”

I added the instructions about source links and what the AI could verify as suggestions for readers writing their own requests. A detailed prompt does not guarantee a correct answer. Before sharing project information, check whether it is appropriate to share that information with the AI service.

Start with one topic you want to follow

The eight routines serve different purposes. Some give a quick news overview, some track changes, and one takes a closer look at a paper. I adjust the schedule and the requested details to match what I want to learn.

To try this approach, choose one topic you care about. Decide where to look, how often to check, and what decisions the information should help you make. I use the summaries as a starting point for choosing which sources to read more closely.

Is AI-Assisted Work Cheaper? A Five-Day Cost Estimate

SeaOtter ( Kentaro Mori ) — Sat, 10 Oct 2026 13:15:48 +0000

Prepared with AI assistance using my work records and a cost-estimate report. The cost comparisons are estimates.

I went back through five days of work done with AI tools and tried to put a price on it, next to what the same work might have cost without generative AI. This post walks through how I did that and what the numbers do and don't say.

A word on framing: everything below is an estimate. None of it is a measured saving. The part I think carries over to other projects is the method: describe exactly what changed, attach evidence, and count human review time before comparing costs. A simple task list is enough. You don't need to build a complicated AI system first.

What actually happened

I reviewed work from October 5–9, 2026, in Japan Standard Time. The report groups it into 35 task-record units across software changes, research, documentation, media preparation, maintenance, and administration.

Those 35 units differ a lot in size and value. Some are improvements to existing applications. Others are documents or small maintenance jobs. A count like this isn't a productivity score, and it definitely doesn't mean 35 new products were built.

Three examples:

An existing memory app received improvements to display, storage, synchronization, backup, and search, plus documentation.
A voice application received recording, playback, and interface repairs, with Windows device checks recorded. Several related fixes were counted together.
A cloud experiment produced one real answer from a small AI model through a web interface. The temporary server was deleted afterward. That completed the experiment's stated stage; a continuing service was outside its scope.

For each entry I needed one sentence describing where “done” ends. Without that, a small successful test can quietly turn into a claim about a whole system.

Evidence and limits go together

Where public code changes exist, the report links them. Other entries rely on work records or my own completion confirmation. Those are different kinds of evidence, so they're labeled differently.

A few limits worth spelling out:

Checking a code-change record does not rerun the software.
A recorded Windows check does not show that every device works.
One browser-based game was confirmed in Chrome, while testing on a real Safari device remained incomplete.

If you want to try this on your own project, four fields per entry go a long way:

The specific change or deliverable.
The evidence available, such as a code-change link or a test result.
The environments or conditions checked.
What remains unfinished or unverified.

Related retries belong under the same outcome. Otherwise a hard repair inflates the task count just because it took several attempts.

How much human work is that?

Next question: how much work might experienced people need to reach the same recorded scope without generative AI?

The assumptions still allow existing repositories, ordinary scripts, templates, tools, and domain knowledge. Rebuilding every application from scratch would describe a different job.

The bottom-up estimates were about 180 hours (low), 310 (central), and 510 (high). The central calculation uses exactly 309 hours.

At an assumed 160 working hours per month, 309 hours is 309 / 160 ≈ 1.93 person-months. That is a unit conversion of the estimate, not a staffing plan or a delivery date.

These were retrospective judgments, not stopwatch measurements or observed hours saved. The range reflects different assumptions about reuse, investigation, and revision. It is not a statistical confidence interval.

I also didn't treat all waiting as labor. A download or an automated test takes time without someone watching it the whole way. Dependencies matter too: a demo has to work before it can be filmed. Total estimated labor can't promise a completion date.

The cost assumptions, spelled out

At a hypothetical JPY7,500 per hour, the central human-work scenario is:

309 hours × JPY7,500 = JPY2,317,500, or about JPY2.32 million.

That hourly rate is a planning assumption, not a verified market rate.

The AI-side reference amount is JPY5,209.92, about JPY5,210. It combines five days of allocated subscription payments with one RunPod prepaid-credit purchase. The credit purchase is money added to an account; actual computing consumption during the period is unknown.

The subscription figures come from the report's billing evidence and user-reported payments, not from current advertised prices. Some allocation details are still uncertain, including the ChatGPT billing cycle. The report also uses an assumed currency conversion where needed.

The most important caveat: that JPY5,210 leaves out my instructions, approvals, and review time, along with other costs. Comparing it directly with a complete human-work estimate would hide a major part of the workflow.

So the report adds a scenario. Suppose human management and review took 40 hours at JPY7,500 per hour. That is 40 / 160 = 0.25 person-months, and the combined amount becomes JPY305,209.92, about JPY305,210. That's roughly 87% below the central hypothetical human-work cost.

![Cost scenarios comparing the central human-work estimate with allocated AI payments and an assumed 40 hours of human oversight.

Figure rendered programmatically with AI assistance. These are planning scenarios with different cost boundaries. The 87% difference is not measured return on investment, and equal output quality has not been independently verified.

The low human-work scenario is JPY892,500, using both lower hours and a lower assumed rate. That smaller baseline matters: rework, mistakes, and extra supervision can eat the apparent advantage, and their actual costs weren't measured here.

A workflow you can check

If you want to try something similar, start with a bounded task: fix a reproducible problem, update a known document, or investigate a specific question. Define the acceptance check before you hand the work off.

![A human-led workflow in which a person defines the task, AI carries out bounded work, and a person reviews evidence before accepting the result.]

Figure rendered programmatically with AI assistance. Recommended division of responsibility. This diagram is not a claim that every task during the reviewed week followed an identical process.

Keep a person responsible for production changes, confidential information, safety, and decisions that need specialist knowledge. If you can't evaluate the result yourself, bring in someone who can.

What I take from this

With the caveats above in place, these estimates favor human-led AI assistance for work with a clear scope and a result you can check: a person keeps the requirements, decisions, and final acceptance, and AI does bounded work in between. That is the setup I'd recommend trying first.

The review also shows why careful records matter more than an impressive task count. For your own trial, record both the output and the human effort needed to make it usable. A small, well-documented comparison will tell you more about what to delegate next than a big number will. One week doesn't make a case for headcount reduction or unattended operation.

Source: Full report with task boundaries, evidence, and calculation assumptions, in Japanese.

Seven Days Managing an AI Development Team

SeaOtter ( Kentaro Mori ) — Wed, 07 Oct 2026 13:23:35 +0000

On the first day, I planned nine tasks. Three got done.

The other six ran into familiar problems: decisions waiting on me, prompts that I had to copy into local GUIs, and uncertainty about what was actually progressing. I had delegated the work but was still carrying much of the coordination.

From October 1 to 7, 2026, I kept a daily log of running a small AI team for personal projects. I used ChatGPT dots as the coordinator, which I call “Docchee,” with other AI tools handling implementation, review, and research.

By Day 7, I was asking the team to switch priorities for an urgent application-preparation task. Between those two points, I learned more about handoffs, approvals, and visibility than I expected.

This is a field report from one person's workflow. It is not a model benchmark or evidence that the setup can safely handle a production incident.

The team I was trying to run

I work in cloud engineering, but for this experiment I took the Product Owner role: define the goal, choose priorities, approve consequential actions, and test whether the result was useful.

The working split looked like this:

Role	Responsibility
Me	Goals, priorities, approvals, and acceptance testing
Docchee	Coordinate tasks, choose an execution environment, and report status
ChatGPT / Codex	Design, review, and implementation when needed
Claude Code	Implementation and fixes
Gemini / Antigravity	Share prototyping, research, and other suitable work
Qwen	A separate track of local and GPU-backed experiments

These were working roles, not a fully automated routing architecture. Assignments changed with the task, available usage allowance, and access to the right environment. I keep the broader workflow in multi-ai-workflow.

Days 1–2: Handoffs were my problem too

My initial assumption was that handing out more tasks would get more work moving. Instead, it produced more places where someone needed my input.

At the start, I was manually moving prompts between interfaces. CLI access was not ready. Some tasks needed a decision I had not made explicit; others needed a local action only I could perform at that point.

The first useful adjustment was simple: make the end goal clearer, and have Docchee tell me what remained instead of making me repeatedly ask.

By Day 2, a workable rhythm was emerging. Human-in-the-loop approvals remained difficult. I wanted routine work to move without interruption, while keeping meaningful control over publishing, permissions, and other consequential actions.

That required more precise task boundaries. “Keep going” was not enough information for every next step.

Days 3–4: Local access improved, but approval boundaries remained

On Day 3, I focused on integrating the team with my Windows environment and preparing CLI-based workflows. Reducing manual copy-paste was a practical improvement.

Usage allowances also became part of task assignment. A difficult design or review could justify a stronger model, while other work could go to a different tool with more capacity available.

I was excited enough in my daily post to describe this as the start of token-aware coordination. The precise state was more modest: we were using available displays and reports to make allocation decisions. Continuous usage collection and dynamic routing were still goals, not a completed control system.

Day 4 exposed another boundary. A task routed through dots to ChatGPT or Codex could still stop for a separate approval. Using tools from the same provider did not make the entire workflow share one permission state.

The useful question became: what exact operation is blocked, in which environment, and what would let it proceed? A generic “waiting for approval” label left too much work for me to reconstruct.

Day 5: Voice instructions and a 30-minute cadence worked well

On Day 5, I ran much of the workflow through voice input. For me, speaking an instruction or review request was often faster than composing it at a keyboard.

I also introduced a status update every 30 minutes across the parallel tasks:

I set the goal and priority.
Docchee organized the work and assigned it.
I reviewed progress and blockers at the next check-in.
I returned decisions, corrections, or new requests.
The work continued.

Numbered items made it easy to say “prioritize item 2” or “resume item 3.” There was a catch: once completed items disappeared and the list was renumbered, a number was no longer a stable task identifier. It needed to be checked against the task name and recent context.

I asked for completed items to be reported once and then removed. That kept the next report focused on open work and decisions.

Thirty minutes is not a universal recommendation. It happened to fit my workflow. It reduced the effort of checking in without requiring me to watch each tool continuously.

Delegating work across Claude Code, Antigravity, and ChatGPT also became more practical. The amount of coordination I could hand off still depended on access and approval in each environment.

Day 6: More kinds of work meant more invisible state

The scope expanded beyond coding and documentation: website onboarding, profile updates, and short video assets. Having an agent inspect a page and work through form fields removed a lot of repetitive effort.

The difficult part was understanding stalled background work.

I could follow a task in a visible chat or browser window. CLI processes and background activity were harder to inspect. If two tasks both said they were progressing but neither returned a result, I did not know which was running, which was waiting, or whether they were interfering with each other.

My hypothesis was that some tasks were competing for shared browser sessions or execution resources. I did not establish that as the root cause of every stall. What I did observe was inadequate visibility and a need for manual intervention.

For the next iteration, I want each active task to expose:

Its current owner and execution environment
Its last confirmed progress and timestamp
Whether it is running or waiting
The specific input, permission, or resource it needs next

This is a design goal rather than a dashboard I can claim is already complete. Shared desktop control needs particular care: parallel research can be useful, while simultaneous edits in the same interface can conflict.

Day 7: An urgent request changed the queue

The final day brought an interrupt: I wanted to prioritize preparation for Anthropic's CVP application. That involved reviewing my existing work, developing security-validation evidence, and contributing to OSS.

I explicitly assigned Astra to the urgent work. Progress felt fast. It would be misleading, however, to attribute the improvement solely to the model. Priority, task definition, and the surrounding workflow had changed too; I did not run a controlled comparison.

One concrete output was a validation report on an existing Mattermost pull request. I posted a comment with the comparison results. This was a testing contribution, not an upstream merge of my code.

In parallel, we prepared a read-only connection between Mattermost and Docchee. A short tunnel startup-and-shutdown trial had been performed, but reading an actual post through that path had not been verified at the time of writing. The integration was still unfinished.

The CVP application was also still in preparation. Useful portfolio work does not automatically establish the eligibility evidence an application asks for. I want the application to describe what I have actually done, without turning planned work or ordinary OSS activity into unsupported credentials.

What Day 7 demonstrated for me was the ability to redirect a development workflow under time pressure. Production incident response remains a separate capability to test.

The operating habits I want to keep

Give the task an observable finish line

I find this handoff format useful:

Goal: What should become possible?
Scope: What can change?
Out of scope: What should be left alone?
Environment: Where will the work run and be tested?
Acceptance: What observation will establish success?
Report: Deliverable, evidence, unverified paths, decisions needed

It reduces ambiguity before the team starts. It also makes it easier to assess a result without rereading an entire conversation.

Separate implementation from review

I use a separate reviewer for changes, including failure paths such as timeouts, cancellation, restart, and cleanup. Findings should come back with evidence or reproduction steps so the implementation agent can act on them.

A second model's agreement is not sufficient evidence by itself. The useful part is the additional inspection and testing it performs.

Report the actual stage of completion

Implemented, passed a mock test, worked in the target environment, and accepted by the user are different milestones.

The same applies to browser work: uploading a file, saving a draft, and publishing a page are separate outcomes. I need the status report to tell me which one happened.

This became especially important as the team handled more kinds of tasks. A broad “done” could hide the exact step I still needed to verify.

End the day with a short voice debrief

The most useful routine was a ten-minute conversation at the end of the day.

I talked through where I had waited, what had become easier, and what I wanted to try next. The visibility problem on Day 6 and the incident-response idea on Day 7 both became clearer through those conversations.

Short instructions helped me move work during the day. A longer conversation helped me reflect on the workflow itself. That combination was a practical form of human-in-the-loop operation for me.

Next: Test emergency workflows in a lab

My next question comes from my cloud-engineering background: how would this team respond to a server failure, a security incident, or a disaster-recovery scenario?

I want to survey existing SRE-agent approaches and design experiments across AWS, Azure, and Google Cloud. Those experiments have not started.

The plan is to use controlled environments with defined permissions and recovery conditions. I want to measure time to initial assessment and recovery, but also incorrect actions, unnecessary changes, approval delays, and missing evidence.

Parallel investigation could help. Changes to the same resource need coordination, and recovery needs a way back if the chosen action makes things worse. These are things to verify before making claims about autonomous incident handling.

What changed after one week

I started by counting tasks. By the end of the week, I was paying more attention to the handoffs between them.

As individual tasks moved faster, the surrounding constraints became easier to see: approvals, shared resources, usage allowances, and acceptance criteria. My own decisions remained part of that system.

The 30-minute check-ins and the end-of-day voice debrief are worth keeping. Next, I want better visibility into stalled work and stronger evidence about how the team behaves when something fails.

Seven days is a short experiment. It was enough to give me a much clearer idea of what to test in week two.

This article reflects my personal projects as of October 7, 2026. I used AI assistance to organize and edit the writing.

Can AI Replace Cloud Engineers? I Put It to the Test

SeaOtter ( Kentaro Mori ) — Tue, 06 Oct 2026 10:16:42 +0000

The result first

I tested how much of my cloud engineering work I could delegate to AI in two experiments.

Level 2: Implement and operate a small environment after a human defines the architecture and safety boundaries.
Level 3: Turn customer requirements into a cloud design that satisfies explicit acceptance criteria.

The live Level 2 experiments completed deployment, selected failure tests, recovery, and cleanup across AWS, Azure, and Google Cloud. The latest Level 3 trial, D5, scored 77/100 in an independent AI review: FAIL against the acceptance criteria, HOLD for adoption.

The distinction matters throughout this article. Level 2 has live execution records. Level 3 evaluates design documents. Neither experiment measures the percentage of a profession that can be replaced.

“Level” is my own classification, not an industry standard or an official mapping to cloud certifications. I did not run the introductory Level 1 stage.

Level 2: Delegate a defined infrastructure lifecycle

What the human specified

The common target was a Layer 7 load balancer serving HTTP through two private VMs. The VMs had no public IP addresses and no direct SSH access from the internet.

Component	AWS	Azure	Google Cloud
Public entry point	Regional ALB	Application Gateway	Global external Application Load Balancer
Backend	Two EC2 instances in separate AZs	Two Linux VMs in separate zones	Two VMs in a regional managed instance group
Outbound path	S3 Gateway Endpoint	NAT Gateway	Cloud Router and Cloud NAT
Remote state	S3	Blob Storage	Cloud Storage

These were implementations of common requirements, not identical architectures. Google Cloud used a managed instance group, while AWS and Azure used individual VMs. A global load balancer did not make the backend multi-region.

The human retained responsibility for requirements, allowed changes, identity and exposure boundaries, and approvals. AI handled Terraform, GitHub Actions, short-lived federated authentication, remote state, monitoring, selected failure tests, recovery, and cleanup.

What counted as completion

I looked beyond a successful terraform apply:

HTTP 200 through the load balancer and healthy backends
Successful short-lived CI authentication and remote-state initialization
No changes after migration, repair, and recovery
Observed alert-state changes for selected failures
A delete-only plan, empty state, and checks for remaining managed resources

The execution records report completion across all three clouds. Cleanup removed 139 Terraform-managed items: AWS 48, Azure 53, and Google Cloud 38. These totals include the application infrastructure and bootstrap resources for state and authentication.

That is not a count of VMs or individually billable resources. “Zero remaining” refers to the managed active resources checked in this experiment. It does not mean an empty account, physical erasure of every retained copy, or a guarantee of no later charges. See the Level 2 final report for the evaluation method and counts.

Three failures made the boundaries concrete

AWS: the instance was private, but its startup dependencies were unreachable.

Insufficient outbound connectivity prevented Nginx installation, and the ALB returned 502. AI diagnosed the problem and added narrowly scoped HTTPS access to the S3 prefix list. The lesson was to review startup dependencies alongside inbound isolation. This S3-based path depended on the selected OS package source; it was not general internet access.

Azure: a successful apply removed tags from 12 resources.

Local and CI inputs differed. The CI apply completed successfully while removing additional tags. The inputs were aligned, and a tag-only repair followed human approval. The exit status alone did not establish that the change matched the intended result.

Google Cloud: missing backend configuration made 18 existing resources look new.

The backend declaration was not tracked in Git. CI could not use the existing remote state and planned 18 additions. The process stopped before apply and repaired the declaration. Unexpected creation, deletion, or replacement needs to be a stop condition, not just another line in the output.

These incidents and their recovery are documented in the failure analysis.

What Level 2 did not establish

Monitoring checks covered selected signals and alert states. External notification receivers were not configured, so delivery to an operator was not demonstrated. The failure scenarios also differed between providers.

The experiments ran sequentially: AWS, then Azure, then Google Cloud, carrying lessons forward. Human-intervention counts used different units. That rules out a clean provider ranking or a defensible intervention-reduction percentage.

There was also no standardized human-only baseline for time, cost, or quality. The final report identifies code and CI controls that need checking before reuse; successful historical execution is not proof that the current checkout reproduces unchanged.

The supported conclusion is narrower: an experienced engineer could delegate a substantial implementation and operations lifecycle in small, disposable environments while retaining design, review, and approval. Production readiness, long-term SLOs, and unattended operation remain outside that result.

Level 3: Can AI produce an acceptable design from requirements?

The second experiment started with a member-facing application: registration, login, profile changes, list/detail views, favorites, and administrative updates.

Its main constraints were:

Initially 10,000 members and one million monthly page views
Dynamic traffic of 10 requests/second normally and 100 requests/second for a 15-minute peak, with a 9:1 read/write ratio; static requests counted separately; 500 concurrent users
Primary API p95 at or below 500 ms and server-side errors below 1%
99.9% monthly availability, including planned downtime and underlying platform failures
RTO of 30 minutes from service-impact onset, including detection and five stable minutes after recovery; RPO of five minutes
Production data and backups stored in Japan; departing members' data removed from active systems within 30 days, backups retained for no more than 35 days, and deletion reapplied after restoration
Five developers and one infrastructure engineer, supporting weekday daytime operations with no guaranteed immediate response at night
JPY 100,000/month including tax for production and minimal development/test environments, including mandatory networking, monitoring, logs, and backups; labor and other exclusions listed separately

The RTO/RPO targets covered process, runtime, database, and single-AZ failures. Region-wide failure was excluded. These were design requirements, not observed service metrics. The customer answers and requirements document preserve the input.

Budget compliance could not depend on free tiers, temporary credits, or long-term commitment discounts.

The tension between a 30-minute recovery target and weekday daytime staffing is a useful example: selecting managed services does not by itself establish who can act, which actions are authorized, or whether unattended recovery works.

Acceptance required more than a total score

The September work was self-assessed: A moved from 61 to 65, and B scored 66. October introduced a reviewer separate from the designer. D4 and D5 used anonymously reviewed, frozen design-and-evidence packages.

A passing design had to meet every condition:

Weighted total of at least 80/100
Every one of 11 categories rated at least 3/5
Six mandatory categories rated at least 4/5
No unresolved P0 items
All six adoption gates marked PASS

The mandatory categories were requirements, architecture, security/IAM, availability, cost, and backup/disaster recovery. No human scoring or adoption approval was obtained. These were independent AI document reviews, not production acceptance tests.

The latest result was 77, not the highest result of 78

Trial	Requested designer configuration	Independent score
D1	GPT-6 Luna / Low	53
D2	GPT-6 Luna / Medium	72
D3	GPT-6 Luna / High	69
D4	GPT-6.1 Sol / Medium	78
D5	GPT-6.1 Sol / High	77

These are requested settings. Actual runtime models and applied reasoning settings are UNKNOWN, including for the reviewers.

The series also carries earlier findings forward. D5 revised D4, added controls and evidence, and used a fresh reviewer. The scores cannot isolate a model or reasoning-effort effect. September self-scores are a different evaluation method and are not part of this series.

The latest authoritative trial is D5, dated October 2, 2026. It failed the acceptance criteria and remained on hold for adoption. The methods and results and D5 report explain the chronology and comparison limits.

Requirements changed separately on October 2 remain unscored and were not D5 input. The score of 77 does not apply to those changed requirements.

Why adoption stayed on hold

Cost was 3/5, below its mandatory minimum of 4. The design did not establish an all-inclusive budget fit with headroom, adequately supported effort for custom controls and rehearsals, or sufficiently concrete cheaper alternatives. Keeping unknown prices explicit was credited; unknowns did not become zero or evidence of affordability.

Deletion gate G3 remained HOLD. Deletion-trigger events, retention of linking information, independent custody of evidence, and responsibility for external services or recipient copies were not approved.

Operations gate G4 also remained HOLD. Named primary and backup owners, approved effort, absence coverage, and authority for unattended recovery were unresolved.

Added controls brought an evidence burden. D5 added a SQL-backed AdmissionCoordinator to principal request paths. Evidence for operation counts, contention, latency, and recovery ordering was insufficient. Complexity fell from 4 to 3, accounting for the one-point total decrease from D4. Complexity 3 meets the general category floor; it is distinct from Cost failing a mandatory minimum.

Unresolved P0 items concerned owner decisions, source access, and prerequisites for live validation, rather than identified P0 desk-design defects. The D5 report and gate record keep these distinctions explicit.

What I would carry into real work

Level 2 demonstrated more than initial code generation. AI investigated provider constraints, authentication mismatches, partial applies, and state inconsistencies, then made scoped repairs and completed cleanup.

Level 3 showed increasingly specific design documents, but no accepted design. Missing prices, owner decisions, and empirical evidence remained missing. A score near 80 does not establish that acceptance is close when mandatory conditions remain unresolved.

My practical takeaway is to specify the verification and decision process alongside the delegated task:

Define completion beyond command success: service behavior, state consistency, recovery, and cleanup
Stop unexpected changes before apply
Separate proposed designs, execution records, measured behavior, and approved decisions
Name the people and permissions behind recovery, operation, and deletion

A useful next experiment would hold inputs, evidence, and evaluation conditions constant while verifying actual model settings. Live performance, recovery, and cost validation would be a separate step. Those are research proposals, not work already completed.

For now, the evidence supports broad delegation of predefined implementation under human control. It does not establish replacement of cloud engineering as a whole, or that a stronger model alone would satisfy the remaining adoption conditions.

Sources and AI disclosure

Source review date: October 6, 2026. References are pinned to Level 2 commit c5d25b6 and Level 3 commit e1ad7ab. The existing Japanese and English slides were also cross-checked. Preparing this article did not rerun cloud operations or produce a new assessment.

AI assisted the structure, drafting, and English wording of this article. The experiment conditions, results, and limitations are based on the linked records.