I have a habit with any framework that claims “durable” or “resilient” in its release notes. I find the process ID and kill it. Not a graceful shutdown, not Ctrl+C. kill -9, the signal nothing gets to clean up after.
Dave R published a crash test of the latest Microsoft Agent Framework update that did exactly this, and it was clean enough that I wanted to reproduce it and then break my own version of it. This post is that exercise: what his test shows, what Microsoft says it does and does not guarantee, a small .NET workflow of my own running against a local Ollama model with a file checkpoint store and a SQLite journal, and a record of which steps ran twice.
Short version before the details: the workflow survives. Your side effects do not get exactly-once delivery for free, and the framework says so plainly.
The test I was trying to reproduce
Dave’s setup, as he describes it in his Towards AI piece: start a countdown workflow that should emit 20 down to 0, kill the host with kill -9 partway through, restart it, reconnect. The first process got 20 through 13 out before it died. The restarted host replayed those eight outputs to the reconnecting client, sent a snapshot, and carried on from 12. Twenty-one outputs arrived in order, with no duplicates, and the whole thing ran locally with no Azure subscription.
That is his run, on his machine. I am repeating it as he reported it, not as something I measured.
The thing that makes it work is the resilient background mode on the Responses host. In .NET it is one option on AddFoundryResponses:
using Microsoft.Agents.AI;
using Microsoft.Agents.AI.Foundry.Hosting;
using Microsoft.Agents.AI.Workflows;
AIAgent agent = workflow.AsAIAgent(
id: "report-workflow",
name: "report-workflow",
includeWorkflowOutputsInResponse: true);
var builder = WebApplication.CreateBuilder(args);
builder.Services.AddFoundryResponses(
agent,
configure: options => options.ResilientBackground = true);
var app = builder.Build();
app.MapFoundryResponses();
await app.RunAsync();
On restart, the host reloads persisted state and picks the workflow checkpoint tied to the saved response. Microsoft’s post is explicit that the restarted process has to rebuild the workflow with the same executor identities. Change an executor’s ID or an agent’s name between deploys and the checkpoint no longer matches.
And then the sentence that I think matters more than the demo. From the September update on the Agent Framework blog: “Recovery does not imply exactly-once execution of external effects.” An interrupted step may run again. If that step sent an email or charged a card, it can do it twice, and the advice is to make downstream operations tolerate retries, for example with idempotency keys.
“None was repeated” in Dave’s test is about the output stream reaching the client. It is not a statement about what the steps did to the outside world. I wanted to see that gap with my own eyes.
What I built instead
I did not want to depend on a Foundry project for this. The hosted Responses path is the polished one, but the mechanism underneath is plain workflow checkpointing, and that runs in-process with nothing but NuGet packages.
My workflow is a countdown with teeth:
- A planner executor receives a number. If it is zero, it yields "liftoff". Otherwise it sends a work item to the worker.
- A worker executor asks a local model (Ollama, llama3.2) for a one-line checklist item, then performs a side effect: it records a "notification" in an outbox table keyed by runId:step:N. Then it yields the line and sends the next number back to the planner.
Two executors in a loop, so each number takes two supersteps. Microsoft’s docs say checkpoints are written at the end of every superstep, after all executors in that step finish. That detail decides everything that follows.
Around the workflow I added three things the framework does not give me:
- A journal in SQLite. Every executor entry and every side effect gets a row with the process ID. This is how I see what replayed, because in-process resume does not replay the output stream the way the hosted mode does.
- An outbox with idempotency keys. INSERT OR IGNORE on a primary key. A replayed step that tries the same effect gets zero rows affected and logs "skipped".
- A token budget ledger that reserves before each model call, settles after, and halts admissions when a call returns with no usage report.
Checkpoints go to disk through FileSystemJsonCheckpointStore. I considered writing a SQLite ICheckpointStore so everything lived in one file, and decided against it for this post. The file store is the one people will actually start with, and I wanted to test the shipped path, not my own storage code.
Setup
You need the .NET 10 SDK and Ollama.
# Ollama installed natively
ollama pull llama3.2
ollama serve
Or in Docker, if you would rather not install it:
docker run -d --name ollama -p 11434:11434 -v ollama:/root/.ollama ollama/ollama
docker exec -it ollama ollama pull llama3.2
I picked llama3.2 partly on Davide Bellone's advice in his Code4IT series on Agent Framework: not every Ollama model supports tool calling (he hit this with phi3:mini), and I wanted a model I could reuse when I add tools later. This workflow does not use tools yet.
Create the project:
dotnet new console -n DurableCountdown -f net10.0
cd DurableCountdown
dotnet add package Microsoft.Agents.AI.Workflows --prerelease
dotnet add package Microsoft.Extensions.AI
dotnet add package OllamaSharp
dotnet add package Microsoft.Data.Sqlite
A warning I learned the hard way across previews: pin the versions you test with. The checkpoint API names have moved (run IDs became session IDs, StreamAsync became RunStreamingAsync), and the Learn page you read may be a version ahead or behind your package. The code below follows the Learn checkpoint docs as of August 2026.
Program.cs
The whole thing is one file. Replace Program.cs with this:
using System.Diagnostics;
using Microsoft.Agents.AI.Workflows;
using Microsoft.Agents.AI.Workflows.Checkpointing;
using Microsoft.Data.Sqlite;
using Microsoft.Extensions.AI;
using OllamaSharp;
// Usage:
// dotnet run -- run <runId> [start] start a new run (default start = 20)
// dotnet run -- resume <runId> resume from the last checkpoint on disk
// dotnet run -- report <runId> print the journal and the replay table
// dotnet run -- clear-halt <runId> a human decides the ledger may continue
//
// Env:
// OLLAMA_URL (default http://localhost:11434), OLLAMA_MODEL (default llama3.2)
// CRASH_AT_STEP=12 SIGKILL this process right after step 12's side effect (run mode only)
// DEADLINE_SECONDS=30 local deadline for each model call
// MISSING_USAGE_AT_STEP=5 pretend the provider returned no usage for step 5
// TOKEN_BUDGET=40000
string mode = args.ElementAtOrDefault(0) ?? "run";
string runId = args.ElementAtOrDefault(1) ?? "demo";
int start = int.Parse(args.ElementAtOrDefault(2) ?? "20");
using var db = new Store("durable.db");
if (mode == "report") { db.PrintReport(runId); return; }
if (mode == "clear-halt") { db.DeleteMeta(runId, "halted"); Console.WriteLine("halt cleared"); return; }
Crash.Armed = mode == "run";
string ollamaUrl = Environment.GetEnvironmentVariable("OLLAMA_URL") ?? "http://localhost:11434";
string model = Environment.GetEnvironmentVariable("OLLAMA_MODEL") ?? "llama3.2";
long budget = long.Parse(Environment.GetEnvironmentVariable("TOKEN_BUDGET") ?? "40000");
IChatClient chat = new OllamaApiClient(new Uri(ollamaUrl), model);
var ledger = new TokenLedger(db, runId, budget);
// Executor IDs are fixed strings. A resumed process must rebuild the same topology
// with the same identities, or the checkpoint will not match.
var planner = new PlannerExecutor(runId, db);
var worker = new WorkerExecutor(runId, db, chat, ledger);
Workflow workflow = new WorkflowBuilder(planner)
.AddEdge(planner, worker)
.AddEdge(worker, planner)
.WithOutputFrom(planner, worker)
.Build();
// One checkpoint directory per run keeps runs from seeing each other's index.
var dir = new DirectoryInfo(Path.Combine("checkpoints", runId));
dir.Create();
using var store = new FileSystemJsonCheckpointStore(dir);
CheckpointManager checkpoints = CheckpointManager.CreateJson(store);
db.ReportOrphanedHolds(runId);
StreamingRun run;
if (mode == "run")
{
db.Journal(runId, "host", start, "start");
run = await InProcessExecution.RunStreamingAsync(workflow, new Tick(start), checkpoints);
}
else if (mode == "resume")
{
string sessionId = db.GetMeta(runId, "session")
?? throw new InvalidOperationException($"no checkpointed session recorded for {runId}");
List<CheckpointInfo> index = [.. await store.RetrieveIndexAsync(sessionId)];
if (index.Count == 0) throw new InvalidOperationException("checkpoint index is empty");
CheckpointInfo last = index[^1];
db.Journal(runId, "host", 0, $"resume from checkpoint {last.CheckpointId} ({index.Count} on disk)");
run = await InProcessExecution.ResumeStreamingAsync(workflow, last, checkpoints);
}
else
{
Console.WriteLine($"unknown mode {mode}");
return;
}
await using (run)
{
await foreach (WorkflowEvent evt in run.WatchStreamAsync())
{
switch (evt)
{
case SuperStepCompletedEvent step when step.CompletionInfo?.Checkpoint is { } cp:
db.SetMeta(runId, "session", cp.SessionId);
db.SetMeta(runId, "last_checkpoint", cp.CheckpointId);
break;
case WorkflowOutputEvent output:
Console.WriteLine($"[pid {Environment.ProcessId}] OUTPUT {output.Data}");
break;
case ExecutorFailedEvent failed:
Console.WriteLine($"[pid {Environment.ProcessId}] EXECUTOR FAILED {failed.ExecutorId}: {failed.Data}");
break;
case WorkflowErrorEvent error:
Console.WriteLine($"[pid {Environment.ProcessId}] WORKFLOW ERROR: {error.Data}");
break;
}
}
}
db.Journal(runId, "host", 0, "stream ended");
// ---------------------------------------------------------------------------
// Messages
public sealed record Tick(int Remaining);
public sealed record WorkItem(int Step);
// ---------------------------------------------------------------------------
// Executors
sealed class PlannerExecutor(string runId, Store db) : Executor<Tick>("planner")
{
public override async ValueTask HandleAsync(Tick tick, IWorkflowContext context, CancellationToken cancellationToken = default)
{
db.Journal(runId, "planner", tick.Remaining, "enter");
if (tick.Remaining == 0)
{
await context.YieldOutputAsync("0: liftoff", cancellationToken);
return;
}
await context.SendMessageAsync(new WorkItem(tick.Remaining), cancellationToken: cancellationToken);
}
}
sealed class WorkerExecutor(string runId, Store db, IChatClient chat, TokenLedger ledger) : Executor<WorkItem>("worker")
{
public override async ValueTask HandleAsync(WorkItem item, IWorkflowContext context, CancellationToken cancellationToken = default)
{
int step = item.Step;
db.Journal(runId, "worker", step, "enter");
// 1. Model call: reserved in the ledger, bounded by a local deadline.
string line = await DescribeStepAsync(step, cancellationToken);
// 2. Side effect, guarded by an idempotency key derived from the work, not from the attempt.
string key = $"{runId}:step:{step}";
bool applied = db.TryRecordEffect(key, line);
db.Journal(runId, "worker", step, applied ? "effect applied" : "effect skipped (duplicate key)");
// 3. The dangerous window: effect done, superstep not yet checkpointed.
Crash.MaybeKill(step);
await context.YieldOutputAsync($"{step}: {line}", cancellationToken);
await context.SendMessageAsync(new Tick(step - 1), cancellationToken: cancellationToken);
}
async Task<string> DescribeStepAsync(int step, CancellationToken callerToken)
{
const long estimate = 400;
string hold = ledger.Reserve($"step-{step}", estimate);
int seconds = int.Parse(Environment.GetEnvironmentVariable("DEADLINE_SECONDS") ?? "30");
using var deadline = new CancellationTokenSource(TimeSpan.FromSeconds(seconds));
using var linked = CancellationTokenSource.CreateLinkedTokenSource(callerToken, deadline.Token);
try
{
ChatResponse response = await chat.GetResponseAsync(
$"In one short sentence, write launch checklist item number {step}. No preamble, no quotes.",
cancellationToken: linked.Token);
UsageDetails? usage = response.Usage;
if (Environment.GetEnvironmentVariable("MISSING_USAGE_AT_STEP") == step.ToString())
usage = null; // simulate a provider or adapter that drops the usage block
ledger.Settle(hold, usage);
return response.Text.Trim();
}
catch (OperationCanceledException) when (callerToken.IsCancellationRequested)
{
// The caller gave up. We still do not know whether the model finished.
ledger.MarkUnknown(hold, "caller cancelled");
throw;
}
catch (OperationCanceledException) when (deadline.IsCancellationRequested)
{
// We stopped waiting. That is not evidence the inference stopped.
ledger.MarkUnknown(hold, "local deadline exceeded");
throw new OutcomeUnknownException($"step {step}: model call outcome unknown after {seconds}s");
}
}
}
// ---------------------------------------------------------------------------
// Crash injection
static class Crash
{
public static bool Armed { get; set; }
public static void MaybeKill(int step)
{
if (!Armed) return;
if (Environment.GetEnvironmentVariable("CRASH_AT_STEP") != step.ToString()) return;
Console.WriteLine($"[pid {Environment.ProcessId}] SIGKILL at step {step}, after the effect, before the checkpoint");
Process.GetCurrentProcess().Kill(); // SIGKILL on Linux and macOS, no finally blocks, no flush
}
}
// ---------------------------------------------------------------------------
// Token budget ledger: unknown usage is a reason to stop, not zero.
sealed class BudgetHaltedException(string message) : Exception(message);
sealed class OutcomeUnknownException(string message) : Exception(message);
sealed class TokenLedger(Store db, string runId, long budget)
{
public string Reserve(string label, long estimate)
{
if (db.GetMeta(runId, "halted") is { } why)
throw new BudgetHaltedException($"admissions halted earlier: {why}");
long committed = db.LedgerCommitted(runId);
if (committed + estimate > budget)
throw new BudgetHaltedException($"budget {budget} would be exceeded: {committed} committed, {estimate} requested");
string holdId = $"{label}:{Guid.NewGuid():N}";
db.Exec("INSERT INTO ledger(hold_id, run_id, label, reserved, actual, status, pid, at) VALUES($h,$r,$l,$e,NULL,'held',$p,$t)",
("$h", holdId), ("$r", runId), ("$l", label), ("$e", estimate),
("$p", Environment.ProcessId), ("$t", DateTime.UtcNow.ToString("O")));
return holdId;
}
public void Settle(string holdId, UsageDetails? usage)
{
long? total = usage?.TotalTokenCount
?? (usage?.InputTokenCount is long i && usage?.OutputTokenCount is long o ? i + o : null);
if (total is null)
{
// A call may have happened and we cannot say what it cost.
// Keep the hold, halt new admissions, and make a human look.
db.Exec("UPDATE ledger SET status='unknown' WHERE hold_id=$h", ("$h", holdId));
db.SetMeta(runId, "halted", $"no usage reported for {holdId}");
throw new BudgetHaltedException($"no usage reported for {holdId}; hold kept, admissions halted");
}
db.Exec("UPDATE ledger SET status='settled', actual=$a WHERE hold_id=$h", ("$a", total.Value), ("$h", holdId));
}
public void MarkUnknown(string holdId, string reason) =>
db.Exec("UPDATE ledger SET status='unknown', label=label || ' (' || $why || ')' WHERE hold_id=$h",
("$why", reason), ("$h", holdId));
}
// ---------------------------------------------------------------------------
// SQLite store: journal, outbox, ledger, meta
sealed class Store : IDisposable
{
readonly SqliteConnection _c;
readonly Lock _gate = new();
public Store(string path)
{
_c = new SqliteConnection($"Data Source={path}");
_c.Open();
Exec("""
PRAGMA journal_mode=WAL;
PRAGMA synchronous=FULL;
CREATE TABLE IF NOT EXISTS journal(
id INTEGER PRIMARY KEY AUTOINCREMENT,
run_id TEXT, pid INTEGER, at TEXT, executor TEXT, step INTEGER, phase TEXT);
CREATE TABLE IF NOT EXISTS outbox(
idem_key TEXT PRIMARY KEY, payload TEXT, pid INTEGER, at TEXT);
CREATE TABLE IF NOT EXISTS ledger(
hold_id TEXT PRIMARY KEY, run_id TEXT, label TEXT, reserved INTEGER,
actual INTEGER, status TEXT, pid INTEGER, at TEXT);
CREATE TABLE IF NOT EXISTS meta(
run_id TEXT, k TEXT, v TEXT, PRIMARY KEY(run_id, k));
""");
}
public int Exec(string sql, params (string Name, object? Value)[] ps)
{
lock (_gate)
{
using var cmd = _c.CreateCommand();
cmd.CommandText = sql;
foreach (var (name, value) in ps) cmd.Parameters.AddWithValue(name, value ?? DBNull.Value);
return cmd.ExecuteNonQuery();
}
}
object? Scalar(string sql, params (string Name, object? Value)[] ps)
{
lock (_gate)
{
using var cmd = _c.CreateCommand();
cmd.CommandText = sql;
foreach (var (name, value) in ps) cmd.Parameters.AddWithValue(name, value ?? DBNull.Value);
return cmd.ExecuteScalar();
}
}
public void Journal(string runId, string executor, int step, string phase)
{
Exec("INSERT INTO journal(run_id, pid, at, executor, step, phase) VALUES($r,$p,$t,$e,$s,$ph)",
("$r", runId), ("$p", Environment.ProcessId), ("$t", DateTime.UtcNow.ToString("O")),
("$e", executor), ("$s", step), ("$ph", phase));
Console.WriteLine($"[pid {Environment.ProcessId}] {executor,-8} step {step,2} {phase}");
}
// Returns true if this call applied the effect, false if the key was already there.
public bool TryRecordEffect(string key, string payload) =>
Exec("INSERT OR IGNORE INTO outbox(idem_key, payload, pid, at) VALUES($k,$v,$p,$t)",
("$k", key), ("$v", payload), ("$p", Environment.ProcessId), ("$t", DateTime.UtcNow.ToString("O"))) == 1;
public string? GetMeta(string runId, string key) =>
Scalar("SELECT v FROM meta WHERE run_id=$r AND k=$k", ("$r", runId), ("$k", key)) as string;
public void SetMeta(string runId, string key, string value) =>
Exec("INSERT INTO meta(run_id, k, v) VALUES($r,$k,$v) ON CONFLICT(run_id, k) DO UPDATE SET v=excluded.v",
("$r", runId), ("$k", key), ("$v", value));
public void DeleteMeta(string runId, string key) =>
Exec("DELETE FROM meta WHERE run_id=$r AND k=$k", ("$r", runId), ("$k", key));
// Settled calls count at their real usage. Held and unknown calls count at their reservation.
public long LedgerCommitted(string runId) =>
Convert.ToInt64(Scalar(
"SELECT COALESCE(SUM(CASE WHEN status='settled' THEN actual ELSE reserved END), 0) FROM ledger WHERE run_id=$r",
("$r", runId)));
public void ReportOrphanedHolds(string runId)
{
lock (_gate)
{
using var cmd = _c.CreateCommand();
cmd.CommandText = "SELECT label, reserved, pid FROM ledger WHERE run_id=$r AND status='held' AND pid<>$p";
cmd.Parameters.AddWithValue("$r", runId);
cmd.Parameters.AddWithValue("$p", Environment.ProcessId);
using var rd = cmd.ExecuteReader();
while (rd.Read())
Console.WriteLine($"[ledger] hold {rd.GetString(0)} ({rd.GetInt64(1)} tokens) from dead pid {rd.GetInt64(2)} stays counted");
}
}
public void PrintReport(string runId)
{
lock (_gate)
{
Console.WriteLine("REPLAYED STEPS (entered by more than one process)");
Console.WriteLine("executor step processes");
Console.WriteLine("-------- ---- ---------");
using (var cmd = _c.CreateCommand())
{
cmd.CommandText = """
SELECT executor, step, GROUP_CONCAT(DISTINCT pid)
FROM journal
WHERE run_id=$r AND phase='enter'
GROUP BY executor, step
HAVING COUNT(DISTINCT pid) > 1
ORDER BY step DESC
""";
cmd.Parameters.AddWithValue("$r", runId);
using var rd = cmd.ExecuteReader();
while (rd.Read())
Console.WriteLine($"{rd.GetString(0),-8} {rd.GetInt64(1),4} {rd.GetString(2)}");
}
Console.WriteLine();
Console.WriteLine("EFFECTS");
using (var cmd = _c.CreateCommand())
{
cmd.CommandText = """
SELECT
(SELECT COUNT(*) FROM outbox WHERE idem_key LIKE $p),
(SELECT COUNT(*) FROM journal WHERE run_id=$r AND phase='effect applied'),
(SELECT COUNT(*) FROM journal WHERE run_id=$r AND phase LIKE 'effect skipped%')
""";
cmd.Parameters.AddWithValue("$p", runId + ":%");
cmd.Parameters.AddWithValue("$r", runId);
using var rd = cmd.ExecuteReader();
rd.Read();
Console.WriteLine($"outbox rows: {rd.GetInt64(0)}, applied: {rd.GetInt64(1)}, skipped as duplicates: {rd.GetInt64(2)}");
}
Console.WriteLine();
Console.WriteLine("LEDGER");
Console.WriteLine("status calls tokens");
Console.WriteLine("-------- ----- ------");
using (var cmd = _c.CreateCommand())
{
cmd.CommandText = """
SELECT status, COUNT(*), COALESCE(SUM(CASE WHEN status='settled' THEN actual ELSE reserved END), 0)
FROM ledger WHERE run_id=$r GROUP BY status
""";
cmd.Parameters.AddWithValue("$r", runId);
using var rd = cmd.ExecuteReader();
while (rd.Read())
Console.WriteLine($"{rd.GetString(0),-8} {rd.GetInt64(1),5} {rd.GetInt64(2),6}");
}
}
}
public void Dispose() => _c.Dispose();
}
A few choices I went back and forth on.
The idempotency key is runId:step:N, derived from the work. My first version included a GUID per attempt, which is the classic mistake: every retry gets a fresh key and the guard never fires.
The journal and the outbox live in the same SQLite file but are written in separate statements. That is fine for this demo because the effect itself is the outbox insert. Mehdi Mohseni makes the stronger point in part 5 of his agent bus series: put the idempotency check and the side effect in one transaction. That gives you exactly-once only when the effect is inside your database. An email, a webhook, a payment API is outside it, and then the key has to travel to the downstream system and be honored there.
The crash is injected with Process.GetCurrentProcess().Kill(), which sends SIGKILL on Linux and macOS. I also ran the manual version with pkill -9 -f DurableCountdown from a second terminal while the run was in flight. The injected one is just repeatable.
Running it
# 1. Start a run and kill it right after step 12's side effect
CRASH_AT_STEP=12 dotnet run -- run launch-1 20
# 2. Start a fresh process and resume from the last checkpoint on disk
dotnet run -- resume launch-1
# 3. Ask the journal what happened
dotnet run -- report launch-1
The first process counts 20, 19, 18 and so on. On step 12 the worker calls the model, writes the outbox row, and dies before it can yield the output or send the next tick. The console shows outputs 20 through 13, then the SIGKILL line, then nothing. No shutdown logging, which is the point.
The resumed process prints the checkpoint it picked and goes straight to worker step 12 enter. It does not start over at 20. The model is called again for 12, the outbox insert is ignored, and the journal records "effect skipped (duplicate key)". Then 11 down to 1, and the planner yields "liftoff".
Here is the timeline, superstep by superstep:
superstep: S1 S2 S3 ... S17 S18
executor: P20 W20 P19 ... P12 W12
process A (pid) ----------->[cp] [cp] [cp]... [cp] model call, outbox row,
| X SIGKILL (no cp for S18)
|
last good checkpoint ----+
|
process B (pid) +-> W12 again: model call #2,
outbox insert ignored
-> P11 -> W11 -> ... -> W1 -> P0 "liftoff"
legend: P = planner, W = worker, cp = checkpoint written at end of superstep
outputs seen by the console: 20..13 from A, 12..1 and "0: liftoff" from B
The report makes it concrete:
REPLAYED STEPS (entered by more than one process)
executor step processes
-------- ---- ---------
worker 12 [pid A],[pid B]
EFFECTS
outbox rows: 20, applied: 20, skipped as duplicates: 1
LEDGER
status calls tokens
-------- ----- ------
held 1 400
settled 20
One step replayed, and it was exactly the step in the window between “effect done” and “checkpoint written”. The planner for 12 did not replay, because its superstep finished and was checkpointed. The model call for 12 ran twice, so I paid for it twice. The orphaned hold from process A is still counted at its 400-token reservation, because nobody can tell me what that call cost. The ledger prints it on startup in process B, and it stays in the total.
Twenty effects, not twenty-one, because the idempotency key did its job. Delete the key and you get twenty-one outbox rows, which in a real system is a customer getting the same message twice. I tried that too, just to watch it happen. It does.
Side by side with Dave’s test:
Dave R (hosted, as he reports it) Mine (in-process, local)
----------------------- ---------------------------------- ---------------------------------
Host Responses host, resilient bg InProcessExecution + checkpoints
Model countdown sample, no model llama3.2 via Ollama
Checkpoint storage host-managed persisted state FileSystemJsonCheckpointStore
Killed after output 13 13 (inside step 12, after effect)
Resumed at 12 12
Client stream replay yes, 8 outputs plus a snapshot no, journal instead
Outputs total 21 21 (20..1 plus liftoff)
Steps that ran twice not reported worker 12
Effects duplicated not applicable 0 with key, 1 without
The difference in the “client stream replay” row is the one I did not appreciate until I built mine. The hosted mode gives the reconnecting client a consistent view of what already happened. In-process checkpointing restarts the computation, but anything the dead process already printed or streamed is gone unless you persisted it yourself. If a user-facing UI depends on the stream, that difference matters more than the checkpoint format.
Unknown outcome is not failure
The second thing I wanted to break was the model call itself.
Run with a deadline the model cannot meet, especially on a cold Ollama instance that still has to load weights:
DEADLINE_SECONDS=1 dotnet run -- run launch-2 5
The worker’s call gets cancelled by the local deadline and the step throws OutcomeUnknownException. Not a failure. Unknown.
That distinction came out of a discussion in my reading this week around a TOOL_OUTCOME_UNKNOWN status, and it is the most useful idea I took from it. When you stop waiting on a call, you have learned something about your patience, not about the call. The model may have finished. The webhook may have fired. The payment may have gone through. Some of that discussion leaned on claims about "Pi Durable" and a "DeepSeek Harness" from JIN's articles; I could not verify those, so I am not repeating them here and you should not take them from me.
Ranknod’s C# Corner piece on cancelling slow AI calls makes the same point with code I borrowed the shape of. Keep the caller’s token and your own deadline as separate sources, link them for the call, and classify by checking the original sources afterwards. A canceled wait is not evidence of a canceled inference. His wrapper also refuses to relabel an unexplained cancellation as a deadline, which I copied: if neither source fired, the exception just propagates.
What “unknown” means in practice in my workflow:
- The ledger marks the hold as unknown and keeps counting it. If the model did run, those tokens were spent.
- The workflow stops at that step. The last checkpoint is before it, so resume will retry it.
- For a side-effecting tool, the retry must reconcile first: look up the idempotency key downstream, and only re-send if it is absent. Treating unknown as failure and blindly retrying is how you double-charge. Treating it as success is how you silently lose work.
I do not have a downstream service in this demo that supports lookups by key, so the reconciliation step is the outbox check itself. In a real system that check is a call to the other side, and if the other side has no way to answer “did you get key X?”, you have a design problem no framework will fix.
The ledger that halts when usage is missing
The third experiment:
MISSING_USAGE_AT_STEP=5 dotnet run -- run launch-3 8
At step 5 I throw away the usage block from the response, which simulates an adapter or provider that does not report it. The ledger marks that hold unknown, records a halt reason in the meta table, and throws. Every later Reserve call refuses until a human runs:
dotnet run -- clear-halt launch-3
dotnet run -- resume launch-3
I took this design from another Ranknod article on token budgets: if a call may have happened and no usage comes back, keep the reservation held and stop admitting new work. It feels heavy-handed the first time it fires. But the alternative is a budget that silently treats unknown as zero, and a budget that can be fooled by a missing field is not a budget.
Two honest caveats. My ledger counts raw tokens, not cost, so it is crude. And with Ollama the token cost is zero dollars, which hides the real costs: my laptop’s time and the quality gap of a small local model. The ledger is there for the halting behavior, which is the same whether the tokens are free or not.
What is preview, and what I did not run
The September update covers more than durability, and a lot of it is still labeled preview. I kept a list so I would stop mixing up what I tested with what I read.
Item Status / limit Where I got it Did I run it?
--------------------------------- ---------------------------------------- ------------------------ -------------
Workflow checkpoint + resume shipping in Microsoft.Agents.AI.Workflows Learn docs yes, this post
Resilient background (Responses) .NET and Python samples, Foundry hosting Microsoft blog no
AG-UI hosting, Python stable release Microsoft blog no
AG-UI hosting, .NET public preview Microsoft blog no
(Microsoft.Agents.AI.Hosting.AGUI.AspNetCore)
Foundry memory preview; 100 scopes per store, Learn docs no
10,000 memories per scope
CodeAct with Hyperlight alpha (Python, April); .NET samples Microsoft blogs no
linked in September
Hyperlight inside hosted agents not available, needs a hypervisor that Dave R (paywalled part, no
hosted agents do not expose not verified by me)
A few notes on those rows.
The .NET AG-UI hosting package is preview, and Microsoft says outright that Python and .NET do not have feature parity. Use the language-specific samples rather than assuming a Python feature exists in .NET.
The Foundry memory limits are on the Learn page for memory in Foundry Agent Service: a maximum of 100 scopes per store and 10,000 memories per scope, with pricing that can change during preview. If your scope is per user, 100 is a ceiling you will hit very quickly, and Microsoft’s own guidance is to derive the scope from authenticated identity. So you are looking at many stores, or a different design.
Hyperlight runs model-generated code in micro-VMs, which means it needs hardware virtualization on the host. The April CodeAct post lists Linux and Windows support with macOS still coming. Dave reports that Foundry hosted agents do not expose the hypervisor Hyperlight needs, so CodeAct in that environment is not an option today. I could not read that section past the paywall to check his reasoning, so treat it as his finding. I am on a Mac, so I did not run CodeAct at all.
About that CodeAct number
The CodeAct figures travel a lot, so here is where they come from.
Microsoft’s April post benchmarks one workload: compute order totals for eight users, with five tools (list users, get orders, get discount rate, get tax rate, compute a line total), same model, same prompt, wired either as normal tools or behind a single execute_code tool. On the run they report, traditional wiring took 27.81 seconds and 6,890 tokens; CodeAct took 13.23 seconds and 2,489 tokens. That is about 52% lower latency and 64% fewer tokens.
That is a vendor-run benchmark, on one workload, in Python. The September post rounds it to roughly 50% and over 60%, and tells you to treat it as workload-specific and measure your own. I agree with them. The workload is close to the best case for CodeAct: lots of small, pure, chainable tool calls. The same post says to keep side-effecting tools as direct, approval-gated calls, and that weak tool descriptions hurt CodeAct more than normal tool calling. If your agent makes two tool calls per turn, there is not much loop to collapse.
The isolation note is worth repeating because it is easy to misread. Hyperlight isolates the code the model writes. Your registered tools still run in your own process with your own permissions. The sandbox does not make a dangerous tool safe.
What I am still unsure about
- Checkpoint granularity under real agents. My workflow has small supersteps. A workflow where one superstep contains a long agent turn with several tool calls will replay all of them on a crash, and only the side-effecting ones need keys. I have not measured how much that costs in practice.
- Index behavior across sessions. There is an open GitHub issue (microsoft/agent-framework #5942) reporting that FileSystemJsonCheckpointStore.RetrieveIndexAsync(sessionId) returns entries from every session in the index on 1.0.0-rc1. I sidestepped it with one directory per run. If you share a directory across runs, check your version before trusting the last entry.
- Whether the hosted mode replays more or less than mine. Dave’s test shows the stream side cleanly. It does not show, and I did not test, which executor steps the hosted mode re-runs after a crash. The documentation says an interrupted step may run again, so I assume the same window exists.
- Checkpoints as a trust boundary. The Learn page warns never to load checkpoints from untrusted sources. A directory of JSON files next to the app is fine on my laptop. In a container with a shared volume, it is an attack surface I have not thought through yet.
What I am taking away
The workflow survived kill -9 and resumed at the right step. That part works, and it is genuinely nicer than rolling my own state machine.
What the crash test does not show, and what Microsoft is honest about, is that “resumed at the right step” means “re-ran the step that was interrupted”. The step I lost was the one that had already done its side effect. Without the idempotency key that would have been a duplicate. Without the ledger rule, the doubled model call and the call with no usage would have quietly disappeared from my budget.
So my checklist for anything I call durable now has three lines: every side effect carries a key derived from the work, every timeout is recorded as unknown and reconciled before retry, and every missing usage report stops the budget instead of counting as zero. The framework handles the checkpoint. Those three are still mine.
Sources: Dave R, “An AI Agent Workflow That Survives kill -9: Inside Microsoft Agent Framework’s Latest Update” (Towards AI, member-only; crash test results are his). Dan Taylor, “What’s new in Microsoft Agent Framework: Interactive experiences, memory, and resilient execution” (Microsoft Agent Framework blog, 24 September 2026). Eduard van Valkenburg, “CodeAct in Agent Framework: Faster Agents with Fewer Model Turns” (Microsoft Agent Framework blog, April 2026; vendor-run benchmark). Microsoft Learn: Agent Framework workflow checkpoints, and Memory in Foundry Agent Service limits. Ranknod, “Cancel Slow AI Calls in .NET Without Hiding the Cause” (C# Corner). Mehdi Mohseni, “Building an Agentic System in .NET, Part 5: Designing an Agent Bus in ASP.NET Core” (Medium). Davide Bellone, “Microsoft Agent Framework for .NET (part 4)” (Code4IT). GitHub issue microsoft/agent-framework #5942.
Tags: dotnet, csharp, ai-agents, microsoft-agent-framework, durable-execution, idempotency, ollama, sqlite, distributed-systems
Top comments (0)