DEV Community: Peter Bednarčík

An EF Core race condition that only showed up after the switch to Azure

Peter Bednarčík — Thu, 15 Oct 2026 09:12:23 +0000

A few months ago, I faced one of those head-scratching problems. Something happened in production that shouldn't happen: a record got closed that nobody closed. Unfortunately, it happened on a critical path in the system, so we needed a fast fix.

Nothing in the error logs. No leads in other log levels either. No recent code change that could explain it. It got me thinking about a recent issue we had: code that had worked fine for years broke after we switched to Azure. My first suspect was a race condition between overlapping Hangfire jobs, but I disproved it: we turned off the second Hangfire server, and the issue persisted. I did notice it happened mostly during rush hours in production.

📝 Note: The real system isn't a warehouse. I changed the domain and names, but the mechanics are exactly the same. The full reproduction is on GitHub.

The setup

For this article, I moved the problem to a warehouse that picks orders in batches. One pick list, one walk through the warehouse, can serve several orders, so pick lists and orders are linked through a link table:

public class PickList
{
    public int Id { get; set; }
    public required string Status { get; set; }
    public List<PickListOrder> Orders { get; } = [];
}

public class PickListOrder // link row
{
    public int PickListId { get; set; }
    public int OrderId { get; set; }
}

When an order ships, the open pick lists that served only that order get closed. A pick list shared with another order stays open until that order ships too.

In our case, shipping order A closed order B's brand-new pick list.

Logging

For this project, a new deployment isn't as straightforward as pushing a button and waiting a few minutes, so we couldn't just try things in production. We needed proof before changing any logic.

Logging was the first thing that could help without breaking anything else. We added a log line before and after every place in the code that closes a pick list, including the pick list, the order it was closed for, and both statuses.

When it happened again, the logs narrowed it down to one place: the code that ships orders closed the pick list. So that was where I started looking.

The query

When an order ships, this query decides which pick lists get closed:

var pickLists = db.PickLists
    .Where(p => p.Status == "Open" && p.Orders.All(o => o.OrderId == orderId))
    .ToList();

foreach (var pickList in pickLists)
    pickList.Status = "Closed";

When I was reading the code, this line didn't catch my attention. It reads exactly like the rule: close the open pick lists whose orders are all this order. A pick list shared with order C doesn't match, which is what we want.

But All() on an empty collection returns true. It's even in the documentation: "…or if the sequence is empty". With no links, nothing breaks the condition, so a pick list without any orders matches every order.

💡 Tip: Search your code for .All( on navigation collections. Each one is true for an entity whose collection is still empty, so each is worth a second look.

Can a pick list exist without its orders? Not once it's saved. The pick list and its links are saved together in one SaveChanges(), which EF runs in one transaction. So nobody should ever see one without the other. Unless they read data that isn't committed yet.

The transaction

Deep in our persistence layer there is a base class, written years ago, that starts every transaction with the following code:

context.Database.BeginTransaction(IsolationLevel.ReadUncommitted);

READ UNCOMMITTED means dirty reads. The transaction sees rows that other sessions have written but not committed yet. The shipping code ran inside one of these transactions.

EF saves a new pick list with its link in two commands, because the link needs the pick list's generated id:

INSERT the pick list and read back its id.
INSERT the link row with that id.

Same transaction, but two round trips to the database. Between them, the pick list exists with zero orders. A dirty read at that moment sees exactly that, and All() matches it. Either one alone would have been fine.

It also explained why it started after the move. We still ran SQL Server with the same database settings, just on an Azure VM instead of a server in a hosted data center. The gap isn't only the network round trip; it also includes the work on the web server between the two commands, and that needs CPU time. I never measured it, but round trips in Azure are likely longer than before, and I suspect thread scheduling plays a part too: the earlier bug I mentioned at the start came down to how threads were scheduled there. During rush hours, more requests compete for the CPU and the network, so the gap gets longer and more requests land in it.

The original intent of that base class was most likely reads that never block: READ UNCOMMITTED, or NOLOCK on every query, was a common way to avoid blocking in older SQL Server codebases. SQL Server 2005 added a safer way, read committed snapshot isolation (RCSI), but turning it on changes every read in the database, so this base class never changed.

How RCSI works
When a row is updated or deleted, SQL Server first copies the old version into a version store (in tempdb, or in the database itself with accelerated database recovery). Readers take no locks on the rows. Each statement reads the data as it was committed when the statement started, using the old versions where needed. The cost is extra storage and a bit more work on every write.

Reproducing it

I couldn't reproduce it locally. I have a fairly fast machine, and the app on it had exactly one user: me. The gap was tiny, and the other request would have to land exactly inside it. Sometimes strong hardware is a disadvantage :D.

So I decided to make the gap bigger instead. EF Core command interceptors run around every command EF sends, so one of them can wait right after the pick list INSERT, while the transaction is still open:

public sealed class DelayAfterCommandInterceptor(string commandTextFragment, TimeSpan delay)
    : DbCommandInterceptor
{
    public override async ValueTask<DbDataReader> ReaderExecutedAsync(
        DbCommand command, CommandExecutedEventData eventData, DbDataReader result,
        CancellationToken cancellationToken = default)
    {
        if (command.CommandText.Contains(commandTextFragment, StringComparison.Ordinal))
            await Task.Delay(delay, cancellationToken);
        return result;
    }
}

This is a shortened version of the one in the repository, which also logs every command it sees. It only overrides the async method. If your code calls SaveChanges(), override ReaderExecuted the same way.

I registered it in debug builds only, for the insert I suspected:

#if DEBUG
optionsBuilder.AddInterceptors(
    new DelayAfterCommandInterceptor("INSERT INTO [PickLists]", TimeSpan.FromSeconds(2)));
#endif

That turns the tiny gap into a two-second one, and then it reproduces every time. This is the data in the repro when the race starts:

Order B gets pick list 3 while order A ships:

[ 0.04s] create ran: INSERT INTO [PickLists] ([Status])
[ 0.04s] create pausing 2.0s, transaction still open
[ 0.05s] ship   shipping order A, ReadUncommitted transaction, closing pick lists that served only A (All())
[ 0.19s] ship   found pick list(s) 1, 3
[ 2.05s] create ran: INSERT INTO [PickListOrders] ([OrderId], [PickListId])
[ 2.05s] create committed
[ 2.06s] check  order B's pick list 3 is Closed   <-- CLOSED BY SHIPPING ORDER A

Pick list 2 stays open, as it should. Pick list 3, caught between its two inserts, gets closed.

Fix

With a repro that fails every time, I could test the fixes instead of guessing. Measured on SQL Server 2022 with EF Core 10:

Query	Isolation	RCSI	Result
`All()`	Read uncommitted	off	❌ closes B's pick list
`Any() && All()`	Read uncommitted	off	✅ correct (the quick fix)
`All()`	Read committed	off	✅ correct, but waits 2.0 s
`All()`	Read committed	on	✅ correct, no waiting
`All()`	Read uncommitted	on	❌ closes B's pick list

 var pickLists = db.PickLists
-    .Where(p => p.Status == "Open" && p.Orders.All(o => o.OrderId == orderId))
+    .Where(p => p.Status == "Open" && p.Orders.Any() && p.Orders.All(o => o.OrderId == orderId))
     .ToList();

The second row is what we shipped: an empty collection can't match any more. It was small and safe, but the dirty read is still there and still sees half-saved data.

Without the dirty read, the bug is gone. Plain READ COMMITTED waits for the other transaction to commit, two seconds here; with RCSI turned on it doesn't wait at all. RCSI alone doesn't help, though: an explicit READ UNCOMMITTED, or a NOLOCK hint, overrides it.

Changing the isolation level in that base class changes every transaction in the application, so it's a separate change with a possibly huge impact on the legacy parts. Code that has read without waiting for years would suddenly wait on writers, or with RCSI read different data than before, and it's hard to tell which parts depend on the old behavior.

EF turns RCSI on

While building the repro, my "RCSI off" scenarios didn't block when they should have. It turned out that when EF Core creates a database with EnsureCreated() or migrations, it also runs ALTER DATABASE ... SET READ_COMMITTED_SNAPSHOT ON (except on Azure SQL Database, where it's on by default anyway). On SQL Server, a database created with a plain CREATE DATABASE has it off.

⚠️ Watch out: a local database created by EF can behave differently from production in anything that involves locks. If you're chasing a locking bug, check both sides first:

SELECT name, is_read_committed_snapshot_on FROM sys.databases;

Try it

pbednarcik / blog-examples

Companion code for blog posts: small examples and runnable repros.

It needs the .NET 10 SDK and a SQL Server where you can create a database. By default, it connects to the local default instance with Windows authentication; set REPRO_CONNECTION to a connection string to use another server. Every scenario drops and recreates a database called DirtyReadRepro.

git clone https://github.com/pbednarcik/blog-examples.git
cd blog-examples/ef-dirty-read-race
dotnet run

That runs all five scenarios and prints a summary of the results behind the Fix table. To run just one, for example, the bug itself:

dotnet run -- all ru rcsi-off

Have you hit a bug that only showed up after moving to the cloud? What was it, and how did you find it?

Thanks for reading.

I used AI to help edit this article.

FlexReadDir: up to 3.8x faster directory listing in Go

Peter Bednarčík — Sun, 11 Oct 2026 21:41:16 +0000

Almost every tool that looks at files asks the same question: What's in this directory, how big is each file, and when did it last change? Backup tools ask it, build caches ask it, sync clients, indexers, and file watchers ask it, thousands of times a minute.

In Go, the usual answer looks like this:

entries, _ := os.ReadDir(dir)
for _, e := range entries {
    info, _ := e.Info() // on macOS and Linux: one more system call per file
    use(e.Name(), info.Size(), info.ModTime())
}

On macOS and Linux, that's one call to list the directory, then one lstat for every file. On Windows, the listing already carries sizes and times, but the standard library still does a lot of work per entry. For 1,000 files, it makes 2,000 to 5,000 allocations, depending on the platform.

macOS can return names and metadata in one call, and Linux has cheaper ways to stat a file than using a full path. os.ReadDir with Info uses neither.

So I decided to write a small Go library that does: FlexReadDir. It lists a directory and gives you every file's size and modification time in one pass.

It's part of something bigger I'm working on, but I wanted to show this piece on its own. I also like optimizing software and getting as much performance as I can out of tools when it makes sense.

Quick start

go get github.com/pbednarcik/flexreaddir/frd

entries, err := frd.ReadDir(".")
if err != nil {
    log.Fatal(err)
}
for _, e := range entries {
    if e.Type.IsRegular() {
        fmt.Printf("%-24s %8d bytes  %s\n", e.Name, e.Size, e.ModTime().Format("2006-01-02 15:04"))
    }
}

ReadDir returns entries sorted by name, exactly like os.ReadDir. Each Entry is a plain struct. Size, times, and file id are filled for regular files only; for directories, links, and other types they are zero:

type Entry struct {
    Name            string
    Type            fs.FileMode
    Size            int64
    ModTimeNanos    int64
    ChangeTimeNanos int64
    FileID          uint64 // inode / file id; 0 if unknown
}

Because Entry is comparable, comparing a file's recorded metadata with the last scan is a single ==:

for _, e := range entries {
    if e.Type.IsRegular() && previous[e.Name] != e {
        fmt.Println("changed:", e.Name)
    }
}

With both the change time and the file id in there, a scanner can even tell a file that was rewritten in place from one that was replaced by another file with the same name, where the filesystem gives a persistent file id.

Benchmark

Platform	Files	`ReadDir` + `Info`	`Readdir`	FlexReadDir	Faster
macOS	1,000	1,382 µs	947 µs	362 µs	3.8×
Linux (VM)	1,000	603 µs	465 µs	376 µs	1.6×
Windows	1,000	164 µs	161 µs	131 µs	1.3×

FlexReadDir here is AppendDir, unsorted and reusing its result slice, as a walker would use it. "Faster" is against os.File.ReadDir with Info for each entry; os.File.Readdir is the older API that returns a FileInfo per entry. Every row is the median of ten interleaved runs on a warm file cache, so it shows warm-cache speed, not disk speed. The Mac is an M5 Max on APFS, the Windows box a Ryzen 9 9950X on NTFS, and Linux runs in a VM on that same Mac, so treat its row as a direction, not a size.

The gain is different on each platform, because the standard library does more extra work on some systems than on others. The macOS result surprised me the most. getattrlistbulk is a real bulk call, and it shows.

How it works

On macOS, it uses getattrlistbulk. You tell the kernel which attributes you want (name, type, size, times, file ID), and it returns records for many files at once. os.ReadDir + Info calls lstat for every file, so most of the 3.8× comes from removing those calls.

Linux has no bulk stat, so FlexReadDir still calls statx once per regular file. The savings come from everything around it. The stat is relative to the open directory instead of a full path; it asks only for the fields it needs, and it reads the getdents64 buffer directly without allocating per entry.

I plan to run it in Docker, so I made sure it works in containers. Some seccomp profiles block statx with EPERM. In that case, FlexReadDir falls back to fstatat, and there is a test with a real seccomp filter for it.

Windows already returns sizes and times with the listing (FILE_ID_EXTD_DIR_INFO), so there is little to gain. The savings come from less work per entry in user space and faster file-name decoding.

No cgo on any platform.

SIMD on Windows

I wrote my first small SIMD programs at university. One of them, for a course assignment, calculated a matrix determinant with AVX, and I compared it with a plain C version, so I saw how fast SIMD can be.

Windows returns file names in UTF-16, and Go strings are UTF-8, so every name must be converted. Most names are plain ASCII, which fits SIMD well. It checks 16 or 32 characters at once and, if they are all ASCII, packs them in one step.

The decoder has three versions: AVX2, SSE2, and plain Go. The right one is picked at runtime, and -tags purego forces plain Go. On long names, like in WinSxS, where names average 94 characters, AVX2 decodes about 4× faster than plain Go, and listing that directory is 17 to 20% faster. All three are tested exhaustively and fuzzed against each other.

Profiling

This stage probably took the most time. Constant reruns, long Google searches, documentation checks, and a lot of waiting on the rebuild-and-retest loop. But that's the price of optimizing.

After the obvious wins, I wanted to know what was left, so I profiled listings down into the kernel. On Windows, I used the Windows Performance Recorder with Microsoft's public symbols.

In my profiles, FlexReadDir's own code was about 4% of the time on macOS, 5% on Linux (a tree walk in Docker), and under 13% on Windows. The rest is mostly the kernel.

Two other things stood out: how much work NTFS does on its own, and how quick Linux is with file name lookups. The Linux part didn't really surprise me. I remember Linus Torvalds saying in a talk that he spent a lot of time keeping cache misses in file name lookups as low as possible.

On a warm cache, 40 to 49% of a Windows listing is NTFS reading ahead in the directory index, and I couldn't do anything about it. Or I just haven't figured it out yet. Access hints when opening the directory didn't change anything. A 16-times-bigger buffer (1 MiB instead of 64 KiB) wasn't faster either, and on the system drive (C:) it was 4.8% slower. In the end, I left it to Windows. At some point, you have to call it good enough, and I'd reached it.

Allocations

I come from the C# world, but I've always been interested in memory and being close to the metal.

Listing 1,000 files takes 4 or 5 allocations when you reuse the result slice. The names from one read share one allocation, and a Lister keeps its 64 KiB buffer, so when you list many directories you can reuse both:

var l frd.Lister
var entries []frd.Entry
for _, dir := range dirs {
    f, err := os.Open(dir)
    if err != nil {
        return err
    }
    entries, err = l.AppendDir(entries[:0], f)
    _ = f.Close()
    if err != nil {
        return err
    }
    // use entries before the next iteration overwrites them
}

Because names share memory, keeping a Name for a long time keeps the others alive too. Use strings.Clone if that matters.

What it doesn't do

It lists one directory and doesn't walk trees. If you need a walk, an example on pkg.go.dev builds one on top of it.

Help wanted

I don't have Linux on real hardware, an Intel CPU, or NFS and SMB shares to test on. If you have any of them, clone the repository and run this. It takes about a minute:

for i in $(seq 10); do go test ./frd -run '^$' -bench 'AppendDir' -benchmem -count 1; done

On Windows, in PowerShell:

1..10 | ForEach-Object { go test ./frd -run '^$' -bench 'AppendDir' -benchmem -count 1 }

The benchmark creates its files in the temp directory, so to test an NFS or SMB share, point TMPDIR (TMP on Windows) at a folder on the share first. Open an issue with the output and a line about the machine.

Try it

GitHub: github.com/pbednarcik/flexreaddir
Docs and examples: pkg.go.dev/github.com/pbednarcik/flexreaddir/frd

It requires Go 1.27 or later, runs on macOS, Linux, and Windows, and is MIT-licensed.

Go was an experiment for me, a way to try something new. It has a garbage collector like C#, so it felt familiar. In the end, it was a great way to learn Go, and I'm sticking with it, at least for now.

Let me know what you think, especially if you write Go. The language is still new to me, and I tried really hard not to write C# in Go. Hopefully I managed :D If something in the API or the code looks odd, tell me.

Thanks for reading.

I used AI to help edit this article.