We built a Rust EDR. macOS notarization tried to kill it.

Over the last few months we shipped Nemesis Blue, an endpoint detection and response agent written in Rust, notarized and running on macOS. This is the build story: why Rust, what notarization actually costs you, the entitlement bug that ate a week, and how you do detection engineering inside Apple's hardened runtime.

No marketing here. Just the parts that were hard.

Why Rust for an EDR

An EDR runs on every machine you protect, with high privilege, all the time. That job description rules a lot of things out. A garbage-collected runtime that pauses mid-scan is not acceptable. A memory-safety bug in an agent that sits just above the kernel is an incident waiting to happen, on your customer's box, with your signature on it.

Rust gave us three things that mattered:

• No GC pauses. Scans run in bounded time, every time.
• Memory safety without a runtime. The agent ships as a single static binary.
• One codebase across platforms. Our crates (nemesis-core, nemesis-scan, nemesis-detect, and friends) build for macOS, Linux, and Windows sensors out of the same Cargo workspace.

That last decision paid off later: once the macOS agent was solid, the Linux build came up end to end in days instead of weeks, because everything below the platform layer was already shared and tested.

Notarization is the part nobody warns you about

If you want a macOS binary to run on someone else's machine without a scary Gatekeeper wall, you notarize it. To notarize, Apple requires the hardened runtime. And the hardened runtime quietly changes the rules your process runs under: no unsigned executable memory, no loading unsigned libraries, restricted debugging, and by default, no JIT.

That "no JIT" line is where we lost a week.

The SIGKILL that ate a week

Our scanning engine is built on YARA-X, the Rust rewrite of YARA. YARA-X compiles rules down and runs them on wasmtime. Wasmtime is a JIT: to execute, it allocates memory, marks it executable, and jumps into it.

Under a normal build, this is invisible. Under the hardened runtime, the kernel refuses to hand you executable-writable pages. The moment the scanner tried to run its first rule, the whole daemon died. Not an exception we could catch. An immediate SIGKILL, with Console showing a code-signature violation.

The symptom was maddening because everything worked in development and died the instant it was signed and hardened. Same binary, different fate, depending on entitlements.

The fix is one entitlement:

<key>com.apple.security.cs.allow-jit</key>
<true/>

Add it, re-sign, re-notarize, and wasmtime can allocate its JIT pages again. Obvious in hindsight. Not obvious at 2am when your only clue is a process that vanishes without a stack trace.

The lesson we took from it: on macOS, your entitlements are part of your runtime behavior, not a checkbox at the end. If a dependency touches executable memory, dynamic libraries, or the network, that shows up here or your process dies in ways your test suite never sees.

Detection engineering under the hardened runtime

With the engine actually running, the real work is detection. Nemesis Blue runs three layers:

Signatures. YARA-X rules for known families and behaviors. Fast, explainable, and the thing an analyst can read and trust.

Machine learning. An XGBoost model trained on the EMBER feature set for static classification of executables. The model is the interesting supply-chain problem: if an attacker can swap your model file, they own your verdicts. So model bundles are signed with Ed25519, and the agent verifies the signature before it will load a model. A model that does not verify does not run. Retraining happens weekly in CI, and it only publishes when the labeled corpus is large enough to be worth shipping. New models roll out to a canary slice of the fleet first, so a bad training run degrades a handful of machines instead of all of them.

IOC matching and quarantine. Indicator feeds for the known-bad, and quarantine to pull a file out of harm's way once something fires.

None of these are novel on their own. The engineering is in making all three run in bounded time, on someone else's laptop, without being the reason their fan spins up.

The 78-second freeze

Memory scanning is where we hit the ugliest performance bug. Our first pass at walking a process's memory regions shelled out and parsed the output. On small processes it was fine. On a large one, it serialized behind a single call that blocked for 78 seconds. For an agent that is supposed to be invisible, 78 seconds of a pegged process is not a bug, it is an eviction notice.

The fix was to stop treating memory enumeration as a subprocess and read the regions directly, with hard caps on region size and early skips for the mappings we never scan. Bounded work, off the hot path. The 78 seconds became milliseconds.

This is the recurring theme of EDR work: correctness is table stakes, but the thing that gets you uninstalled is being heavy. Every scan has to answer "what did this cost the user."

Shipping, and staying shipped

The agent ships as a notarized .pkg. But shipping once is not the hard part. Keeping a fleet current is.

Nemesis Blue agents check for updates at boot and every few hours after, on a stable channel, and self-install a newer signed and notarized package when one is published. Pushing a release is: bump the version, build the package, notarize it, publish the manifest. Because updates are gated behind signing and notarization, a compromised update server still cannot push a payload the agents will accept. The agents only run what Apple and our signing key both vouch for.

What it honestly does today

Here is the part most product posts skip. As of today, Nemesis Blue does detection: scanning, YARA-X, the ML classifier, IOC matching, quarantine, and monitoring, all in a notarized agent you can install right now.

What it does not do yet is real-time blocking. Full inline enforcement (killing a malicious process before it executes, streaming kernel events, process attribution) depends on Apple's Endpoint Security entitlement, which is a manual review with a multi-week queue. We have it submitted. Until it clears, enforce mode stays behind the flag, and we would rather say that plainly than imply a capability we cannot ship. The moment the entitlement lands, the blocking path is already built and waiting for it.

Takeaways

If you are building a signed, notarized native agent on macOS:

• Treat entitlements as runtime behavior. allow-jit for anything wasm or JIT, and audit every dependency for what the hardened runtime will forbid.
• Sign your models, not just your binary. A swapped model is a swapped verdict.
• Measure what every scan costs the user. Heavy gets you uninstalled faster than wrong.
• Be honest about what is gated. The entitlement queue is real, and pretending otherwise just burns trust with the exact people who will read your Console logs.

Nemesis Blue is the defensive half of what we are building at Nemesis Labs. The offensive half, Nemesis Red, is an autonomous pentest engine, and it has its own build story that is nothing like this one. That post is next.

If you want to follow along or kick the tires, we are at nemesislabs.xyz.