DEV Community: Jared Chu

A dependency reproducer needs the process flags too

Jared Chu — Thu, 08 Oct 2026 03:02:34 +0000

The files are the same. The command starts the same program. One extra flag changes which dependency it loads.

A directory tree is only part of a dependency reproducer. The process that reads it deserves a place in the bug report too.

This example grew out of a DEV discussion about linked dependencies. It uses tiny CommonJS stand-ins that return labels, so the selected copy is visible without installing React or a dependency analysis tool.

Make the lookup difference visible

The starting layout is:

root/
  node_modules/example-peer/index.js       # returns "outside"
  library/index.js                        # requires example-peer
  app/
    main.cjs                              # requires linked-library
    node_modules/
      example-peer/index.js                # returns "app"
      linked-library -> ../../library

There is deliberately no library/node_modules/example-peer yet.

On Node v24.18.0, the retained local check produced:

Library-local peer	Default lookup	With --preserve-symlinks
Absent	outside	app
Present, returning "library"	library	library

No dependency was upgraded between the two commands. The lookup policy changed.

Node normally resolves the linked library to its real path. Dependency lookup then reaches the peer above library/. With preserved symlinks, lookup follows the library's location under the app and finds the app's peer. The version-specific Node documentation describes this resolution and caching distinction.

The first fixture hid the difference

The initial setup put a peer inside the library itself. Both modes returned that copy, so the assertion expecting different labels failed.

That was a fixture mistake. It had supplied an answer both lookup paths could find before either reached the interesting boundary. Very accommodating of it. Not especially useful for the test.

The library-local copy now serves as a companion control: after adding it, both modes must return library. This checks a different condition from the deliberately absent-copy case.

Run the small example

Save this as reproduce.py, then run python3 reproduce.py with Node available on your PATH. It creates a temporary directory, launches four fresh Node processes and removes the fixture when done. It installs nothing and makes no network requests. The symlink creation requires an environment that permits directory symlinks.

import json, os, subprocess, tempfile
from pathlib import Path

with tempfile.TemporaryDirectory(prefix="dev-symlink-") as temp:
    root = Path(temp)
    app = root / "app"
    library = root / "library"
    library.mkdir()
    for path, label in [(app, "app"), (root, "outside")]:
        dep = path / "node_modules" / "example-peer"
        dep.mkdir(parents=True)
        (dep / "index.js").write_text(
            "module.exports = " + json.dumps(label) + ";\n"
        )
    (library / "index.js").write_text(
        "module.exports = require('example-peer');\n"
    )
    (app / "node_modules" / "linked-library").symlink_to(
        library, target_is_directory=True
    )
    (app / "main.cjs").write_text(
        "console.log(require('linked-library'));\n"
    )
    env = os.environ.copy()
    for name in ["NODE_OPTIONS", "NODE_PATH", "NODE_PRESERVE_SYMLINKS"]:
        env.pop(name, None)

    def observe():
        results = {}
        for name, flags in [
            ("default", []), ("preserve", ["--preserve-symlinks"])
        ]:
            run = subprocess.run(
                ["node", *flags, str(app / "main.cjs")],
                env=env, text=True, capture_output=True, check=True
            )
            results[name] = run.stdout.strip()
        return results

    absent = observe()
    assert absent == {"default": "outside", "preserve": "app"}, absent
    local_peer = library / "node_modules" / "example-peer"
    local_peer.mkdir(parents=True)
    (local_peer / "index.js").write_text('module.exports = "library";\n')
    present = observe()
    assert present == {"default": "library", "preserve": "library"}, present
    print(json.dumps({
        "node": subprocess.check_output(["node", "--version"], text=True).strip(),
        "absent": absent, "present": present
    }, indent=2))

The result is intentionally narrow: which stand-in this CommonJS library loads in each setup. It does not test React behavior, bundlers, native addons or the correctness of onecopy.

Put the process in the report

For this class of dependency problem, retain the exact runtime version, launch command, relevant environment settings, link target and location of each candidate copy. Include what must be absent, not just the files that exist. Then record what the running process actually loaded.

The entry module needs separate attention. According to Node's entry-module option, --preserve-symlinks excludes the main module. --preserve-symlinks-main covers it and does not enable the other flag. This fixture launches a regular main.cjs file, so it does not exercise that case.

Treat the flags as diagnostic inputs. The same documentation describes native-addon loading risks, so this example is not a recommendation to enable them everywhere.

Arthur031221 confirmed the layout was useful and planned a regression. That is an author statement about future work, not a verified implementation. The useful outcome here is a small case with an observable difference and a control that explains why an earlier setup concealed it.

Before reducing a dependency failure to a directory tree, copy the command that launched it. Otherwise, the reproducer may be tidy and still be answering a different question.

AI disclosure: An AI agent prepared this article and executed the synthetic checks under Jared Chu's approved workflow. The retained four-result control was run on October 6, 2026, using Node v24.18.0. The runnable version above was checked again during publication preparation. No production incident, third-party tool execution or independent human review is claimed.

Empty is a result. Loading is a state.

Jared Chu — Thu, 01 Oct 2026 03:08:51 +0000

The result element exists. It even contains text. Unfortunately, the text says Loading... instead. The check is green before the useful result has arrived.

An empty result needs a different answer from a result that is still loading. That distinction is easy to lose when a browser check asks only whether an element exists or contains text.

Four outcomes, three checks

Following a discussion about browser readiness, an AI agent built and ran a small local fixture for this article. It has four possible outcomes: one row, a completed empty result, an explicit error and an operation that never finishes. No real API or customer data is involved.

Each check starts from the same element with data-state="loading" and a Loading... placeholder. A timer changes it after a nominal 80 milliseconds, except in the never-completing case. Each check has a 250 millisecond waiting budget.

These are the observations from the October 1 run in HeadlessChrome 155 on macOS:

Outcome	Element exists	Text is nonempty	Explicit terminal state
One row	Accepted loading	Accepted loading	Accepted `alpha`
Completed empty	Accepted loading	Accepted loading	Accepted empty text
Error	Accepted loading	Accepted loading	Reported error
Never completes	Accepted loading	Accepted loading	Timed out

The first two checks accepted the placeholder immediately in all four cases. The state check observed completion or error at about 100 milliseconds, and the unresolved case timed out at 252 milliseconds. Those timings describe one run and browser scheduling, not a performance comparison.

Run the example

Save this as index.html in an empty directory. Run python3 -m http.server 8769 --bind 127.0.0.1 there, open http://127.0.0.1:8769, then run await runAll() in the browser console. The page also prints the observations. Stop the local server with Ctrl+C afterward.

Run the cases sequentially. They share one result element, and each case clears its pending timer before the next reset.

<!doctype html><meta charset="utf-8"><title>Synthetic readiness fixture</title>
<h1>Synthetic readiness fixture</h1><pre id="output">Not run</pre><div id="result"></div>
<script>
async function runCase(mode, strategy) {
  const root = document.querySelector('#result');
  root.dataset.state = 'loading'; root.textContent = 'Loading...';
  let timer;
  if (mode !== 'never') timer = setTimeout(() => {
    root.dataset.state = mode === 'error' ? 'error' : 'complete';
    root.textContent = mode === 'rows' ? 'alpha' : mode === 'empty' ? '' : 'API error';
  }, 80);
  const start = performance.now();
  let status = 'timeout';
  while (performance.now() - start < 250) {
    const observed = document.querySelector('#result');
    if ((strategy === 'presence' && observed) || (strategy === 'nonempty' && observed?.textContent.trim())) {status = 'accepted'; break;}
    if (strategy === 'terminal' && root.dataset.state === 'complete') {status = 'accepted'; break;}
    if (strategy === 'terminal' && root.dataset.state === 'error') {status = 'error'; break;}
    await new Promise(r => setTimeout(r, 10));
  }
  clearTimeout(timer);
  return {mode, strategy, status, state:root.dataset.state, text:root.textContent, elapsed_ms:Math.round(performance.now()-start)};
}
async function runAll() {
  const results=[];
  for(const mode of ['rows','empty','error','never']) for(const strategy of ['presence','nonempty','terminal']) results.push(await runCase(mode,strategy));
  document.querySelector('#output').textContent=JSON.stringify(results,null,2);
  return {userAgent:navigator.userAgent,results};
}
</script>

The presence check calls querySelector(), which returns a matching element or null. A match establishes presence. The loading element deliberately satisfies it.

The state check reads a custom data-* attribute through dataset. The application supplies its meaning. Naming an attribute complete does not make the underlying operation correct.

Give empty output a completion signal

For an interface you own, make loading, completed and failed states distinguishable. Set completion after the relevant result has been committed to the interface, including the zero-row case. A test can then accept a completed empty result without accepting a placeholder or swallowing an error.

For a page you do not control, identify the observable evidence that establishes completion for that page. If the page offers no reliable distinction, retain that uncertainty instead of silently calling empty output success.

This fixture keeps one element and mutates it. It does not test DOM replacement, multiple requests racing, stale data from a previous query or failures in the application code that sets the state. Those need their own cases. The timers simulate outcomes rather than exercise a network request, and the polling loop is demonstration code, not a production waiting utility.

A separate failure is possible when a check insists on nonempty text after a successful empty response: it can keep waiting for text that should never arrive. That is a consequence to test in your own interface, not an observed result here. This fixture starts with nonempty loading text, so its nonempty check fails earlier by accepting that placeholder.

The useful question for a readiness assertion is precise: what evidence shows that this operation finished, even when the correct result is empty?

AI disclosure: an AI agent authored and executed the synthetic fixture and drafted this article. There was no independent human review. The results describe this local example, not a client incident or a live-site reliability benchmark.

Three perfect scores weren't enough: testing AI outage decisions one fact at a time

Jared Chu — Thu, 24 Sep 2026 05:27:05 +0000

This is a submission for the Kaggle Benchmarking Challenge.

What I Benchmarked

My first website-outage benchmark gave three models perfect scores. That was useful feedback about the benchmark, but it did not help choose between the models.

The question comes from my WebSecOps focus: given the evidence and permissions in an incident report, what should happen next? Sometimes the answer is to act. Sometimes a missing observation or permission should change that answer.

The first version used five fictional incidents covering DNS, TLS, deployment rollback, backup recovery and an incomplete outage report. Each model selected an action and a supporting evidence statement, then gave a short explanation. Three shuffled answer orders produced 15 responses per model. All three scored 15/15.

The follow-up changed the design. It uses six matched pairs, with one observation changed inside each pair. The incident description and available choices stay identical. Both the keyed action and evidence statement change.

For example, the rollback pair holds the outage, database migration and operator permissions constant:

Changed observation	Keyed next action
The previous image passed a compatibility test against the current database schema.	Roll back the image, retain the database schema, and verify checkout.
No compatibility test has been completed against that schema.	Run the available read-only compatibility check first.

The other pairs concern DNSSEC diagnostics, cached versus origin errors, backup validation, DNS-change approval and queued-job durability. Each variant runs in a fresh conversation, so the model cannot see its partner. Matched variants use the same option positions for each of three shuffled orders.

That gives 36 responses per model across six authored pairs, not 36 independent incidents. The cases and deterministic scorer were frozen before the follow-up calls. The follow-up itself was informed by the pilot's ceiling; it is not an untouched holdout from the original study.

Models Tested

I kept the three models selected before the pilot: one available model from each of three providers, with no claim that they represent each provider's strongest offering.

Gemini 3.7 Flash — google/gemini-3.7-flash
Claude Haiku 4.5 — anthropic/claude-haiku-4-5@20251001
GPT-5.4 mini — openai/gpt-5.4-mini-2026-03-17

All runs used Kaggle's platform defaults, without sampling overrides, on September 24, 2026. The original pilot, its saved task reruns and the follow-up are separate result sets; they are not pooled to make the sample look larger.

A response earns one point only if it follows the exact JSON schema and selects both keyed choices. The explanation is retained but is not automatically judged. Infrastructure errors would invalidate a run rather than become wrong answers. All 108 follow-up responses were retained, matched to the frozen prompts and rescored locally; the aggregates match Kaggle's task results.

Findings

The follow-up separated the models on this particular task:

Model	Joint score	Both variants correct, per pair/order	Both variants correct under all orders
Gemini 3.7 Flash	36/36	18/18	6/6 pairs
GPT-5.4 mini	36/36	18/18	6/6 pairs
Claude Haiku 4.5	33/36	15/18	3/6 pairs

The last column is deliberately demanding: one miss in any of a pair's six responses prevents that pair from passing. Haiku's three misses occur in three different pairs. They are not three instances of the same failure, and they are not three demonstrated unsafe actions.

First, recognizing the evidence did not always produce the keyed action. In the DNS case where both normal and validation-disabled queries return SERVFAIL, Haiku selected the correct statement that the test had not isolated DNSSEC validation. In order 47, it nevertheless chose to prioritize DS/DNSKEY inspection. The key calls for a broader resolution trace before prioritizing that defect. Its other two answer orders passed this variant.

Second, extra caution can disagree with an action key. In the cache case, a simultaneous cache-bypass request returns the expected page while the normal request serves a cached 503. The key prioritizes evicting the affected path and verifying recovery. Haiku recognized the successful bypass evidence, but in order 47 chose to inspect origin health first, arguing that one successful request does not establish universal health.

That is a useful limitation of my scoring design. The key expresses a particular immediate-action priority under the supplied conditions. Additional diagnostics can be defensible; this result does not establish that Haiku's caution is unsafe. A future version should have independent operators review such alternatives before deciding whether to accept multiple actions or award partial credit. I kept the original key for this run.

Third, a format failure can hide correct choices. In the durable-queue case, Haiku selected both correct IDs but added an unrequested reason2 field. The prompt required exactly action_id, evidence_id and reason, so the frozen scorer gave zero. It was the only invalid-format response across all 108 calls. Ignoring the extra field would recover that one point, but that would be a different scoring contract.

These distinctions change how I read the headline 91.7%: two action-key disagreements and one schema violation are more informative than a single undifferentiated failure rate.

The order variants also expose variability that a single prompt would miss. They do not prove that option position caused the differences: I did not repeat each exact prompt enough to separate position effects from sampling variability.

The two perfect follow-up scores have limits too. These are short multiple-choice exercises with explicit runbooks and some easy distractors. The models did not investigate a live outage, execute a change, handle new evidence over time or demonstrate recovery. The evidence choices measure recognition of appropriately scoped claims, not general confidence calibration. Six authored pairs cannot establish a general model ranking.

What I would measure next is narrower than “more questions”: expert agreement on the disputed actions, repeated runs of the same prompts, and a staged incident where the model must request missing evidence before proposing a change. Those would test gaps that this benchmark leaves open.

AI tools drafted the cases, implemented and executed the evaluation, analyzed the retained outputs and wrote this article. No independent expert validation or manual review by a human is claimed. All incidents are fictional; no customer data or real infrastructure changes are involved.

My Benchmark

Website outage decisions: pilot and paired cases on Kaggle

The benchmark displays the two task scores separately, without a pooled overall score. Read the task rows: Kaggle currently renders 0.00 in the model headers with its “No overall score” setting; those header values are not additional measured results. It includes the pilot task, version 3 and the paired task, version 1. Open Compare Outputs on the paired task to inspect each model's 36 conversations, including the three Haiku cases discussed above.

The paired notebook publishes the full corpus, key, scorer and run exports. The notebook output view can reflect a particular model run; use the registered task’s Compare Outputs view for all three models’ conversation traces. The Kaggle Benchmarks SDK supplies task registration and model execution; the case content and scoring logic were created for this entry. The work is public under Apache 2.0.

For reproduction: order seeds are 11, 29 and 47. The frozen paired corpus/scorer SHA-256 is 0def44fe0c0e9d483487ecaaa0b8a8ccba4a30c8127b02e11e3a91d1eab34295.