DEV Community: Mackenzie

GhostTrace: A Local Workbench for Looking Inside Language Models

Mackenzie — Wed, 28 Oct 2026 02:39:34 +0000

A language model gives you an answer. Understanding what changed inside the model—and whether that change matters to its output—is a harder problem.

I built GhostTrace to make that investigation easier to inspect, repeat, and share.

It’s a local-first research engine and visual workbench for controlled prompt comparisons and mechanistic interpretability experiments. You can compare model responses, inspect internal activations, run targeted interventions, and export evidence that someone else can replay without loading the model.

Source code on GitHub

Start with evidence you can explore

One decision shaped the project: opening a research tool should give you a way to understand it before asking you to download model weights.

GhostTrace includes recorded measurements from Qwen2.5-0.5B-Instruct running locally on CPU in FP32.

To explore them:

npm ci
npm run dev

Open http://127.0.0.1:5173 and choose Open recorded smoke, or open the Research lab → Recorded research case.

After setup, recorded replay requires no model download or running inference backend. The workbench loads saved evidence rather than generating new measurements.

There’s also an in-app walkthrough, captions, a guided demo, and searchable help.

From prompt comparisons to internal interventions

The original workflow compares controlled prompt conditions. It records generated responses and aligned residual activations, with repeat, label-only, and wording controls.

The bundled smoke diagnostic completed all 42 planned units, covering three development scenarios, two template families, and three instruction conditions.

That establishes a small working diagnostic. A full confirmatory study still requires additional review and evidence.

The newer Research Lab goes deeper. Its recorded copying example lets you inspect:

A component graph with attention-head and grouped-query attention structure.
Signed activation tiles for residual streams, attention and MLP inputs, head contributions, and selected MLP channels.
Logit-lens readouts showing how intermediate representations relate to target-token predictions.
Patching and ablation experiments that measure how targeted changes affect the output.
Gradient attribution alongside exact intervention checks.

The goal is to connect an observation to an experiment: identify a component, inspect its activity, change it under a defined control, and measure the result.

A visualization needs an evidence trail

An activation difference can be interesting without explaining a behavior.

A component might change because of literal prompt wording, token position, sequence length, or ordinary instruction following. Gradient attribution provides another useful signal, but its relationship to an actual intervention needs checking.

GhostTrace keeps those distinctions visible.

For example, replacing the entire final residual stream can serve as an endpoint positive control. A successful replacement does not establish that we discovered a circuit.

The recorded research case also includes dictionary projections and a small trained affine lens. Their limits matter:

The coordinate-identity dictionary is a diagnostic basis, not a learned sparse autoencoder.
Reconstruction error remains visible.
The affine lens was fitted on 24 development prompts and evaluated on eight held-out prompts.
Those small datasets do not establish broad predictive quality.

Negative effects and recovery values outside the usual range remain in the evidence. They’re part of what happened.

Replay is part of the research workflow

Sharing a screenshot is easy. Sharing enough information to inspect the experiment is harder.

GhostTrace supports portable evidence bundles, integrity checks, and offline metric recomputation. Saved results can be viewed without loading model weights.

That gives the project two useful workflows:

Explore recorded evidence: inspect outputs, graphs, activations, and controls.

Run local experiments: configure conditions and targets, queue capture or intervention jobs, then export the resulting evidence.

The live stack uses React, TypeScript, and Vite for the interface; FastAPI for the API; SQLite-backed job storage; and a single explicit worker. Model capture uses direct Transformers hooks.

To start the local stack:

uv sync --frozen --extra test
npm run dev:full

Live inference requires a separate, explicit model setup. Starting the application or opening saved results does not download model weights.

Keeping research claims bounded

The project includes automated checks for API contracts, storage and recovery, evidence handling, browser workflows, and real-model behavior.

Human review remains a separate requirement.

Dataset review, blinded ratings, a full confirmatory pilot, and final human release QA are still pending. Missing evidence stays pending.

That boundary is especially important for the original persona experiments. An instruction labeled “remorseful” describes a prompt condition. A response expressing regret does not establish a feeling, and a representation difference does not establish a psychological state.

GhostTrace is a tool for investigating model behavior. The conclusions still have to be earned through the experiment.

What I want to improve next

The next work is completing the human review process, expanding the evidence beyond small diagnostics, and making it easier to follow an experimental question through to its controls and results.

I’m particularly interested in feedback from people building interpretability tools:

What information do you need to inspect before you trust an activation or intervention result?

Explore GhostTrace on GitHub

Watch the walkthrough

See GhostTrace in action:

Watch on YouTube

Static demo

SlothLab: HacktoberFest Week 3 Challenge

Mackenzie — Mon, 26 Oct 2026 02:35:24 +0000

An AI research loop can generate another experiment long after it has stopped learning anything useful. I wanted a small laboratory where I could inspect what the agent ran, why it ran it, and why it stopped.

That became SlothLab + LazyLoop: a local scientific research application with a Streamlit workspace, an Ollama-compatible model connection, reproducible experiments, and a controller that treats compute as a finite resource.

Source code · Static evidence explorer · Video help guide

The video is an actual automated browser recording. It explores a previously completed investigation, downloads its research bundle, and reopens that history in Guest mode. It does not show a new autonomous investigation finishing in 96 seconds.

Start with a narrow scientific question

The first experiment family studies oracle prior-probability correction under synthetic class prevalence shift.

In practical terms: if the class balance changes, when does adjusting predicted probabilities improve Brier score? The experiment supplies the true evaluation prevalence as an oracle input. That assumption is part of the research question, not a capability to infer prevalence from arbitrary real-world data.

This scope gives the agent something measurable to do. It proposes hypotheses and registered experiment specifications; deterministic Python procedures execute the science. It cannot invent a shell command and call that an experiment.

The workflow is:

Question → hypotheses → validated candidates
                    ↓
           execute / reuse / reject
                    ↓
         measurements → interpretation
                    ↓
         independent confirmation or stop

A model explanation remains provisional. It does not become a scientific finding simply because it sounds confident.

Let code enforce the research boundary

LazyLoop validates proposed conditions, checks budget feasibility, finds verified exact-result matches, and tracks whether the investigation is progressing.

For example, this is real validation code from slothlab/lazyloop/controller.py:

allowed_seeds = (
    config.exploratory_seeds
    if spec.phase == "exploratory"
    else config.confirmatory_seeds
)
if spec.seed not in allowed_seeds:
    raise ValueError("Seed does not belong to its predeclared phase partition")

That check expresses a research decision: confirmation seeds are reserved before exploration. The agent cannot quietly recycle exploration into independent confirmation by changing the label.

Exact-result reuse is another useful distinction. Repeating an identical experiment can return a verified cached result. Changing the seed can create a genuinely independent replication. A cache hit must never count as a second independent measurement.

What the completed example actually showed

One recorded investigation reached its declared operational completion criteria with:

Five original experiment executions.
One verified exact-result reuse.
Three independent confirmation seeds for a fixed condition.
10,712 model tokens.

The mean confirmatory delta Brier was approximately −0.00619, where delta means corrected minus baseline. The three confirmation measurements supported improvement for that particular synthetic condition.

That is a scoped result. It does not show that correction helps every dataset or that this agent generally saves compute. Fixed-schedule replay comparisons exercised reuse, but the repeated autonomous evaluation did not establish an efficiency advantage with sufficient confirmatory coverage. Those are different claims, so the reports keep them separate.

The original example and measured reports are available for inspection.

Make the application usable between experiments

The Streamlit interface handles investigation design, measurements, decisions, pause/resume, reports, and comparisons. Background research runs in a separate local process, so closing a browser tab does not itself abandon a job.

I also added the features that make repeated use less awkward:

Light, Dark, and System appearance.
Remembered local browser profiles with separate histories.
Guest research using JSON documents and artifact files, without SQL.
Portable ZIP bundles that can be downloaded and reopened.
A learning area with chapter navigation, captions, transcripts, practice questions, and a research-design checklist.

A browser profile is a convenience on a trusted local installation, not a cross-device identity service. Guest mode uses temporary files; it is not a promise of secure erasure. Imported archives are validated data, and opening one starts no inference.

Inference can run locally or on a reachable remote Ollama-compatible HTTP(S) server. Scientific experiments and research files stay on the app host. The adapter does not support arbitrary OpenAI/Anthropic APIs or authenticated gateways.

A static demo should tell the truth

GitHub Pages cannot run the Python research application. The static demo instead lets visitors filter real recorded measurements, inspect an experiment table, read the limitations, and download research-design notes.

Those notes are a planning worksheet, not executable configuration. The page makes no model calls. The help page embeds the YouTube walkthrough after an explicit playback action and explains how to run the full app.

This makes the project explorable without pretending a static website is carrying out scientific work.

Verification and remaining work

The 0.2.0 acceptance run passed 214 tests, including live Ollama checks, job pause/resume, portable-history validation, and JSON-only research workflows. The later live browser walkthrough additionally exercised real bundle download, guest restore, and local-profile creation.

The application is intended for a single-user Linux installation bound to loopback. It is not a reviewed public multi-tenant service. Execution isolation depends on the host; a restricted subprocess fallback is weaker than working OS namespaces. Passing tests does not remove those deployment limits.

The next useful work is broader scientific families and stronger evaluation across independent autonomous trials. I would rather preserve an insufficient-evidence outcome than turn a demo into an efficiency claim the measurements do not support.

Try it locally

With Python 3.12+, Ollama running separately, and an installed model:

git clone https://github.com/mso-docs/LazyLab.git
cd LazyLab
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -e '.[dashboard]'
export OLLAMA_MODEL='your-installed-model'
slothlab init
slothlab doctor
slothlab dashboard

Open http://127.0.0.1:8501. Start with a small, explicit question and a finite budget.

I would especially value feedback on where the evidence trail is unclear, which stopping decisions deserve stronger tests, and which first-run steps need better explanation.

Related DEV discussions

A Local-First Coding Agent Needs a Measurable Boundary, by @_firelinks, argues for making an agent's limits inspectable. That is a useful lens for assessing SlothLab's registered procedures and explicit budgets.

How Do We Evaluate AI Agents? A Practical, End-to-End Framework for Reliability and Scale, by @kuldeep_paul, discusses evaluating trajectories and efficiency alongside outcomes. SlothLab's distinction between fixed replay, autonomous runs, and scientific confirmation follows that broader evaluation concern; it does not depend on the commercial tools discussed in that article.

ReasearchReady: Side Hustle AI Challenge Week 2

Mackenzie — Mon, 19 Oct 2026 01:47:34 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle.

Demo Website

What I Built

ResearchReady gives researchers a private second look at a manuscript before peer review. Upload a LaTeX project, inspect source-linked findings, compare scientific claims with candidate manuscript evidence, and export a review report.

It combines reproducible submission checks with optional open-model review: broken citations and missing files are checked without inference; claim assessment and reviewer questions use an Ollama model. Every AI assessment requires human verification.

The potential customer is a small research lab, independent researcher, or research consultant handling unpublished drafts. The side-hustle hypothesis is paid local installation, model configuration, and support around an Apache-2.0 application. The value to test is whether a team would pay to keep control of its manuscripts while reducing review setup and preparation work. Customer demand, time savings, and willingness to pay are not yet validated; I am not claiming paying users.

Demo

Video Demo

The app includes two synthetic manuscripts: one with seeded defects and one without deterministic submission defects. Choose Try the synthetic sample, run submission checks, and open a finding to inspect its exact source. Select qwen3.5:9b and enable research integrity and reviewer simulation to reproduce the AI workflow.

Actual PDF export from the flawed sample · Raw flawed audit · Raw valid audit

An actual ResearchReady report from a synthetic manuscript, with summary counts and source-linked findings

In the final live smoke run, the flawed manuscript claimed a 15% improvement while reporting 70% versus 72% accuracy. ResearchReady linked that mismatch to the results source and questioned a universal-generalization claim supported only by one synthetic experiment.

The comparison manuscript described a two-percentage-point gain without claiming statistical significance. Its final run produced no AI warnings: one comparison had candidate evidence, while lack of independent corroboration for raw accuracy figures remained a neutral observation.

Synthetic sample	Submission findings	AI warnings	Reviewer questions	End-to-end time
Flawed	8	2	3	17.12 s
Valid	0	0	3	16.12 s

These examples were used during iteration. They are smoke tests, not a held-out accuracy benchmark. Timing includes application and polling overhead on a remote Spark reached through SSH; hardware was not independently measured. The run summary records the exact model digest and settings.

Code

ResearchReady source and setup instructions

The workspace uses React and TypeScript, FastAPI, SQLite, and Ollama. Manuscript source stays in the backend's local project directory. The browser never receives the inference server address. JSON and paginated PDF exports support a review handoff.

How I Built It

The AI workflow uses qwen3.5:9b, Q4_K_M, served through Ollama. The server reports an Apache-2.0 model license. Claim quotes must match supplied raw source text; the application computes their locations. Lexical TF-IDF retrieval finds candidate evidence, excluding the claim's own occupied lines while preserving nearby source passages. Assessments and reviewer questions may cite only supplied passage IDs.

Live testing found problems that mocked tests could not. Default model thinking exhausted structured-output attempts. The runtime rejected bounded reviewer schemas with a grammar-parser error. Excluding an entire source chunk also hid evidence next to the claim and produced misleading results.

The fixes were to disable optional thinking for these structured requests, send a portable generation schema while retaining full Pydantic response validation, and retrieve nearby source spans without allowing a claim to support itself. Assessments now distinguish missing evidence from defects and descriptive percentage-point changes from statistical significance.

Thirty backend regression tests pass, including archive safety, exact quote grounding, evidence-location validation, status persistence, and report exports. Tests and the frontend production build also pass in an isolated installation without the private environment file. GitHub Actions runs the verification checks after pushing.

Why Does Open Innovation Matter?

A research team should control which model and machine process its unpublished work. Compatible open-weight models can be served through the team's own Ollama endpoint, without a proprietary inference API key. A local endpoint keeps processing on the same machine; a trusted remote endpoint, such as the Spark used for this evaluation, receives analysis content.

Open code makes the prompts, parsing, retrieval, and validation inspectable. Teams can adapt checks, replace a model, and run the workspace on infrastructure they control. Local or self-hosted inference still has hardware and electricity costs, but does not require a proprietary per-request inference bill.

The present scope is LaTeX manuscript review. It does not fetch cited papers, interpret figure pixels, establish statistical validity, or replace scientific judgment. The next useful validation is an independently labeled evaluation and real researcher feedback.

OutFit: A Little Movement. A Little More Outside.

Mackenzie — Mon, 12 Oct 2026 02:22:12 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass

What I Built

Getting outside can start with a small decision: what can I realistically do today?

I built OutFit, a fitness-planning prototype for adults who want to start moving or return to a regular routine. It turns a goal, available days, and recent activity into a seven-day plan with easy walking, gentle run-walk sessions, mobility, and rest.

The idea is to make planning a short step toward doing something outside. Open Today, check the activity, and head out. Afterward, record what you actually did and add a recovery check-in.

Plans can change as the week unfolds. If someone reports low energy or poor sleep, they can preview an adjustment to their remaining sessions before accepting it. Completed activity stays in their history.

OutFit encourages outdoor movement while allowing an indoor alternative when going outside is unsuitable. It is a habit-building prototype, with conservative planning limits, rather than a medically validated fitness service.

Demo

Watch the OutFit demo on YouTube

To try it locally, the README includes setup instructions and a walkthrough covering onboarding, plan generation, activity logging, and recovery adjustments. You can explore the core loop without a running model through the labelled fallback planner.

Code

View the OutFit repository on GitHub

mso-docs / OutFit-Fitness-Trainer

Made for the Hacktoberfest Week 1: Touch Grass on DEV.to!

OutFit

Licensed under the MIT License.

A little movement. A little more outside.

Live Demo Site

OutFit is a responsive, local fitness-planning prototype for the Touch Grass hackathon. It helps an adult choose a realistic goal, preview and accept a seven-day plan, record activity and recovery, and review an adjustment. Local Ollama selects activities within deterministic policy limits. A validated fallback keeps the core loop working without a model.

Demo video

Watch the OutFit hackathon demo on YouTube.

Run locally

Use Node 24 LTS and npm. From this repository:

npm ci
cp .env.example .env
npm run build
npm start

Open http://127.0.0.1:3001 on the same computer. The server creates and migrates data/outfit.db automatically. .env loads at startup; existing environment variables take precedence. Stop with Ctrl+C. Restarting preserves saved profiles, logs, check-ins, and plans.

For development:

npm run dev

Open http://127.0.0.1:5173. Vite proxies /api to the loopback Fastify server…

View on GitHub

The repository includes the application, shared planning rules, automated tests, setup documentation, and verification notes.

How I Built It

OutFit uses React, TypeScript, and Vite for the interface, with a Fastify API and SQLite for the local server edition. A separate browser edition stores application data in the browser and can connect directly to a user-configured Ollama endpoint.

The AI planner runs through Ollama. The verified setup used qwen3.8:27b to generate structured seven-day proposals from the user’s profile and planning context.

The model works within an activity catalog and explicit policy limits. Its output passes through Zod schema validation and additional checks for dates, available days, activity choices, training budgets, and recovery constraints. Users review and accept a preview before it becomes their plan.

If inference fails or the response does not pass validation, OutFit produces a clearly labelled deterministic fallback. That keeps planning usable when a model is unavailable.

One important design decision was keeping activity instructions in application code. The model selects structured activities; fixed text supplies the instructions. This makes the resulting plan easier to inspect and keeps generated prose from changing the prescribed activity.

I also added a trainer chat for questions about habits and the saved plan. Chat does not automatically change accepted plans.

Real Ollama planning and trainer responses were verified using synthetic data. The repository records 69 passing automated tests, plus desktop and mobile browser checks. The browser edition’s deployed HTTPS connection to real Ollama remains a separate verification step.

Why Does Open Innovation Matter?

OutFit handles information about someone’s activity, recovery, and personal routine. Running inference on a machine they control makes privacy part of the architecture.

With Ollama, users can choose an installed model and run the planner without sending their planning context to a hosted model provider. After the software and model downloads, the same-computer setup can operate using local resources.

Open tooling also makes experimentation practical. I can change the model, inspect the prompts, revise the validation rules, and test the same planning contract without rebuilding around a particular provider’s API. There is no required hosted-model subscription or per-request API bill, although inference still requires suitable hardware.

A closed API could generate a plan, but it would introduce an external service dependency and require sending context to that service. Local inference gave me control over those choices.

For OutFit, that control supports the purpose of the project: help someone decide on a manageable activity, then leave the screen and do it.

Call Mac: Patient AI Tech Support for My Mom

Mackenzie — Mon, 05 Oct 2026 02:20:06 +0000

This is a submission for the Hacktoberfest Weekend Challenge: Build for a Friend

What I Built

My mom regularly needs help with technology, and I can usually only help her over the phone.

The hard part is figuring out what she sees. I’m describing a button while she’s looking at a different screen, or explaining a step that assumes something she hasn’t learned yet. A small technical problem becomes a long conversation.

I built Call Mac to give her patient guidance when I can’t be beside her.

She describes a problem in her own words. Call Mac offers one manageable next step, then waits for her response: Fixed it, Still broken, or I don’t understand. She can also describe what happened or what she sees.

That feedback shapes the next response. The assistant can ask for missing context, explain an instruction, or change approach instead of continuing through a generic checklist.

Saved profiles remember devices and technical comfort. Confirmed fixes stay in local memory, so future conversations can draw on what worked before. Guest mode lets someone try it without saving a profile.

I wanted to build something around the way my mom needs help: clear instructions, room to ask questions, and patience when the first attempt doesn’t work.

Demo

Watch the Call Mac video demo.

Check out the GitHub Pages demo.

The local application includes saved profiles and remembered fixes.

The repository also includes a separate browser demo with a labeled fictional walkthrough and an option to connect directly to the visitor’s own Ollama installation. That version keeps conversations in the current tab and does not include persistent memory.

Code

View Call Mac on GitHub.

The repository includes installation guides, screenshots, automated tests, and fictional scenarios for evaluating model behavior.

How I Built It

Call Mac uses Python, Streamlit, SQLite, Pydantic, and Gemma through Ollama.

Gemma handles the contextual conversation. Each request includes the user’s problem, device profile, previous attempts, observations, and relevant remembered fixes. The model chooses whether to ask a question, give an instruction, explain an existing step, or change approach.

I used gemma2:9b for most of my experiments. The application defaults to gemma3:4b as a smaller starting point for users, and supports selecting other installed Gemma models.

I request structured responses through Ollama and validate them with Pydantic before displaying them. Invalid output gets one retry. The application also rejects exact and near-identical failed instructions, although it cannot reliably detect every semantic repetition.

Successful fixes are stored in SQLite and retrieved through keyword overlap. They are presented to the model as previous experiences, not guaranteed solutions.

Feedback is saved before the next inference request. If the model server fails, the user can retry without losing what they already reported.

Recognized liquid-damage situations have explicit safety routing that takes priority over ordinary troubleshooting.

The readiness review passed 44 Python tests and browser integration checks covering validation, retries, privacy behavior, and mobile layout. These checks use controlled responses: they verify application behavior, not the correctness of every model answer.

Why Does Open Innovation Matter?

Tech-support conversations can contain personal information: device details, filenames, and descriptions of what someone sees on their screen.

Gemma’s open weights and Ollama make it possible to run inference on the same computer that stores those conversations. After downloading dependencies and model weights, that setup can work without a hosted AI service or per-request API payments.

For a tool built for my mom, that control matters. I can inspect and change the prompts, adjust the response rules, and replace the model without rebuilding the application around a different provider.

A closed API could power a similar interface, but it would introduce an external inference dependency. Local inference gives Call Mac a useful alternative.

There are still boundaries: SQLite is unencrypted, and choosing a remote Ollama server sends conversation context to that server. Those details are documented.

Call Mac is an MVP. It accepts text, cannot inspect the device, and can give incorrect advice. My next priority is trying it with my mom and discovering which instructions still assume too much.

My Agent Session

I used Codex to review the repository, run automated checks, investigate the browser-demo deployment, and help draft this submission.

The session below contains curated excerpts from that final review. It does not include the earlier implementation history. Private machine paths and credentials were omitted.

Call Mac: challenge readiness review and submission writeup

You

is this repo ready to go? i also need a writeup for the challenge but it doesnt need to be commited here:

Agent

I’ll check the repo’s setup, demo workflow, and validation, then draft a challenge writeup you can paste into DEV. I’ll use the DevRelay challenge and community skills to check the submission requirements and inform the review.

Agent

The main release blocker is that nearly all the app files are still untracked; the committed repo currently contains only the README and agent instructions. The project fits the open-weight AI requirement, but the writeup needs a true account of one specific person it’s for. The deadline is October 5 at 2:59 AM Eastern.

You

My mom; she always needs help with technology and it is hard because I can only provide help over the phone

Agent

▸ exec_command python -m unittest discover -s tests -q

Input

$ python -m unittest discover -s tests -q

Output

Ran 44 tests in 8.169s
OK

▸ exec_command python tools/export_demo_contract.py --check

Input

$ python tools/export_demo_contract.py --check

Output

Demo prompt and schema match the Python app.

▸ exec_command python -m pip check

Input

$ python -m pip check

Output

No broken requirements found.

Agent

I see the app has now been committed and pushed, and the Pages deployment is running—that clears the missing-code blocker. The secret scan found no leaks in the publishable files. There’s no LICENSE file yet; I recommend choosing one so others have permission to reuse the code. The writeup will use the existing video link while Pages finishes deploying.

Agent

▸ exec_command python tests/browser_demo.py

Input

$ python tests/browser_demo.py

Output

Browser demo checks passed: sample, live API flow, validation, retries, privacy, safety, and mobile layout.

Agent

The repo is ready for a challenge submission using your video demo. The browser demo isn’t deployed yet. The Pages workflow passed its tests but failed because Pages isn’t enabled. In Settings → Pages → Source, select GitHub Actions, then rerun it. You can submit with the video without waiting for Pages. I also recommend adding a LICENSE. Live model quality wasn’t revalidated in this review.

View the review session.

Prize Categories

Best Use of Gemma

Gemma powers Call Mac’s contextual troubleshooting, follow-up questions, and explanations. Most of my experimentation used gemma2:9b; the app ships with gemma3:4b as its default.

Hello World!

Mackenzie — Tue, 16 Dec 2025 16:52:39 +0000

Hope to build exciting things here! Looking forward to learning with everyone here!