DEV Community: JT

Your Mac Is a Supercomputer. It's Time We Benchmarked It Like One.

JT — Thu, 05 Mar 2026 02:30:59 +0000

Why open source local AI benchmarking on Apple Silicon matters - and why your benchmark submission is more valuable than you think.

The narrative around AI has been almost entirely cloud-centric. You send a prompt to a data center somewhere, tokens come back, and you pretend not to think about what that costs in latency, money, or privacy. For a long time, that was the only game in town.

That's changing fast.

Apple Silicon - from the M1 to the M4 Pro/ Max shipping in machines today, with M5 Max on the horizon - has quietly become one of the most capable local AI compute platforms on the planet. The unified memory architecture means an M4 Max with 128GB of RAM can run models that would require a dedicated GPU workstation in any other form factor. At laptop wattages. Silently. Offline. Without sending a single token to a third party.

This isn't a niche enthusiast story anymore. It's a real shift in how developers, researchers, and privacy-conscious professionals are choosing to run AI workloads. And it comes with a problem we haven't solved yet: we don't have great, shared, community-driven data on how these machines actually perform in the wild.

That's what I built Anubis OSS to help fix.
App Page

The Fragmented Local LLM Ecosystem

If you've spent time running local models on macOS, you've felt this friction. The tooling is scattered and siloed:

Chat wrappers like Ollama, LM Studio, and Jan are excellent at what they do — conversation - but they're not built for systematic performance testing.
Hardware monitors like asitop, macmon, and mactop give you a beautiful CLI view of GPU and CPU utilization, but they have no concept of what the LLM is doing, which model is loaded, or what the prompt context size is.
Eval frameworks like promptfoo require YAML configs and terminal fluency that puts them out of reach for a lot of practitioners.

None of these tools correlate hardware behavior with inference performance in a meaningful, accessible way. You can watch your GPU spike during a generation pass, but you can't easily answer: Is Gemma 3 12B Q4_K_M more watt-efficient than Mistral Small 3.1 on an M3 Pro? How does TTFT scale with context length on an M4 with 32GB vs. 64GB? Which quantization gives the best tokens-per-watt on the Neural Engine?

Anubis answers those questions. It's a native SwiftUI app — no Electron, no Python runtime, no external dependencies — that runs benchmark sessions against any OpenAI-compatible backend (Ollama, LM Studio, mlx-lm, vLLM, and more) while simultaneously pulling real hardware telemetry via IOReport: GPU utilization, CPU utilization, GPU/CPU/ANE/DRAM power in watts, GPU frequency, process memory including Metal allocations, and thermal state.

Every run is logged, exportable as CSV or Markdown, and — optionally — submittable to the community leaderboard.

Why the Open Dataset Is the Real Story

The leaderboard submissions aren't just a scoreboard. They're the beginning of something more interesting: a real-world, community-sourced performance dataset across diverse Apple Silicon configurations, model families, quantizations, and backends.

Think about what that dataset contains:

Tokens per second across M1, M2, M3, M4, M5+ chips with varying unified memory configurations
Time to first token (TTFT) as a function of prompt length and model size
Watts-per-token efficiency across quantization levels (Q4, Q5, Q8, fp16) for the same model family
Backend-specific variance — the same model, same chip, different runner (Ollama vs. mlx-lm vs. LM Studio)
Thermal throttling behavior under sustained inference loads

This data is hard to get any other way. Formal benchmarks from chipmakers are synthetic. Reviewer benchmarks cover a handful of models on a handful of chips. Nobody has the time or hardware budget to run a comprehensive cross-product matrix.

But collectively, the community does.

How Model Runner Developers Can Use This

If you're building or maintaining a backend like Ollama, LM Studio, or mlx-lm, community benchmark data tells you things your internal testing can't:

Which chip/memory configurations are underperforming relative to their theoretical bandwidth? If an M3 Pro with 36GB is consistently underperforming an M2 Max with the same VRAM on a specific model class, that's a signal worth investigating in your memory management or Metal compute path.
Where is TTFT worst? Time to first token is often the user-perceived latency that matters most. If community data shows TTFT degrading sharply on longer contexts for certain quantizations, that's a tuning opportunity.
What's the real-world power envelope? Synthetic benchmarks don't capture sustained thermal behavior. Community submissions do.

How Model Tuners and Quantization Authors Can Use This

The dataset is equally valuable if you're working on GGUF quantizations, MLX conversions, or fine-tuned adapters:

Quantization efficiency curves across real hardware tell you where the quality/performance tradeoff actually lands for end users, not just on a benchmark server.
ANE utilization patterns — which quantization levels or architectures make better use of the Neural Engine — are nearly invisible without this kind of community telemetry at scale.
Memory footprint data (including Metal/GPU allocations tracked via proc_pid_rusage) shows whether your quantization is actually reducing real-world memory pressure or just parameter count.

If you're shipping a new quantization of a popular model and want to understand how it performs across the installed base of Apple Silicon hardware your users actually own, this dataset is the closest thing to field telemetry you'll have access to.

The Apple Silicon Trajectory Makes This Urgent

M5 is coming. The M4 Ultra hasn't even shipped in the Mac Pro yet. The memory ceiling on Apple Silicon keeps rising — 192GB on the current M4 Max configuration means models that were cloud-only a year ago can run locally today.

Each generation, the gap between "what you can run locally" and "what you need the cloud for" narrows. We're already past the inflection point for most 7B–13B models. We're approaching it for 30B–70B classes on high-end configs. The 100B+ frontier is a matter of time and memory density.

The decisions that model runners and quantization authors make right now - about memory management, about Metal optimization, about ANE scheduling — will determine how well that new hardware gets utilized by the community. Having a community dataset that shows real-world performance gaps is one of the best early-warning systems we have.

Why You Should Participate

Running a benchmark in Anubis takes about two minutes. Submitting it to the leaderboard takes one click. But here's why it's worth your time beyond the leaderboard:

Your hardware configuration is probably underrepresented. The M4 Pro with 48GB, the M2 Max with 96GB, the M3 Ultra — the matrix of chip × memory × thermal environment × backend is enormous. Every submission fills in a cell of that matrix that nobody else may have covered.

Your workload patterns matter. If you're running long-context inference, or using a backend that others aren't, or benchmarking a model that just dropped — that data is genuinely novel and useful to the ecosystem.

The dataset is open. This isn't data that disappears into a corporate analytics pipeline. It's a community resource, available for anyone building tools, writing research, or trying to optimize for the platform.

The project needs the stars. Anubis OSS is working toward 75 GitHub stars to qualify for Homebrew Cask distribution, which would make it dramatically easier for people to install and run. If you find value in what the project is doing, a star is a genuinely meaningful contribution to its reach.

Get Started

Download Anubis OSS from the latest GitHub release — it's a notarized macOS app, no build required
Run a benchmark — load any model in your preferred backend, pick a prompt preset, hit run
Submit your results to the community leaderboard
Star the repo at github.com/uncSoft/anubis-oss to help us hit Homebrew distribution

The local AI era is here. Let's build the shared infrastructure to understand it.

Anubis OSS is GPL-3.0 licensed. Built in Swift, no external dependencies, privacy-first — your benchmark data is submitted voluntarily and never includes anything beyond hardware specs and model performance metrics. A limited version is also available as part of The Architect's Toolkit bundle on the Mac App Store.

Questions or contributions? Open an issue or PR on GitHub.

Slidepad is cool but..

JT — Thu, 05 Mar 2026 02:15:38 +0000

What if it was half the price and had double the features, and was the available in the apple app store? I gotchu..
DevPad

Open Source Local LL Benchmarking and Leaderboards on MacOS

JT — Wed, 04 Mar 2026 23:11:18 +0000

Anubis

Local LLM Testing & Benchmarking for Apple Silicon | Community Leaderboard
Github

Anubis is a native macOS app for benchmarking, comparing, and managing local large language models using any OpenAI-compatible endpoint - Ollama, MLX, LM Studio Server, OpenWebUI, Docker Models, etc. Built with SwiftUI for Apple Silicon, it provides real-time hardware telemetry correlated with full, history-saved inference performance - something no CLI tool or chat wrapper offers. Export benchmarks directly without having to screenshot, and export the raw data as .MD or .CSV from the history. You can even OLLAMA PULL models directly within the app. The binary is signed with an Apple Dev certificate. We are trying to get to 75 stars to distribute as a Cask via Homebrew. Please consider submitting your results to the leadersboards, the dataset is also open-source, and we are trying to use the data with researchers to enhance how models are trained and perform.

Why Anubis?

The local LLM ecosystem on macOS is fragmented:

Chat wrappers (Ollama, LM Studio, Jan) focus on conversation, not systematic testing
Performance monitors (asitop, macmon, mactop) are CLI-only and lack LLM context
Evaluation frameworks (promptfoo) require YAML configs and terminal expertise
No tool correlates hardware metrics (GPU / CPU / ANE / power / memory) with inference speed in real time

Anubis fills that gap with three integrated modules - all in a native macOS app.

Leaderboard Submissions Now Available! Submit directly through the app

The dataset is robut and open source - check it out here, please contribute!

Features

Benchmark

Real-time performance dashboard for single-model testing.

Select any model from any configured backend
Stream responses with live metrics overlay
8 metric cards: Tokens/sec, GPU %, CPU %, Time to First Token, Process Memory, Model Memory, Thermal State, GPU Frequency
7 live charts: Tokens/sec, GPU utilization, CPU utilization, process memory, GPU/CPU/ANE/DRAM power, GPU frequency - all updating in real time
Power telemetry: Real-time GPU, CPU, ANE, and DRAM power consumption in watts via IOReport
Process monitoring: Auto-detects backend process by port (Ollama, LM Studio, mlx-lm, vLLM, etc.) with manual process picker
Detailed session stats: peak tokens/sec, average token latency, model load time, context length, eval duration, power averages
Configurable parameters: temperature, top-p, max tokens, system prompt
Prompt presets organized by category (Quick, Reasoning, Coding, Creative, Benchmarking)
Session history with full replay, CSV export, and Markdown reports
Expanded full-screen metrics dashboard
Image export: Copy to clipboard, save as PNG, or share - 2x retina rendering with watermark, respects light/dark mode

Arena

Side-by-side A/B model comparison with the same prompt.

Dual model selectors with independent backend selection
Sequential mode (memory-safe, one at a time) or Parallel mode (both simultaneously)
Shared prompt, system prompt, and generation parameters
Real-time streaming in both panels
Voting system: pick Model A, Model B, or Tie - votes are persisted
Per-panel stats grid (9 metrics each)
Model manager: view loaded models and unload to free memory
Comparison history with voting records

Leaderboard (New in 2.1)

Upload your benchmark results to the community leaderboard and see how your Mac stacks up against other Apple Silicon machines.

One-click upload from the benchmark toolbar after a completed run
Community rankings sorted by tokens/sec with full drill-down into performance, power, and hardware details
Filter by chip or model to compare like-for-like (e.g. all M4 Max results, or all Llama 3.2 runs)
Data Explorer — interactive pivot table and charting powered by FINOS Perspective
Privacy-first: no accounts, no response text uploaded — just metrics and a display name
HMAC-signed submissions with server-side rate limiting

Auto-Update (New in 2.3)

Anubis checks for updates automatically via Sparkle and notifies you when a new version is available.

Automatic checks on launch with user-controlled frequency
Manual check via the app menu (Anubis OSS > Check for Updates...) or Settings > About
Updates are code-signed, notarized, and verified with EdDSA before installation

Vault

Unified model management across all backends.

Aggregated model list with search and backend filter chips
Running models section with live VRAM usage
Model inspector: size, parameters, quantization, family, context window, architecture details, file path
Automatic metadata enrichment for OpenAI-compatible models - parses model IDs for family and parameter count, scans ~/.lmstudio/models/ and ~/.cache/huggingface/hub/ for disk size, quantization, and path
Pull new models, delete existing ones, unload from memory
Popular model suggestions for quick setup
Total disk usage display

Screenshots

GPU Core detail

Arena Mode

Settings (add connections with quick presets)

Vault - View model details, unload, and Pull models directly for Ollama

Supported Backends

Backend	Type	Default Port	Setup
Ollama	Native support	11434	Install from ollama.com - auto-detected on launch
LM Studio	OpenAI-compatible	1234	Enable local server in LM Studio settings
mlx-lm	OpenAI-compatible	8080	`pip install mlx-lm && mlx_lm.server --model <model>`
vLLM	OpenAI-compatible	8000	Add in Settings
LocalAI	OpenAI-compatible	8080	Add in Settings
Docker ModelRunner	OpenAI-compatible	user selected	Add in Settings

Any OpenAI-compatible server can be added through Settings > Add OpenAI-Compatible Server with a name, URL, and optional API key.

Hardware Metrics

Anubis captures Apple Silicon telemetry during inference via IOReport and system APIs:

Metric	Source	Description
GPU Utilization	IOReport	GPU active residency percentage
CPU Utilization	`host_processor_info`	Usage across all cores
GPU Power	IOReport Energy Model	GPU power consumption in watts
CPU Power	IOReport Energy Model	CPU (E-cores + P-cores) power in watts
ANE Power	IOReport Energy Model	Neural Engine power consumption
DRAM Power	IOReport Energy Model	Memory subsystem power
GPU Frequency	IOReport GPU Stats	Weighted average from P-state residency
Process Memory	`proc_pid_rusage`	Backend process `phys_footprint` (includes Metal/GPU allocations)
Thermal State	`ProcessInfo.thermalState`	System thermal pressure level

Process Monitoring

Anubis automatically detects which process is serving your model:

Port-based detection: Uses lsof to find the PID listening on the inference port (called once per benchmark start)
Backend identification: Matches process path and command-line args to identify Ollama, LM Studio, mlx-lm, vLLM, LocalAI, llama.cpp
Memory accounting: Uses phys_footprint (same as Activity Monitor) which includes Metal/GPU buffer allocations - critical for MLX and other GPU-accelerated backends
LM Studio support: Walks Electron app bundle descendants to find the model-serving process
Manual override: Process picker lets you select any process by name, sorted by memory usage

Metrics degrade gracefully - if IOReport access is unavailable (e.g., in a VM), Anubis still shows inference-derived metrics.

Requirements

macOS 15.0 (Sequoia) or later
Apple Silicon (M1 / M2 / M3 / M4 / M5 +) - Intel is not supported
8 GB unified memory minimum (16 GB+ recommended for larger models)
At least one inference backend installed (Ollama recommended)

Getting Started

1. Install Ollama (or another backend)

# macOS - install Ollama
brew install ollama

# Start the server
ollama serve

# Pull a model
ollama pull llama3.2:3b

2. Build & Run Anubis

git clone https://github.com/uncSoft/anubis-oss.git
cd anubis-oss/anubis
open anubis.xcodeproj

In Xcode:

Set your development team in Signing & Capabilities
Build and run (Cmd+R)

Anubis will auto-detect Ollama on launch. Other backends can be added in Settings.

3. Run Your First Benchmark

Select a model from the dropdown
Type a prompt or pick one from Presets
Click Run
Watch the metrics light up in real time

4. Submit to the Leaderboard

After a benchmark completes, click the Upload button in the benchmark toolbar to submit your results to the community leaderboard. Enter a display name and your run will appear in the rankings — no account required. Only performance metrics and hardware info are submitted; response text is never uploaded.

Building from Source

# Clone
git clone https://github.com/uncSoft/anubis-oss.git
cd anubis-oss/anubis

# Build via command line
xcodebuild -scheme anubis-oss -configuration Debug build

# Run tests
xcodebuild -scheme anubis-oss -configuration Debug test

# Or just open in Xcode
open anubis.xcodeproj

Dependencies

Resolved automatically by Swift Package Manager on first build:

Package	Purpose	License
GRDB.swift	SQLite database	MIT
Sparkle	Auto-update framework	MIT
Swift Charts	Data visualization	Apple

Architecture

Anubis follows MVVM with a layered service architecture:

┌─────────────────────────────────────────────────────────────┐
│                    PRESENTATION LAYER                       │
│   BenchmarkView    ArenaView    VaultView    SettingsView   │
├─────────────────────────────────────────────────────────────┤
│                      SERVICE LAYER                          │
│   MetricsService   InferenceService   ModelService   Export │
├─────────────────────────────────────────────────────────────┤
│                    INTEGRATION LAYER                        │
│  OllamaClient  OpenAICompatibleClient  IOReportBridge  ProcessMonitor │
├─────────────────────────────────────────────────────────────┤
│                    PERSISTENCE LAYER                        │
│   SQLite (GRDB)              File System                    │
└─────────────────────────────────────────────────────────────┘

Views display data and delegate to ViewModels. ViewModels coordinate Services. Services are stateless and use async/await. Integrations are thin adapters wrapping external systems (Ollama API, IOReport, etc.).

Project Structure

anubis/
├── App/                    # Entry point, app state, navigation
├── Features/
│   ├── Benchmark/          # Performance dashboard
│   ├── Arena/              # A/B model comparison
│   ├── Vault/              # Model management
│   └── Settings/           # Backend config, about, help, contact
├── Services/               # MetricsService, InferenceService, ExportService
├── Integrations/           # OllamaClient, OpenAICompatibleClient, IOReportBridge, ProcessMonitor
├── Models/                 # Data models (BenchmarkSession, ModelInfo, etc.)
├── Database/               # GRDB setup & migrations
├── DesignSystem/           # Theme, colors, reusable components
├── Demo/                   # Demo mode for App Store review
└── Utilities/              # Formatters, constants, logger

Backend Abstraction

All inference backends implement a shared protocol, making it straightforward to add new ones:

protocol InferenceBackend {
    var id: String { get }
    var displayName: String { get }
    var isAvailable: Bool { get async }

    func listModels() async throws -> [ModelInfo]
    func generate(prompt: String, parameters: GenerationParameters)
        -> AsyncThrowingStream<InferenceChunk, Error>
}

Data Storage

All data is stored locally - nothing leaves your machine.

Data	Location
Database	`~/Library/Application Support/Anubis/anubis.db`
Exports	Generated on demand (CSV, Markdown)
Preferences	UserDefaults

Troubleshooting

Ollama shows "Disconnected"

# Make sure Ollama is running
ollama serve

# Verify it's accessible
curl http://localhost:11434/api/tags

No GPU metrics

GPU metrics require IOReport access via IOKit
Some configurations or VMs may not expose these APIs
Anubis will still show inference-derived metrics (tokens/sec, TTFT, etc.)

High memory usage

Use Sequential mode in Arena to run one model at a time
Unload unused models via Arena > Models > Unload All
Choose smaller quantized models (Q4_K_M over Q8_0)

Model not appearing

Click Refresh Models in Settings
Ensure the model is pulled: ollama pull <model-name>
For OpenAI-compatible backends, verify the server is running and the URL is correct

Contributing

Contributions are welcome. A few guidelines:

Follow the existing patterns - MVVM, async/await, guard-let over force-unwrap
Keep files under 300 lines - split if larger
One feature per PR - small, focused changes are easier to review
Test services and integrations - views are harder to unit test, but services should have coverage
Handle errors gracefully - always provide errorDescription and recoverySuggestion

Adding a New Backend

Create a new file in Integrations/ implementing InferenceBackend
Register it in InferenceService
Add configuration UI in Settings/
That's it - the rest of the app works through the protocol

Support the Project

If Anubis is useful to you, consider buying me a coffee on Ko-fi or sponsoring on GitHub. It helps fund continued development and new features.

A sandboxed, less feature rich version is also available on the Mac App Store if you prefer a managed install.

License

GPL-3.0 License — see LICENSE for details.

Other projects: DevPad · Nabu