Beyond TOPS: Building an Open-Source Neuromorphic FPGA Benchmarking Suite

Johan Fernandes — Tue, 20 Oct 2026 10:21:21 +0000

Why traditional AI benchmarks fail for brain-inspired computing, and how we're fixing it with open-source tooling.

🧠 The Problem: Measuring a Sports Car with a Ruler

If you've been following AI hardware trends, you've probably seen benchmarks like MLPerf throwing around impressive numbers: "100 TOPS!" "1000 TOPS!" These metrics work great for traditional deep learning accelerators that crunch dense matrix multiplications.

But here's the thing: neuromorphic computing doesn't work like that.

Spiking Neural Networks (SNNs) are:

Sparse: Most neurons are idle most of the time
Event-driven: Computation only happens when spikes arrive
Asynchronous: No global clock ticking away

Measuring an SNN with TOPS (Tera Operations Per Second) is like measuring a hybrid car's fuel efficiency in gallons per hour instead of miles per gallon. The metric is technically valid, but it completely misses the point.

What we actually care about for neuromorphic hardware:

Spikes Per Second (SOPS): True event-driven throughput
Energy per Spike (pJ/spike): The holy grail for ultra-low-power edge AI
Neuron Density: How many neurons per LUT/FF?
Routing Efficiency: SNNs are highly interconnected; routing is often the real bottleneck

🔍 The Solution: NeuroMetric-FPGA

I've been working on an open-source benchmarking suite called NeuroMetric-FPGA that's specifically designed to evaluate neuromorphic FPGA designs using metrics that actually matter.

Architecture Overview

The suite has three modular layers:

workloads/: Standardized neuromorphic datasets (N-MNIST, DVS Gesture) converted to spike-train format
rtl/: Reference SNN implementations in Verilog (currently a baseline Leaky Integrate-and-Fire neuron array)
host/: Python telemetry engine that parses Vivado/Quartus reports and auto-generates benchmark reports

The key innovation? Automated parsing of FPGA toolchain reports to extract real hardware metrics, combined with neuromorphic-specific performance calculations.

📊 Real Results: ZedBoard Baseline

Let's look at some actual data from our baseline LIF (Leaky Integrate-and-Fire) neuron array implemented on a Xilinx Zynq-7020 (ZedBoard).

Key Insight: This ultra-lightweight design uses less than 1% of the FPGA's resources, leaving massive headroom for scaling up to larger neuron arrays or adding complex routing fabrics.

Critical Finding: Only 20.7% of the power is used for active computation. The rest is static leakage. This reveals a major optimization opportunity: clock gating and power gating could dramatically improve efficiency for event-driven workloads.

The difference between "total" and "dynamic" energy per spike is striking: 4.8x improvement when you account for only the active computation. This is exactly the kind of insight that traditional benchmarks miss.

🛠️ How It Works

The Python telemetry engine (benchmark.py) does the heavy lifting:

# Parse Vivado utilization report
lut_match = re.search(r'\|\s*Slice LUTs\s*\|\s*(\d+)\s*\|', content)
if lut_match:
    hw_metrics["LUT"] = int(lut_match.group(1))

# Parse power report
total_power_match = re.search(r'\|\s*Total On-Chip Power \(W\)\s*\|\s*([\d\.]+)', content)
if total_power_match:
    power_metrics["total_power_w"] = float(total_power_match.group(1))

# Calculate neuromorphic metrics
sops = total_spikes / time_sec
energy_per_spike_pj = (power_w * time_sec * 1e12) / total_spikes

The script automatically generates a clean Markdown report that's ready to be committed to GitHub and shared with the community.

🔮 What's Next?

This is just Part 1 of the project. Here's what's on the roadmap:

Phase 2: Real-World Workload Validation (Coming Soon)

A baseline is only as good as the data it processes. In the next iteration, we are upgrading the pipeline to handle real neuromorphic workloads:

AXI4-Stream Integration: Replacing the simple button/switch interface with a proper AXI4-Stream data path to ingest high-throughput event data.
N-MNIST / DVS Gesture Pipeline: Implementing a full classification pipeline to measure not just raw SOPS, but actual temporal classification accuracy vs. resource trade-offs.
Design Space Exploration: Scaling the neuron array from 8 to 64+ neurons to observe how routing congestion and dynamic power scale non-linearly.

Phase 3: Framework Integration (Lava & snnTorch)

Intel's Lava and frameworks like snnTorch are becoming the standard for software-side SNN development. We are exploring:

Using NeuroMetric-FPGA as a hardware validation backend for Lava models.
Automating the mapping of software-defined SNN layers to our parameterized FPGA RTL.
Cross-platform benchmarking: Comparing our FPGA metrics against CPU/GPU software simulations of the same network.

Phase 4: Community Contributions

We're actively seeking contributions for:

Intel Quartus support: Parse utilization and power reports from Altera/Intel tools
New datasets: DVS Gesture, SHD (Spiking Heidelberg Digits), CIFAR10-DVS
Larger FPGA boards: ZCU102, Alveo U250, Lattice iCE40
Advanced neuron models: Izhikevich, Hodgkin-Huxley

🤝 Join the Hacktoberfest Challenge!

This project is an official submission for the Hacktoberfest 2026 Open-Source AI Challenge, and we need your help!

Whether you're an FPGA engineer, ML researcher, or Python developer, there's something for you:

🟢 [good first issue]: Add a parser for Intel Quartus reports
🟢 [good first issue]: Convert a new dataset to spike-train format
🟡 [help wanted]: Implement AXI-Stream telemetry for high-throughput data transfer
🔵 [documentation]: Improve the API docs or write tutorials

Check out the repo: github.com/LiquidSilicion/NeuroMetric-FPGA

💡 Why This Matters

The neuromorphic computing community is growing rapidly, but we lack standardized, open-source benchmarking tools. Most research groups build their own custom evaluation scripts, making it nearly impossible to compare results across different papers and hardware platforms.

NeuroMetric-FPGA aims to change that by providing:

Standardized metrics that actually reflect neuromorphic efficiency
Automated report generation for reproducible research
Open-source tooling that anyone can extend and improve

If we can get the community to rally around a common benchmarking framework, we'll accelerate innovation in brain-inspired computing and make it easier to identify the most promising architectures for ultra-low-power edge AI.

📚 Resources

NeuroMetric-FPGA GitHub: github.com/LiquidSilicion/NeuroMetric-FPGA
NeuroBench: The foundational neuromorphic benchmarking framework
Intel Lava: Open-source neuromorphic software framework
Catalyst N1/N2: Open-source neuromorphic processors with Loihi parity

Built with ❤️ for the open-source hardware community. If you find this project useful, please give it a ⭐ on GitHub and share it with your fellow FPGA engineers!

Questions? Drop a comment below or open an issue on GitHub. Let's build the future of neuromorphic benchmarking together!

DEV Community: Johan Fernandes