DEV Community: Systima

The Builder Test: 6 Questions to Evaluate a Fractional Head of AI

Systima — Mon, 23 Mar 2026 20:57:34 +0000

The fractional Head of AI market has split into two models:

brokers (firms that deploy generalists from a roster)
practitioners (specific people who have built and shipped AI systems in production).

Both have their place:

Brokers work for companies starting their AI journey. Practitioners are what you need when the architectural decisions have real consequences.

How to tell which you are getting

Six questions:

Have they personally built and shipped AI features in production? Not managed a team that did. Not advised. Personally built. If no, everything else is secondary.
Can they make architectural decisions, not just recommend them? "Implement RAG for customer support" is a roadmap. Choosing the embedding model, the chunking strategy, the retrieval evaluation framework; that is an architectural decision. A strategist writes the roadmap. A practitioner makes the decisions that determine whether the feature works.
Can they tell you what is wrong with your current architecture? Not "you should consider a vector database." Specifically: "your chunking strategy is producing poor retrieval because your documents have inconsistent heading structures, and your embedding model is not optimised for your domain vocabulary."
Can they evaluate vendors at the architectural level? Opening the technical documentation, assessing the API design and data residency implications, determining compatibility with your infrastructure. Not comparing feature matrices.
Do they know the difference between a demo and a production system? The gap between a POC and a system that handles edge cases, scales under load, and degrades gracefully is where most AI initiatives die.
If you are in a regulated industry, do they understand the regulatory landscape? For companies with EU AI Act exposure, architectural decisions now carry regulatory weight. For legal, fintech, or medtech, sector-specific regulators add further obligations.

The diagnostic questions to ask

Before you engage anyone:

Will I work directly with the decision-maker, or is there an intermediary layer?
Has this person shipped AI systems, or is their experience advisory?
Is the first deliverable an assessment of where I am, or a forward-looking strategy deck?
Can they evaluate my existing architecture, or only recommend new initiatives?

If the answers point to a broker, that is fine for early-stage AI guidance. If the decisions matter, you need the practitioner.

Full post here: https://systima.ai/blog/what-to-look-for-fractional-head-of-ai

Open-Source EU AI Act Compliance Scanning for CI/CD

Systima — Sat, 14 Mar 2026 14:14:44 +0000

We built a CLI tool that scans your codebase for EU AI Act compliance risks.

npx @systima/comply scan analyses your repository to detect AI framework usage, traces how AI outputs flow through the program, and flags patterns that may trigger regulatory obligations.

It runs in CI and posts findings on pull requests (no API keys required).

Under the hood it performs AST-based import detection using the TypeScript Compiler API and web-tree-sitter WASM across 37+ AI frameworks. It then traces AI return values through assignments and destructuring to identify four patterns:

conditional branching on AI output
persistence of AI output to a database
rendering AI output in a UI without disclosure
sending AI output to downstream APIs

Findings are severity-adjusted by system domain. You declare what your system does (customer support, credit scoring, legal research, etc) and the scanner adjusts accordingly.

Example:

a chatbot routing tool using AI output in an if statement produces an informational note
a credit scoring system doing the same produces a critical finding

We tested it against Vercel’s 20k-star AI chatbot repository; the scan took about 8 seconds. Example PR comment with full results: https://github.com/systima-ai/chatbot-comply-test/pull/1

Comply ships as an npm package, a GitHub Action (systima-ai/comply@v1), and a TypeScript API. It can also generate PDF reports and template compliance documentation.

Repo and explanation:
https://systima.ai/blog/systima-comply-eu-ai-act-compliance-scanning

Feedback welcome on the call-chain tracing approach and whether the domain-based severity model makes sense.

What Engineering Teams Actually Need to Do for EU AI Act Compliance

Systima — Fri, 06 Mar 2026 21:41:16 +0000

The EU AI Act entered into force on 1 August 2024. High-risk AI systems must comply by August 2026. Most guides to the Act are written by lawyers. This one is for the engineers who have to build the systems that generate compliance evidence.

The engineering problem

Articles 9 through 15 define six technical obligations for high-risk AI systems: risk management, data governance, technical documentation, record-keeping and logging, transparency, and human oversight, plus accuracy, robustness, and cybersecurity requirements.

These aren't abstract policy goals. They're engineering requirements that need to be designed, built, tested, and maintained.

What the guide covers

Risk classification: Mapping your system's function to Annex III categories with a documented, repeatable process
Technical documentation (Annex IV): What the documentation must contain and how to make it a first-class development artefact rather than an afterthought
Audit logging (Article 12): Structured, immutable logging of inputs, inference decisions, confidence scores, human overrides, and configuration changes
Conformity assessment: The self-assessment process for Annex VI and how documentation, logging, and risk management converge into auditable evidence

We've also open-sourced a reference implementation of Article 12 audit logging for the Vercel AI SDK: aiact-audit-log (https://github.com/systima-ai/aiact-audit-log).

Read the full guide

The full guide goes deeper into each area with architectural patterns, implementation considerations, and references to the relevant Articles and Annexes:

EU AI Act Engineering Compliance Guide (https://systima.ai/blog/eu-ai-act-engineering-compliance-guide)

The August 2026 deadline is approaching. If your team is deploying AI in regulated domains, the time to build compliance into your architecture is now.