Built a CI for LLM workflows that runs on Ollama

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle

What I Built

Almost everyone is building something with LLMs these days. Chatbots, email writers, support assistants, you name it. But there is one problem nobody talks about: prompts break silently. You change one line, the model starts behaving differently, and you only find out when your client or user complains. For example: a freelancer edits a prompt on Friday evening, and by Monday the client's bot is writing "TODO" in customer emails. Not a good look.

So I built something useful for exactly this.

I created PromptCI, that is a CI tool for LLM workflows. You describe your workflow in a simple YAML file, keep your prompts in separate template files, and add checks on the output. Change a prompt, run one command, and it tells you if the behaviour drifted.

Who would pay for it? Freelancers and small agencies who build AI workflows for clients. Testing before shipping is something you can actually sell as a service, instead of just hoping nothing breaks :))

Demo

Well, no fancy website this time. It's a command line tool, so try it yourself rather than trusting my words:

pip install git+https://github.com/Soumyabrataop/promptci.git

Then run promptci init and promptci run. It works offline, so no API keys needed.

Code

Soumyabrataop / promptci

CI for LLM workflows: YAML pipelines, prompt templates, and regression gates — running on open-weight models, on your laptop.

PromptCI

Note

This is one of the very first, iteration of PromptCI versioned 0.1.0. Upcoming versions with features will be available soon.

CI for LLM workflows: YAML pipelines, prompt templates, and regression gates — running on open-weight models, on your laptop.

Prompts are code. They break silently. PromptCI treats them like code: describe your LLM workflow in YAML, keep prompts in versioned template files, and gate every change with assertions and output snapshots. Edit a prompt, run promptci run --frozen, and find out immediately whether the model's behavior drifted — instead of hearing about it from a user.

Built on open source end to end: PromptCI is MIT-licensed, and by default it talks to Ollama, so your workflows run on open-weight models (Llama, Qwen, Mistral, Gemma…) with no API keys, no cloud bills, and no data leaving your machine. It works fully offline — promptci init scaffolds…

View on GitHub

How I Built It

• Language: Python
• Workflows: YAML pipelines with Jinja2 prompt templates
• AI: Ollama, so it runs open-weight models like Llama, Qwen, Mistral and Gemma on your own laptop
• Checks: contains, regex, length, JSON schema, and an optional LLM judge that scores the answer
• Snapshots: every output is saved, and --frozen mode fails if anything has changed
• CI: clean exit codes, so it works in any CI including GitHub Actions

The whole test suite runs offline using a built-in mock model.

Why Does Open Innovation Matter?

Testing prompts on every change only makes sense if every run is free. With a paid API, that bill grows very fast. Open models made this possible:

• It's free to run. Test as many times as you want.
• Your data stays yours. Prompts often contain real customer text, and with local models nothing leaves your machine.
• No lock-in. Changing the model is just one line in the YAML.
• It works offline. Yes, even on a train.

For a testing tool, trust matters. Anyone can read the code, so "read the code" beats "trust us".

Team

Built by:

• @apparely

Now go test your prompts before your users do.