This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle
This is my submission for the Hacktoberfest Open-Source AI Challenge — Week 2: Side Hustle.
What I Built
What if an instruction is technically correct but still dangerous to execute?
Consider this deployment instruction:
Reset the database, migrate the schema, and restore the latest backup if necessary.
It sounds straightforward. But which database should be reset? What does “latest” mean? When exactly is a restore necessary?
Different engineers—or AI agents—could interpret the same instruction differently and produce entirely different action sequences. In an infrastructure environment, that ambiguity can have serious consequences.
That's the problem I'm tackling with Semantic Chaos Monkey (SCM), an AI-assisted adversarial testing platform designed to identify ambiguity in operational procedures and AI-agent instructions.
Instead of merely checking whether an instruction reads well, SCM is designed to:
- Generate multiple plausible interpretations of an instruction.
- Validate proposed action sequences against a defined set of allowed operations.
- Simulate their consequences using deterministic, synthetic environments.
- Compare execution traces and identify meaningful differences.
- Suggest clearer instructions and retest them against the same safety rules.
The core idea is simple: test what an instruction could cause, not just what it appears to mean.
Who could benefit from it?
SCM is intended for DevOps engineers, AI-agent developers, platform engineering teams, and organizations that rely on operational runbooks.
For a startup, it could eventually help catch ambiguous deployment procedures before they reach production. For an AI engineering team, it could provide a reproducible way to test whether an agent's instructions leave room for conflicting interpretations.
The product is being designed with a local-first approach, keeping the initial testing environment isolated from real infrastructure.
Demo
Project status: Product definition and MVP development.
The complete end-to-end workflow is not yet verified as a working demo. The repository documents the proposed architecture, requirements, safety constraints, and implementation plan.
GitHub repository: https://github.com/HerambInamke/week-2-DEV-Challenge
I'll update this section with a live demo or walkthrough when the working MVP is ready.
Code
🔗 GitHub: https://github.com/HerambInamke/week-2-DEV-Challenge
The repository is released under the Apache-2.0 license.
The project structure includes:
-
client/— React and TypeScript frontend with Vite and Tailwind CSS. -
ai/— Python AI engine, schemas, fixtures, documentation, and tests. -
backend2/— Additional backend project code.
The intended architecture uses FastAPI and Pydantic for API contracts and validation, a replaceable model-provider interface for AI-generated interpretations, and a deterministic simulation engine for evaluating their consequences.
The focus is not just generating an AI response. It is making the reasoning outcomes testable, reproducible, and reviewable.
How I Built It
I approached SCM as an engineering problem rather than another general-purpose AI chatbot.
The proposed system follows a structured workflow.
1. Generate interpretations
Given an operational procedure, the AI layer proposes three to five plausible interpretations, including assumptions, source references, and ordered actions.
2. Validate the proposed actions
Each interpretation must conform to structured schemas, parameter requirements, preconditions, and a fixed catalog of permitted actions. Unsupported actions are rejected rather than executed through a fallback mechanism.
3. Simulate the consequences
The accepted action sequences are evaluated against identical initial states in a synthetic environment. Deterministic application logic handles the simulated state transitions.
4. Identify divergence
SCM is designed to compare action ordering, affected resources, resulting state, and configured safety invariants. This helps distinguish harmless wording differences from interpretations that could produce materially different outcomes.
5. Repair and retest
The system proposes a clearer version of the original instruction and evaluates the revision against the same configured tests. The intention is to verify whether the identified ambiguity has actually been addressed rather than simply assuming that the rewritten instruction is safe.
Technology stack
The proposed stack includes:
- Frontend: React, TypeScript, Vite, and Tailwind CSS.
- Backend and API: Python, FastAPI, and Pydantic.
- Testing: pytest and structured test fixtures.
- Storage: SQLite for the initial local MVP.
- AI integration: A replaceable model-provider interface.
One of the most important architectural decisions is separating AI-generated interpretations from their execution.
The model proposes possible meanings; deterministic application logic validates and simulates their consequences. The initial scope excludes production credentials, live infrastructure changes, unrestricted shell execution, and real deployment operations.
A language model suggesting an action sequence is not proof that the sequence is safe. SCM is being designed around that distinction.
Note: The architecture and workflow above describe the intended MVP. They should not be interpreted as a claim that every feature has already been implemented.
Why Does Open Innovation Matter?
Reliable AI systems should not be something only large companies with proprietary infrastructure can investigate.
Open innovation allows developers to inspect the implementation, challenge assumptions, contribute adversarial test cases, extend the simulator, and adapt the approach to their own engineering workflows.
For SCM, openness matters in three specific ways.
Transparency: Engineers should be able to inspect how proposed actions are validated, how simulated outcomes are compared, and which rules determine whether a test passes or fails.
Reproducibility: Saved test cases and execution traces can make it possible to review the same ambiguity without depending on another AI generation producing identical wording.
Flexibility: A replaceable model-provider interface is intended to support experimentation with different models without tying the entire platform to one vendor.
A closed AI API can generate interpretations, but that alone does not provide a transparent, deterministic testing environment tailored to a team's operational rules.
By developing SCM in the open, I want to make it possible for other engineers to challenge the design, contribute test scenarios, and help establish better practices for testing natural-language instructions.
My Agent Session
I'll add a shareable DevRelay agent-session link here once one is available.
Prize Categories
I'll update this section after confirming the applicable categories and eligibility requirements in the official challenge rules.
Final Thoughts
As AI agents become more involved in software development, infrastructure management, and operational workflows, the instructions we give them become increasingly important.
A procedure can look perfectly reasonable while leaving critical decisions undefined. Reviewing the wording alone will not always reveal the consequences of those missing details.
Semantic Chaos Monkey is my attempt to explore a more rigorous approach: turn ambiguous instructions into testable scenarios, compare their consequences, and verify proposed corrections.
The project is still taking shape, and there is plenty of engineering work ahead. I'm sharing it early because open-source feedback can help challenge the assumptions and improve the design before the system grows further.
If you're working on AI agents, DevOps, or software reliability, I'd welcome your feedback and contributions.
🔗 Explore the project: https://github.com/HerambInamke/week-2-DEV-Challenge
Top comments (0)