DEV Community: Yurii Baranov

Find the Vulnerability Hiding Between the Features

Yurii Baranov — Wed, 28 Oct 2026 05:30:56 +0000

Security Scenario Explorer — a proposed security-testing direction built on Combinatorics Framework JVM.

Most security tools look for known weaknesses. But some vulnerabilities appear when individually valid requests interact in an unexpected order.

A permission changes while a session remains active. A retry arrives after a timeout. A webhook is delivered twice. An AI agent reads untrusted content while holding access to a privileged tool. Each operation may look legitimate in isolation; the sequence may still violate a security boundary.

Security Scenario Explorer is a proposed workbench for investigating these interaction-driven failures systematically. It builds on Combinatorics Framework JVM, a scenario-modeling and execution framework developed by Yurii B.

It is not positioned as a replacement for vulnerability scanners, DAST, or fuzzers. Its intended focus is security behavior that depends on combinations of context, state, action order, and exceptional events, therefore - reassures you even more in considered direction of potential risks.

Why investigate combinations?

Broken access control remains the top category in the OWASP Top 10:2025, which also includes security misconfiguration, authentication failures, insecure design, and mishandling of exceptional conditions. The OWASP API Security Top 10 highlights broken object-level authorization. For AI applications, the OWASP Top 10 for LLM Applications:2025 includes prompt injection and excessive agency.

These risks often require more than a single request to investigate. The relevant question is not only “Does this endpoint reject a bad input?” but also “Does the system preserve its security rules after this sequence of valid actions and unexpected events?”

Model the security question

Instead of starting only from a list of known vulnerability patterns, describe what can vary, what must remain true, and how much execution is acceptable:

Context: identities, roles, scopes, tenants, sessions, resource ownership, and configuration.
Action sequences: authenticate, create, read, share, grant, revoke, retry, and repeat—in the allowed orders.
Unexpected events: expired credentials, duplicate requests, interrupted workflows, and repeated webhook delivery at defined points in a sequence.
Security invariants: rules that must hold across every tested combination and ordering.

The framework plans the experiment before execution, reports candidate counts, and supports budgeted t-way selection using a greedy strategy; it does not claim to find a mathematically minimal test set. Candidates run through configured adapters. Verdicts distinguish application failures from harness or infrastructure problems, and failure triage can reduce a failing case to a small reproducer for verification and regression testing.

What could it investigate?

Authorization and tenant isolation. Can a user access another tenant’s resource after a role change, or does a stale session retain a capability that should have been revoked?

Authentication and session lifecycle. What happens when credentials expire or are revoked in the middle of a workflow?

Business-logic abuse. Can retries, replays, duplicate transactions, or unusual action ordering violate a financial or data-access invariant?

Configuration interactions. Can a combination of settings or feature flags make a restricted operation reachable?

AI-agent boundaries. Can untrusted content influence an agent to use a tool beyond its granted permissions or bypass an approval requirement?

Evidence, not labels

An AI assistant can help formulate threat hypotheses, build scenario models, interpret results, and suggest follow-up experiments. The human remains responsible for the authorized scope, the security properties being tested, and approval of consequential actions.

A finding should not be accepted because a tool or model calls it a vulnerability. It should be supported by a clearly stated invariant, an observed violation, and a reproducible test. Before trusting a security oracle, calibrate it with a known test defect and confirm that the test can detect the failure it is supposed to detect.

Current status and limitations

The underlying Combinatorics Framework JVM already supports context, sequence, and event modeling; constraints; plan and candidate counts; budgeted candidate selection; execution through configured adapters; verdict classification; failure-witness triage; checkpoints; and an AI-operator workflow with approval gates.

The security-specific layer is still planned, not demonstrated. That includes reusable security oracles, a scope guard, operating-system-level isolation, a concurrent-dispatch adapter, and ready-made scenario packs beginning with API authorization. Defined event injection covers chosen points in a sequence; it does not by itself test two requests arriving at precisely the same time.

There is also an important safety limitation today: generated Java runs inside the JVM without an operating-system sandbox. A security edition needs explicit scope declarations and target allow-lists, execution limits, timeouts, secret redaction, auditable run records, and isolation for generated code before it can be presented as safe for broader security-testing use.

The first proof

The first milestone should be a reproducible authorization or business-logic flaw that a single-request test misses. Publish the scenario, the invariant it violates, the observed result, and a minimal reproducer - then rerun it as a regression test.

The goal is not to claim that every attack can be enumerated. It is to make a defined security investigation more systematic, bounded, repeatable, and re-viewable.

Model what must remain secure. Explore the interactions that could break it. Preserve the evidence.

Security Scenario Explorer is a proposed direction, built on Combinatorics Framework JVM. If you work in AppSec, penetration testing, or security-minded QA, share a real cross-feature failure pattern that would benefit from a reproducible scenario model.

Combinatorics Framework - re-assuring safety behavior more, saving QA jobs as a test-space scenario designer-architect/supervising AI

Design full test-space scenarios at once and delegate precised evidence-based exploratory testing to your AI

Yurii Baranov — Mon, 26 Oct 2026 03:56:43 +0000

You Can’t List Your Unknown Bugs, but You Can Model What Varies.

Combinatorics-Framework-JVM turns a model of settings, action histories and optional events into planned test cases, and keeps the experiment inspectable from design through evidence.

⚡️ Straight to the point for most impatient/busy-reader who respects human's work (let your AI read 'AI_DELEGATION.md' after dowloading the release, and it will answer you live): Combinatorics-Framework-JVM

“What kinds of bugs does my system under test have?”

It sounds like the obvious first question for any testing tool. It’s also the wrong one. If you could answer it, you’d already have the tests.

The defects that hurt are the ones nobody knew to name: a guest, a coupon and a repeated payment request, each fine alone and wrong together. AI coding assistants sharpen the problem. An assistant can implement the feature, generate the unit tests and explain why the code should work. It can’t tell you which interactions those tests exercised.

So don’t start from bugs. Start from three questions you can answer:

What varies? Settings, inputs, the order of actions, events that may or may not happen.
What must stay true? An invariant, such as “the same payment request creates at most one charge.”
How much am I willing to run? A bounded budget of cases you can actually inspect.

Bugs aren’t the input to that exercise. They’re the output.

Investigation, not diagnosis

Combinatorics-Framework-JVM is built for exactly that kind of investigational testing: a Java workbench with a desktop app (guif), a command line and an embedding API. You describe the variations, it generates test candidates, runs them through your configured runners and helps you examine the results. State a model, run a bounded experiment, read the evidence, refine the model, repeat.

It won’t discover unknown defects for you. It turns a test idea into a visible model, a bounded run and results another engineer can inspect.

Model the context, and the history

Take a small checkout:

What varies	Choices
Customer	Guest, registered
Payment method	Card, wallet, cash on delivery
Channel	Website, API
Pricing policy	Current price, price captured earlier

That’s 24 static combinations. Run them all. Now add history: submit, time out, retry, a price change at different moments. The input values barely changed. The history did, and no parameter table shows it.

So the Framework models three kinds of variation separately:

Static context: account type, payment method, feature flags. Covering selection can reduce this part to a declared interaction strength.
Action sequences: what happens first, next and again, including order and repetition.
Sudden or optional events: a timeout, a duplicate request, a price change, including when it strikes and the case where it never does.

Sequences and sudden events are then crossed with every selected context, not squeezed into the same matrix. That stops you from flattening everything into interchangeable parameters and assuming you tested the histories that matter. For a workflow bug, the timeline deserves more attention than a big parameter matrix.

Constraints need the same care. An impossible combination can be excluded. A valid input the application should reject (cash on delivery over the API, say) deserves a negative test, not a deletion. The modeler decides which question a test answers. The generator can’t.

A model you can read in one minute

Two modes, three actions. Every candidate gets one mode and two different actions in a chosen order (AB, BA, AC, CA, BC, CB): 2 × 6 = 12 candidates. Save this as first-scenario.json:

{
  "spec_version": "1",
  "title": "Two modes and ordered pairs of actions",
  "slots": [
    {
      "sheet": "MODE",
      "alias": "choose_one",
      "raw": true,
      "values": ["safe", "fast"]
    },
    {
      "sheet": "OPS",
      "alias": "permute(2)",
      "flags": ["FW_Concatenator="],
      "raw": true,
      "values": ["A", "B", "C"]
    }
  ]
}

And these settings as first-scenario.properties:

cf.coverage.strength=0
cf.coverage.budget=0
cf.plan.autoPromoteSingleVerb=true

Then, with JDK 25 and fw-app.jar in your working folder:

java -Djava.awt.headless=true -jar fw-app.jar validate --scenario first-scenario.json --config first-scenario.properties
java -Djava.awt.headless=true -jar fw-app.jar plan --scenario first-scenario.json --config first-scenario.properties
java -Djava.awt.headless=true -jar fw-app.jar candidates --scenario first-scenario.json --config first-scenario.properties --limit 12

Validate, inspect the plan, read the candidates, all before anything runs. These are plain-text candidates for learning the model, not programs to execute.

Prefer spreadsheets? The classic workbook style (data sheets plus an FW_Seq control sheet) expresses the same model, and the wizard in guif guides a first scenario.

When order is the bug

A basic retry test submits a request twice with the same key and confirms that one order exists. Now change the history: the provider accepts the first request, its response times out, a confirmation arrives while the client is retrying, and the client tries again. Is there still exactly one charge and one order?

A small timeline model might include these histories:

History	Events the harness controls
A	Submit → success → retry the same request
B	Submit → timeout → retry → confirmation
C	Submit → timeout → confirmation → retry
D	Submit → timeout → retry → retry → confirmation

The invariant: reusing a request key must never create a second charge or a second order. Check durable business state (charges, orders, how they relate), not just the final HTTP status.

One honest boundary: the Framework generates ordered choices, permutations, repetitions and optional events. It doesn’t control a thread scheduler or a remote payment provider. Your adapter has to make those events controllable: hold and release a callback, simulate a timeout, reset state. Once it does, you vary the histories without hand-copying a test method for every combination.

A planted defect, caught

(quick self-test)
You don’t have to take this on faith. The project’s manual ships a self-contained checkout demo whose stand-in system has a deliberately planted lost-order defect. validate, plan and candidates report 72 generated cases, 24 removed by one constraint and 48 executable. The full run ends with 44 PASS and 4 DOMAIN_FAIL, with no harness failures. The repo’s validation report records this on JDK 25 and a 0.1.0-SNAPSHOT build.

A larger showcase in the manual runs 4,128 generated Cucumber scenarios against an in-JVM checkout engine and 714 browser scenarios (Selenium, headless Firefox) against a dummy web shop, both with planted defects hiding in action order and event timing. A second run of each design gave the same verdict for every case.

Two things to notice:

A run that finds defects still exits with code 0. A failure is a result, not a crash, so read the verdicts. Keep product failures (DOMAIN_FAIL) apart from harness outcomes such as BROKEN, TIMEOUT and INFRA_FAIL.
Every defect here was planted. That shows the harness can see a fault, not that it finds ones nobody knows about. It’s why the manual says to calibrate the oracle with a known-failing case before trusting a model. The unknown bugs are what the experiment is for.

What comes back

Failures arrive as cases, not red lights: the candidate, its verdict, its measurements and the context to reproduce it. Checkpoints and exports keep the evidence, and triage groups failures into findings with representative witnesses. The documented “minimal witness” is the failing case with the least source in its group: a good place to start, not a proven mathematical minimum. Values associated with elevated failure rates are leads, not causes.

When nothing fails, you still get something: “within this model, at this strength, no violation observed.” That isn’t proof of absence. It’s a claim another engineer can inspect, challenge and extend.

What you bring, and what it doesn’t promise

The Framework generates and organizes. You make it real. The model and the invariant are yours: it doesn’t discover what must stay true (a far easier question than “what is broken?”, but not a free one). You also bring:

An adapter that connects candidates to your system and controls its state and dependencies.
An oracle: assertions that encode the invariant. A generated case without a meaningful check is just activity.
A controlled environment with a reset strategy, plus the judgment about which failures matter and what went untested.

And be honest about the limits:

Pairwise isn’t magic. It covers pairs of modeled values, not every three-way defect, action order or timing failure, and the selection is greedy, so it never claims a minimum suite. With fewer than three varying factors, every pair is already the whole space and nothing is reduced. Its value is that you can state what you selected and inspect it.
One verdict per case. The first failing check names the verdict and hides later ones, so report every check as a metric too.
No OS sandbox for Java. Java cases run inside the Framework’s own JVM with your permissions, so treat every run as code execution on your machine: use a dedicated workspace and keep production out until the adapter, assertions and side effects are clear. Python cases run as separate processes and can be sandboxed with bwrap or Docker.

The same machinery, for AI experiments

Prompts × retrieval strategies × thresholds is a combinatorial space too. Three retrieval strategies, two prompts and two thresholds make 12 configurations. Evaluate each on 50 questions, twice: 1,200 evaluations, not 12. Mind that arithmetic when a provider bills per request.

Multi-goal analysis, including a Pareto front, keeps cost and quality trade-offs visible instead of collapsing them into one early score. It describes what you measured, not the best system that could exist. Bring an evaluation method you trust, because there is no universal judge of LLM answers, and test the evaluator with a deliberately wrong answer. Generation is deterministic; a remote model isn’t, so record versions, settings and rate limits.

Hand your assistant an experiment, not a vibe

The model and the recorded evidence are files, so a second engineer, or a later assistant session, can pick them up without reconstructing a chat. Give your assistant evidence, not a symptom:

Here is the failing candidate (submit → timeout → retry → confirmation) and the two charge IDs it returned. The invariant is one request key, at most one charge. Propose a mechanism that fits this evidence, a focused change, and which existing cases would distinguish your explanation from the alternatives.

That asks for a testable claim instead of a guessed fix. Then rerun the same candidate and its surrounding histories against the change: if a repair fixes one sequence and breaks another, the regression stays visible.

The repo is built for this hand-off. The manual, AI_DELEGATION.md, is written for an AI operator acting for a human supervisor, and the desktop’s AI_delegation button copies its path for you to paste. Copying a path isn’t consent. The manual’s first rule is no agreement, no operation: the assistant needs your explicit yes, then must ask before it finalizes the design, executes any case (a smoke run comes first), builds or installs anything, or touches the network, files outside the agreed workspace or your system under test. It must report what it did and didn’t check, and never present a plausible explanation as an observed result. You stay the director, and the license makes that a condition.

Try it small

Pick a behavior you already understand, ideally a defect you’ve already fixed.
Write three or four factors, one troublesome sequence and one invariant.
Stay small enough to preview every candidate, and change one thing at a time.
Inject a fault into a local implementation. Does the scenario catch it? Fix it and rerun.
Save the scenario, run ID, checkpoint and evidence.

If you can express the behavior, understand the generated cases and recognize a useful failure, it fits. The manual itself rates a few situations as poor fits: one or two parameters, purely continuous spaces, no reliable oracle, and state you can’t reset. Use unit or property tests, fuzzing or an optimizer there, and keep the plain unit test whenever it already pins a defect down. Reach for a scenario model when the variations themselves are getting hard to track.

Requirements, status and license

First of all: AS IS..., no warranty
Requirements: JDK 25, a full JDK with the compiler rather than a bare JRE, because generated cases are compiled in memory. The core workflow needs no Python, no relational database and no containers. Python 3.11+ and the Python add-on are optional, for Python candidates.
Status: first public release (Oct 2026), one maintainer. It’s early-stage, so validate it on your own workload before it carries anything critical.

The case for it is specific. Interacting settings, action sequences and timed events get one shared model, the model becomes inspectable candidates, and the results stay available for review and retesting. For the bugs that live between individually working features, and for AI-assisted work that needs evidence rather than confidence, that’s worth a small trial.

Don’t ask it where your bugs are. Tell it what varies and what must stay true, then read what comes back.

P.S. You are a Scenario Test-space Architect/Designer (puzzle-items of any kind) and a supervisor of AI (who operates on "Combinatorics Framework" as a precise tool for all of these)

🧩🧩🧩🧩🧩

GitHub project and downloads

🚀 Brand new testing approach with Combinatorics Framework

Yurii Baranov — Sat, 05 Sep 2026 15:16:11 +0000

Unit tests?
Ancient history (AI writes those for you).
Pairwise matrices?
Static parameters only.
How do you test dynamic, stateful scenario spaces in complex systems to be tested?
Meet Combinatorics Framework.

It’s a declarative scenario-space engine:

🧩 Compose: Higher-order dataflow algebra (permutations, subsets, joins of prior result tables)

⚡️ Inject: Mid-sequence rules, constraints, and optional interventions before execution

⚙️ Execute: Real candidate scenarios in sandboxed System-Under-Test environments

📊 Analyze: Multi-objective outcomes via Pareto-front evaluation

From AI-agent red-teaming and protocol resilience to stateful system validation—stop running static matrices. Start executing real scenario spaces.

Master the mindset: ZEN_OF_COMBINATORICS.

Combinatorics Framework by Yurii Baranov

▶ GitHub: https://github.com/ghYura/Combinatorics-Framework-QA-edition

▶ YouTube: https://www.youtube.com/@yuriibaranov

Combinatorics Framework
Beyond what AI can conceive, beyond what humans can execute alone

P.S. I care about QA jobs for humans. In a world going crazy over AI hype, this framework isn't built to replace testers—it’s built to elevate QA engineers into Scenario Architects who design execution spaces no plain AI can figure out on its own.

automation #testing #softwareengineering #qa #computerscience #aievals #devops