DEV Community: Roman Chudov

ChaosKit: Code-Level Chaos Engineering framework for Go

Roman Chudov — Fri, 28 Nov 2025 03:50:13 +0000

Chaos engineering tools usually target infrastructure: pods, nodes, networks, CPU pressure. But many real failures happen inside the application logic - in retries, compensations, concurrency, or state transitions.

Infrastructure tools can’t simulate a panic inside a function, a 15ms delay at the wrong moment, or an internal invariant break.

ChaosKit solves that by bringing chaos engineering directly into Go code.

GitHub: https://github.com/rom8726/chaoskit

Why Code-Level Chaos Matters

While building the workflow engine, it became clear that most critical failures were logic-level, not cluster-level:

a compensation step panics;
a goroutine leak accumulates over time;
a tiny delay causes a race condition.

ChaosKit is designed to expose these failures early - inside your unit and integration tests.

How ChaosKit Works

ChaosKit provides controlled "chaos points" you can insert into your code:

chaoskit.MaybeDelay(ctx)
chaoskit.MaybeError(ctx)
chaoskit.MaybePanic(ctx)

By default, these are no-ops.

Chaos injectors activate only when:

the binary is built with -tags=chaos, and
you attach a chaos context:

   ctx := chaoskit.AttachChaos(context.Background())

This ensures that chaos never triggers accidentally in production.

Injectors, Validators, Scenarios

ChaosKit consists of three core building blocks:

Injectors

Simulate failures:

delays
random errors
panic injection
ToxiProxy-based network faults
custom logic

Example:

Inject("latency", injectors.RandomDelay(5*time.Millisecond, 20*time.Millisecond))

Validators

Check invariants after each step:

goroutine count
recursion depth
no infinite loops
custom validation

Example:

Assert("goroutines", validators.GoroutineLimit(100))

Scenarios

Define steps, injectors, validators, and repetition:

scenario := chaoskit.NewScenario("workflow").
    WithTarget(engine).
    Step("run", ExecuteWorkflow).
    Inject("panic", injectors.PanicWithProbability(0.15)).
    Assert("limit", validators.GoroutineLimit(100)).
    Repeat(50).
    Build()

Chaos Inside `go test`

ChaosKit integrates directly with Go’s testing package:

chaostest.RunChaos(t, "workflow", target, func(s *chaoskit.ScenarioBuilder) *chaoskit.ScenarioBuilder {
    return s.
        Step("inc", func(ctx context.Context, target chaoskit.Target) error {
            chaoskit.MaybeDelay(ctx)
            target.(*TestTarget).Increment()
            return nil
        }).
        Inject("delay", injectors.RandomDelay(5*time.Millisecond, 20*time.Millisecond))
},
    chaostest.WithRepeat(5),
    chaostest.WithDefaultThresholds(), // 95% success required
)

This allows developers to run chaos experiments during normal test runs, without extra infrastructure.

Useful for:

CI-friendly probabilistic chaos tests
invariant checks under failure
catching race conditions and leaks
validating retry/compensation logic

Runtime behavior

ChaosKit is engineered to avoid accidental activation:

Build tag isolation (\-tags=chaos) Chaos code is excluded from production binaries.
Explicit chaos context Without AttachChaos, everything is a no-op.
Network chaos is external ToxiProxy never touches production traffic.
Monkey patching disabled by default Used only in tests, only if explicitly enabled.

When ChaosKit Is Valuable

Great fit for:

workflow/Saga engines
stateful services
concurrency-heavy components
libraries with retry logic
business logic with invariants

Not ideal for:

pure infrastructure chaos (use Chaos Mesh, Litmus)
tests that can’t integrate Maybe* calls

Conclusion

ChaosKit focuses on the part of the system that is hardest to test and easiest to break: your code’s internal failure paths.

If your Go project relies on workflows, compensations, or complex state transitions, ChaosKit helps ensure correctness under unpredictable conditions - without requiring Kubernetes, Docker, or cluster chaos tools.

Repo: https://github.com/rom8726/chaoskit

Floxy — a lightweight workflow engine for GoLang with saga-style compensation

Roman Chudov — Thu, 23 Oct 2025 04:10:47 +0000

Floxy — a lightweight workflow engine for Go with saga-style compensation

When building distributed systems or long-running business processes, we often need workflows that can retry, rollback, and recover — without introducing the complexity of Temporal.

That’s where Floxy fits in.

What is Floxy?

Floxy is a lightweight workflow engine for Go.

It provides a simple builder API and a deterministic runtime that supports saga-style compensation, save points, and idempotency control — all without external dependencies or background daemons.

It’s designed to be small, composable, and transparent.

Key features

Deterministic state machine execution
Retry and compensation policies
Partial rollback via save points
Idempotency control (WithStepNoIdempotent)
Parallel, fork, and join support
Event-driven persistence for observability

Example

wf := floxy.NewBuilder("order_saga", 1).
    Step("reserve_funds", "ReserveFunds").
        OnFailure("refund_funds", "RefundFunds").
    Step("ship_order", "ShipOrder").
        OnFailure("cancel_shipping", "CancelShipping").
    Step("notify_user", "Notify").
    Build()

If ship_order fails:

Floxy marks it as failed.
Executes its compensation handler cancel_shipping.
Rolls back previous steps (refund_funds).
Marks the workflow as failed deterministically.

Retry and Idempotency

In Floxy, MaxRetries defines the total number of allowed handler calls, including the first one.

MaxRetries = 1 → one call only (no retries).
MaxRetries = 3 → up to three total executions.

By default, steps are idempotent.
To mark a step as non-idempotent, use:

go Step("charge_card", "ChargeCard", floxy.WithStepNoIdempotent())

This ensures the handler runs exactly once, even if it fails.

Deterministic State Model

Floxy models workflows as a finite state machine:

pending → running → completed ↓ failed → compensation → rolled_back

Each state transition is explicit and persisted in storage, which makes recovery and auditing reliable and predictable.

When to use Floxy

Floxy is ideal for:

Transactional workflows with rollback logic (saga pattern)
ETL or provisioning pipelines
Payment, booking, and inventory systems
Temporal-like semantics without Temporal’s infrastructure

Learn more

GitHub: github.com/rom8726/floxy

The repository includes a detailed docs/ENGINE_SPEC.md describing the internal runtime model, retry semantics, and compensation flow.

If you find Floxy useful, give it a star and share feedback.

Floxy: deterministic workflows for real systems, built the Go way.

Goodbye, Sentry!

Roman Chudov — Wed, 23 Jul 2025 09:51:36 +0000

Introducing Warden: A Self-Hosted Error Monitoring Platform

Roman Chudov ・ Jul 21

#programming #monitoring

[Boost]

Roman Chudov — Tue, 22 Jul 2025 20:42:12 +0000

Roman Chudov

Jul 21 '25

Introducing Warden: A Self-Hosted Error Monitoring Platform

#programming #monitoring

Comments

1 min read

Introducing Warden: A Self-Hosted Error Monitoring Platform

Roman Chudov — Mon, 21 Jul 2025 13:29:15 +0000

Today I’m releasing the first version of Warden, a self-hosted error tracking and analytics platform designed as a private, Sentry-compatible alternative.

Website: https://warden-project.tech

What is Warden?

Warden helps developers and teams monitor, analyze, and respond to application errors in real time. It is designed for companies that require full control over their observability stack and need to operate in isolated or secure environments.

Warden is compatible with Sentry SDKs, making integration seamless with minimal changes to existing applications.

Core Features

Error tracking using Sentry-compatible protocols
Two-Factor Authentication (2FA)
Role-Based Access Control (RBAC) with team and project isolation
Release analytics (new issues, regressions, fix time statistics)
Notification system supporting Email and Telegram
Admin dashboard with superuser controls
LDAP integration for user and group synchronization
SAML SSO support (including Keycloak)
Rate limiting

Why Warden?

Warden is built for teams that cannot rely on cloud-based SaaS platforms due to compliance, legal, or security constraints. It provides observability infrastructure that runs entirely under your control, without external dependencies.

What’s Next

Planned improvements include:

Developer assignments and resolution workflows
GitHub and GitLab integration
Advanced analytics dashboards
Multi-tenant support

Learn more at https://warden-project.tech

[Boost]

Roman Chudov — Thu, 10 Jul 2025 13:15:04 +0000

Roman Chudov

Jul 10 '25

testy: YAML-Based Functional Tests for Go HTTP APIs

Comments

3 min read

testy: YAML-Based Functional Tests for Go HTTP APIs

Roman Chudov — Thu, 10 Jul 2025 12:43:52 +0000

If you've ever written functional tests for a Go HTTP server, you’ve probably found yourself stuck between two extremes:

Writing black-box tests with curl or Postman clones (slow, no debug, hard to maintain)
Writing verbose Go test code for every API call, mocking, assertions, cleanup, etc.

That’s why I created testy — a declarative testing framework for Go HTTP APIs, built on top of YAML and your actual http.Handler.

Why `testy`?

Here’s what makes testy different:

Declarative tests written in plain YAML
Real http.Handler execution — your actual app code, not a subprocess
Full PostgreSQL fixture support via pgfixtures
Powerful mocking, assertions, and SQL checks
Great for debugging and stepping through real code

Debug Your Tests Like Real Code

Because testy runs requests through your actual Go http.Handler, you can set breakpoints in your API handlers and run tests in your IDE’s debug mode (go test -v -run <TestName>).

This makes test runs feel like regular request handling — not like opaque black-box e2e scripts.

You don’t need to spin up your app on a port. testy hits your router directly — whether it’s from net/http, Gin, Chi, Echo or anything else.

Declarative Testing With YAML

Test scenarios are written as YAML lists of steps:

- name: get_user
  request:
    method: GET
    path: /users/123
  response:
    status: 200
    headers:
      Content-Type: application/json
    json: |
      {
        "id": 123,
        "name": "Alice"
      }

No Go code. No boilerplate. Just requests and expectations.

Dynamic Placeholders and Context

testy supports smart templating:

{{step.response.field}} — reference values from earlier steps
{{ENV_VAR}} — use environment variables (like UUID, tokens, etc.)
Used in URLs, headers, bodies, SQL queries, everywhere

Example:

- name: get user
  request:
    path: /users/{{create_user.response.id}}

Works with Your Real Code

A minimal Go test using testy looks like this:

testy.Run(t, &testy.Config{
  Handler:     api.Router(),            // your real app router
  ConnStr:     os.Getenv("TEST_DB"),    // PostgreSQL
  CasesDir:    "./tests/cases",         // YAML test cases
  FixturesDir: "./tests/fixtures",      // pgfixtures
})

That’s it.

You can run it with:

go test ./...

And debug it with your IDE as usual. Put a breakpoint in your handler — it just works.

Real-Life Example: Creating a Project

From create.yml:

- name: create project with team
  fixtures:
    - empty_db
    - developers_team

  steps:
    - name: login
      request:
        method: POST
        path: /auth
        headers:
          Content-Type: application/json
        body: {"username": "admin", "password": "Warden123!"}
      response:
        status: 200
        headers:
          Content-Type: application/json

    - name: add_project
      request:
        method: POST
        path: /projects/add
        headers:
          Authorization: "Bearer {{login.response.access_token}}"
          Content-Type: application/json
        body:
          name: "Project With Team"
          team_id: 1
      response:
        status: 201
      dbChecks:
        - query: SELECT name FROM projects WHERE team_id = 1
          result:
            - name: "Project With Team"

No test logic in Go. It’s all in the scenario.

Powered by `pgfixtures`

Each scenario can load fixtures into the DB before it starts — using pgfixtures.

Fixtures are described in YAML, and support:

Truncating tables
Resetting sequences
$eval(SELECT ...) expressions

public.users:
  - id: 1
    name: "Alice"
    created_at: $eval(SELECT NOW())

Mocks for Outgoing Requests

You can define HTTP mocks directly in the test file:

mockServers:
  notification:
    routes:
      - method: POST
        path: /send
        response:
          status: 202
          headers:
            Content-Type: application/json
          json: '{"status":"queued"}'

mockCalls:
  - mock: notification
    count: 1
    expect:
      method: POST
      path: /send
      body:
        contains: "Joseph"

Mocks are verified and auto-spun for you. No third-party servers needed.

Highlights

Pure YAML: easily readable, editable by devs and QA alike
Real code path: debug API as usual — same context, same stacktrace
Built-in mocks: test integrations and verify outbound calls
DB fixtures & assertions: validate everything inside and out
Fast, IDE-friendly, minimal

Perfect for:

Testing REST API behavior end-to-end
Validating JSON responses and DB state
Reproducing edge cases and regression bugs
Debugging logic by stepping through real code paths

Try It Out

go get github.com/rom8726/testy@latest

Set up your test directory like:

/tests
  ├─ cases/     # YAML test scenarios
  ├─ fixtures/  # YAML DB fixtures

Then run go test — and debug away.

Open Source

testy is Apache-2.0 licensed and open to contributions.
If you like writing tests like a human, check it out:

GitHub: github.com/rom8726/testy

Ansible Playbook for PostgreSQL Patroni Cluster

Roman Chudov — Wed, 09 Jul 2025 03:56:03 +0000

I've recently published an Ansible playbook for deploying a PostgreSQL high-availability cluster powered by Patroni, with all the essential components included. It's a ready-to-go setup for development and testing environments.

🔧 What’s inside:

PostgreSQL 16 with Patroni for automatic failover
etcd for distributed state management
PgBouncer for connection pooling
HAProxy for routing between master and replicas
Keepalived for floating Virtual IP
Node Exporter + Postgres Exporter for monitoring

🧩 Architecture:
Client → Keepalived VIP → HAProxy → PgBouncer → PostgreSQL

🚀 The setup is simple: define your servers in inventory.ini, set a few variables, and run make up. In just a few minutes, you’ll have a fully working Patroni cluster ready for testing or experimentation.

⚠️ Disclaimer: This is a dev/test configuration. Don’t use it in production without proper hardening (passwords, TLS, firewalls, limited SSH, etc).

👉 GitHub: https://github.com/rom8726/ansible-patroni

DEV Community: Roman Chudov

ChaosKit: Code-Level Chaos Engineering framework for Go

Why Code-Level Chaos Matters

How ChaosKit Works

Injectors, Validators, Scenarios

Injectors

Validators

Scenarios

Chaos Inside go test

Runtime behavior

When ChaosKit Is Valuable

Great fit for:

Not ideal for:

Conclusion

Floxy — a lightweight workflow engine for GoLang with saga-style compensation

Floxy — a lightweight workflow engine for Go with saga-style compensation

What is Floxy?

Key features

Example

Retry and Idempotency

Deterministic State Model

When to use Floxy

Learn more

Goodbye, Sentry!

Introducing Warden: A Self-Hosted Error Monitoring Platform

Roman Chudov ・ Jul 21

[Boost]

Introducing Warden: A Self-Hosted Error Monitoring Platform

Introducing Warden: A Self-Hosted Error Monitoring Platform

What is Warden?

Core Features

Why Warden?

What’s Next

[Boost]

testy: YAML-Based Functional Tests for Go HTTP APIs

testy: YAML-Based Functional Tests for Go HTTP APIs

Why testy?

Debug Your Tests Like Real Code

Declarative Testing With YAML

Dynamic Placeholders and Context

Works with Your Real Code

Real-Life Example: Creating a Project

Powered by pgfixtures

Mocks for Outgoing Requests

Highlights

Perfect for:

Try It Out

Open Source

Ansible Playbook for PostgreSQL Patroni Cluster

Chaos Inside `go test`

Why `testy`?

Powered by `pgfixtures`