DEV Community: Tiana

8 architectural bugs we found auditing 29 microservices (with code)

Tiana — Fri, 08 May 2026 22:54:52 +0000

We audited a 29-service Kotlin/Spring + Go + React platform. Found 14 issues - 8 already in our release branch on the way to QA. Sharing all 8 with file paths, code snippets, and exact fixes. The pattern at the end: dev environment masks, healthcheck lies, silent fallback to defaults, config drift between two places.

If you're going to grep your codebase after reading this, the four greps to start with are at the bottom.

1. ML weights missing from the Docker image

Symptom: prediction endpoint returns 500 in k8s. Locally fine. CI passes.

Root cause chain:

ml-api-service/.gitignore:58   →  models/         # excluded
ml-api-service/.dockerignore   →  missing
ml-api-service/Dockerfile:35   →  COPY models/ ./models/
docker-compose.yml:976         →  ./ml-api-service/models:/app/models  (bind mount)

When .dockerignore is absent, Docker BuildKit falls back to .gitignore. models/ is excluded from the build context. COPY models/ copies an empty directory. The bind mount in docker-compose hides this locally.

The lying healthcheck:

@app.get("/ready")
async def ready():
    return {"status": "ok"}  # TODO: real check

Fix: explicit .dockerignore, track weights in git, real readiness probe:

@app.get("/ready")
async def ready():
    model_dir = Path("/app/models")
    if not any(model_dir.glob("*.joblib")) and not any(model_dir.glob("*.onnx")):
        raise HTTPException(status_code=503, detail="No models loaded")
    return {"models_loaded": len(list(model_dir.iterdir()))}

Switch HEALTHCHECK in Dockerfile to /ready.

2. MapStruct + Kotlin `is`-prefix → silent `false`

Symptom: five mappers return false for every isActive, isDemoData, isIgMember, isSubscriptionActive, regardless of DB state.

Root cause: Kotlin compiles isActive into a getter named isActive() (no get prefix). MapStruct introspects the getter and reads the property as active. The DTO constructor expects isActive. Names don't match. MapStruct falls back to default. Build green, no warning.

Fix per field:

@Mapper(componentModel = "spring")
abstract class UserMapper {
    @Mapping(
        target = "isActive",
        expression = "java(entity.isActive())"
    )
    @Mapping(
        target = "isDemoData",
        expression = "java(entity.isDemoData())"
    )
    abstract fun toDto(entity: User): UserDto
}

Fix permanent: in build.gradle.kts,

kapt {
    arguments {
        arg("mapstruct.unmappedTargetPolicy", "ERROR")
    }
}

Compiler now fails on the next mismatch instead of swallowing.

3. Edge-agent default config: `vessel-001` and `insecure_skip_verify: true`

Symptom: none yet - caught before first prod deploy.

The bad default:

# edge-agent/config.yaml
vessel_id: "vessel-001"
mqtt_broker: "tcp://mosquitto:1883"
cloud_url: "http://gateway:8080"
mongodb_uri: "mongodb://mongodb:27017"
tls:
  insecure_skip_verify: true

Fix: split into two files. config.yaml (production template, empty placeholders, TLS strict) and config.demo.yaml (dev values). Refuse to start on bad placeholders:

func validateConfig(cfg *Config) error {
    if cfg.VesselID == "" {
        return errors.New("vessel_id is required")
    }
    if matched, _ := regexp.MatchString(`^vessel-\d+$`, cfg.VesselID); matched {
        return errors.New("vessel_id matches placeholder pattern, refusing to start")
    }
    if cfg.TLS.InsecureSkipVerify && !cfg.AllowInsecureFromCLI {
        return errors.New("insecure_skip_verify requires --allow-insecure-tls")
    }
    return nil
}

4. Three different env-variable prefixes

Symptom: env-based configuration silently ignored. Found when somebody overrode MQTT_BROKER and the agent ignored it.

The drift:

// edge-agent/internal/config/config.go
viper.AutomaticEnv()
viper.SetEnvPrefix("AGENT")     // → AGENT_VESSEL_ID

// edge-agent/cmd/edge-agent/cli/root.go
viper.BindEnv("vessel-id", "EDGE_VESSEL_ID")  // → EDGE_VESSEL_ID

# docker-compose.yml
edge-agent:
  environment:
    EDGE_AGENT_VESSEL_ID: "v-042"   # → EDGE_AGENT_VESSEL_ID

Fix: canonical prefix EDGE_AGENT_*, explicit binds for backward compat, integration test:

viper.BindEnv("vessel_id",
    "EDGE_AGENT_VESSEL_ID",  // canonical
    "AGENT_VESSEL_ID",       // legacy
    "EDGE_VESSEL_ID",        // legacy
)

func TestEnvOverridesConfig(t *testing.T) {
    t.Setenv("EDGE_AGENT_VESSEL_ID", "test-vessel-99")
    cfg := loadConfig()
    if cfg.VesselID != "test-vessel-99" {
        t.Fatalf("env override ignored: got %q", cfg.VesselID)
    }
}

5. One hypertable in two Postgres schemas

Symptom: dashboards show stale data. Logs clean. Continuous aggregates maintaining empty views.

The double-create:

-- docker/init-db/01_create_databases.sql
CREATE TABLE public.telemetry_values (...);
SELECT create_hypertable('public.telemetry_values', 'time_measured');

-- ingest-service/db/changelog/001-init-schema.sql
CREATE TABLE ingest.telemetry_values (...);
SELECT create_hypertable('ingest.telemetry_values', 'time_measured');

ingest writes to ingest.telemetry_values. The view unified_vessel_positions (created by init script) reads public.telemetry_values.

Fix: delete the public-schema table from the init script. Single source of truth = the service's own migrations.

6. WebClient calls with no timeouts, no circuit breakers

Symptom: when one service crash-loops, every caller's reactor worker pool parks on calls that won't return. Cascade.

The default:

// passage-service-client.kt
@Bean
fun passageWebClient() = WebClient.create("http://passage-service:8080")
// no responseTimeout, no connectTimeout

Fix:

@Bean
fun passageWebClient(): WebClient {
    val httpClient = HttpClient.create()
        .responseTimeout(Duration.ofSeconds(5))
        .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, 2000)
    return WebClient.builder()
        .baseUrl("http://passage-service:8080")
        .clientConnector(ReactorClientHttpConnector(httpClient))
        .build()
}

Plus Resilience4j circuit breaker on critical paths:

@CircuitBreaker(name = "passage-service", fallbackMethod = "fallbackVoyagePlan")
suspend fun getVoyagePlan(vesselId: UUID): VoyagePlan {
    return passageWebClient.get()
        .uri("/voyage-plans/{vesselId}", vesselId)
        .retrieve()
        .awaitBody()
}

private fun fallbackVoyagePlan(vesselId: UUID, ex: Throwable): VoyagePlan =
    VoyagePlan.empty(vesselId)

7. Eurostat GPKG can't open inside Spring Boot fat JAR

Symptom: passage-service crash-loops on @PostConstruct. Every downstream that calls it (see #6) cascades.

Root cause: searoute-core ships marnet.gpkg (SQLite) inside its JAR. SQLite needs a filesystem path. Spring Boot fat JAR is zip-inside-zip. SQLite can't open it.

Fix chain.

Gradle task to extract at build:

// passage-service/build.gradle.kts
val extractMarnetGpkg by tasks.registering(Copy::class) {
    val searouteJar = configurations.runtimeClasspath.get()
        .files
        .first { it.name.startsWith("searoute-core") }
    from(zipTree(searouteJar)) {
        include("marnet.gpkg")
    }
    into("$projectDir/src/main/resources/marnet/")
}

tasks.named("processResources") {
    dependsOn(extractMarnetGpkg)
}

Runtime extraction in @PostConstruct:

@PostConstruct
fun init() {
    val target = Paths.get(workingDir, "marnet.gpkg")
    if (!Files.exists(target)) {
        javaClass.getResourceAsStream("/marnet/marnet.gpkg").use { input ->
            Files.copy(input!!, target)
        }
    }
    try {
        spatialIndex = loadFromGpkg(target.toString())
        eurostatAvailable = true
    } catch (e: Exception) {
        log.warn("Eurostat unavailable, falling back to bisection routing", e)
        eurostatAvailable = false
    }
}

healthcheck start_period: 120s in docker-compose to give time for extraction.

8. Keycloak realm without fixed user IDs

Symptom: every fresh Keycloak deploy generates new user UUIDs. JWT.sub diverges from users.id in DB. Data isolation silently breaks.

The bad realm import:

{
  "users": [
    {
      "username": "admin",
      "email": "admin@example.com",
      "credentials": [{"type": "password", "value": "admin123"}]
    }
  ]
}

Fix: pin id to match DB:

{
  "users": [
    {
      "id": "d0000000-0000-0000-0000-000000000003",
      "username": "admin",
      "email": "admin@example.com",
      "credentials": [{"type": "password", "value": "admin123"}]
    }
  ]
}

DB migration must match:

INSERT INTO users (id, email)
VALUES ('d0000000-0000-0000-0000-000000000003', 'admin@example.com');

Integration test that catches this:

@Test
fun `JWT sub matches pinned admin UUID`() {
    val token = loginAs("admin", "admin123")
    val sub = parseJwt(token).getClaim("sub").asString()
    assertEquals("d0000000-0000-0000-0000-000000000003", sub)
}

The four greps

If you're auditing your own system after reading this, here's where to start.

# 1. What's in your image that isn't in build context?
diff <(cat .gitignore) <(cat .dockerignore)
find . -name Dockerfile -exec grep -H "COPY" {} \;

# 2. What does your /health actually verify?
grep -rn "fun health\|/health" --include="*.kt" --include="*.py" --include="*.go"
# Look for stubs returning {"status":"ok"} unconditionally.

# 3. Inter-service calls without timeouts
grep -rn "WebClient.create\|FeignClient\|RestTemplate(" --include="*.kt"
# Cross-check against responseTimeout / circuit breaker config.

# 4. Configuration drift
grep -rn "ENV\|env\|prefix" --include="*.go" --include="*.yaml"
# Look for diverging prefixes/keys across config init, CLI, compose, and helm.

What ties these together

Operationally, all eight are the same class of bug.

Some piece of the dev environment was filling in for what production wouldn't (bind mount, internal docker-compose DNS, exploded Gradle classpath, persistent dev DB). The healthcheck or build signal was reporting fine when the service wasn't. A library somewhere was defaulting silently - MapStruct to false, WebClient to an unbounded wait, Keycloak to a random UUID. Configuration was duplicated in two places and had drifted.

The four greps above are pointed at exactly those four. Run them.

Focus Factor fixed my sprint planning. Here is the method and a tool I built for it

Tiana — Tue, 21 Apr 2026 06:12:00 +0000

The pain

When I was a team lead, sprint planning was my constant pain.

Every planning meeting was the same. Engineers gave estimates in hours, I stacked the sprint to the top of the team's calendar, everyone said "we can do this", and two weeks later we were past the release date and nobody was surprised.

The retro would conclude "we need to be more realistic" and we would repeat the exact same pattern next sprint.

The fix

My department head showed me Focus Factor. The core idea is embarrassingly simple: a developer does not write product code for 8 hours a day.

A working day has:

DSMs, code review, estimation, demo, retro, planning (the recurring process load)
Support tickets, bug fixes, monitoring, legacy work
Meetings outside the recurring process
Fly-in tasks that land during the sprint
Shortened days, vacations, other projects, external commitments

Once you subtract all of that, the real product time per sprint is usually 50-75% of the calendar, not 100%.

So you plan the sprint from that real number, not from "working days × 8".

What the metric actually is

Total Time    = working days × 8
Task Time     = Total - (Processes + Support + Meetings + Fly-in + Wind)
Focus Factor  = Task Time / Total Time × 100

Color buckets we used:

0 - 49% red - the team is drowning in operational work
50 - 74% yellow - it works but it is unstable
75%+ green - strong focus on the product

A realistic ceiling is 70-75%. If you see 90% sustained, your data is lying.

A couple of related metrics that paid off quickly:

Stability Factor = 1 - (Fly-in / Total Time)           // < 0.85 means unstable planning
Support Ratio    = Support / Total Time                // > 0.25 means quality problems
Adjusted Focus   = Task Time / (Total Time - Wind)     // fair for partial availability
Team Focus       = Σ Task Time / Σ Total Time          // the one you actually manage by

The key mindset shift: Focus Factor is not a productivity metric for people. It is an indicator of organizational noise.

If focus drops, the problem is in the system (support load, fly-in discipline, meeting bloat), not in someone's motivation.

From Excel to a tool

We ran this in Excel for months. Formulas, colored cells, weekly updates, per-team tab. It worked, but it was painful to compare sprints, keep the roster in sync, and share a clean view with stakeholders.

So I wrote a small open-source tool to stop living in spreadsheets.

Repo: https://github.com/tiana-code/sprint-pulse

Stack: Kotlin + Spring Boot 3, JPA, Liquibase, H2 or Postgres, React + TypeScript + Vite + Tailwind + Recharts.

Planning from Focus Factor

A sprint in Planning status has an editable hour matrix per member. Focus Factor, Adjusted Focus, Stability and Support Ratio recalculate live. When the sprint starts or closes, the table becomes read-only - this matters, because editing history after the fact destroys trend data.

Sprint report

Every sprint gets a full report: per-member hours, team totals, KPIs, a time-breakdown chart per member, and an auto-generated recommendation (focus critical, support high, stability low, healthy).

Compare sprints over time

Pick any two or more sprints, get a delta column and a side-by-side chart. This is where you catch "our focus dropped from 73% to 45% and support load doubled - something broke in the last release cycle".

Team-level view

Two screens:

Teams comparison - one row per team with average and latest Focus Factor, capacity, stability, support share
Trends - Focus Factor, time breakdown and stability-support lines over sprints for a single team

Shareable report link

Every sprint has a public read-only link you can drop into a work chat for sprint review. No auth. Good for a quick "here is what we delivered, here is what the numbers say" without screenshots.

What is next

Jira integration. The plan is to pull sprint scope, members, working days and support hours from Jira / Tempo and stop retyping.

Run it locally

git clone https://github.com/tiana-code/sprint-pulse
cd sprint-pulse

# backend
cd backend && ./gradlew bootRun
# http://localhost:8081, login admin / admin

# frontend
cd ../frontend && npm install && npm run dev
# http://localhost:5173

Two demo teams with four to seven sprints are seeded at startup so you can click around without typing anything. SEED_ENABLED=false disables it for prod.

Your turn

What metrics and tools actually fixed sprint planning for you? Not what a book recommended - what you were still using after six months. Drop them in the comments.