DEV Community: Vladimir Nemergut

I gave an AI agent my infrastructure instead of my codebase. Every bug was the same bug.

Vladimir Nemergut — Tue, 11 Aug 2026 20:30:45 +0000

A while back I wrote about making a fleet of Claude Code agents talk to each other over Discord. Four agents, one per project, hub-and-spoke so they can't wake each other into a loop.

That design has held. What's changed is everything around it.

The fleet is eight agents now. One of them exists only to administer the others. There's a pool of pre-authorised spare identities so a short-lived agent can be spawned in seconds. And the newest member doesn't write code at all — it runs things.

That last one is where this article comes from. I expected the hard part to be capability. It was verification.

First, the parts that made growth cheap

Two pieces of infrastructure turned "add an agent" from an afternoon into a command.

A management agent. One bot holds the Discord Admin role and does fleet operations through the Discord API rather than by chatting: creating channels, setting permission overwrites, onboarding new bots, auditing who can see what. It's the only member whose job is the fleet itself.

Worth being precise about the topology, because it's two different graphs on one picture: messages all route through the hub, while management runs from the admin bot to everything. A drawing that shows only message flow hides half the system.

A pool of spare identities. This one's the useful trick.

Discord exposes an API for almost everything — channels, permissions, roles, members. It exposes no API for creating an application. That's a hand-click in the Developer Portal, every time.

Which means "spawn an ephemeral agent" can't be automated end to end… unless you pay that cost once per slot instead of once per spawn. So: three spare bot applications, created by hand, registered, and permanently listed in the hub's allowBots. Claiming one is a script. Releasing it is a script. The manual step happened once, in advance.

claim   -> set channel overwrite, hand over the token, launch
release -> revoke the overwrite, park the identity

The pool's promise is that its members are interchangeable. Hold that thought.

Then the fifth kind of agent

The newest member doesn't have a codebase. It has my NAS, my backups, my media server, and a couple of web portals I'd otherwise log into by hand. Errands rather than engineering.

When an agent writes code, its mistakes hit a compiler, a type checker, a test suite, and a diff I read before merging. Several independent systems object before anything reaches production.

When an agent operates, it gets a status code. And a status code is a claim.

In its first week it produced a string of confident, wrong conclusions. Every one was the same mistake in a different costume: it read a signal that represents a state and treated it as the state.

Here's the one that would have cost me real data.

Three weeks of successful backups, backing up nothing

The NAS runs a weekly backup script. Its log said this, every week:

Result=success

The agent checked properly, because I'd asked it to audit rather than glance. The archives existed, on schedule, correctly named. What it found:

The external drive had silently unmounted three weeks earlier. The backup path still existed — as an empty directory on the root filesystem. tar had been dutifully archiving that empty directory ever since. Each archive was 10,240 bytes: a tar header and nothing else.

Result=success was true. The wrapper script had exited 0. That was all it had ever meant.

Three weeks of backups, three green results, zero bytes of data. Nobody finds that until a restore.

Then it kept happening

Once the pattern had a name, it was everywhere:

The signal	Read as	What it actually meant
`HTTP 200` on a delete	the item was deleted	the request was accepted
`Ok.` on login	the password is correct	the password was never checked — subnet auth was on
`SMART: PASSED`	the drive is healthy	no attribute crossed its threshold (24,152 sectors were already reallocated)
`published: false` in a file	the article is unpublished	someone last wrote that line locally; it had been live a week
`HTTP 202` on a fetch	the page was fetched	a bot wall returned a zero-length body
`0 unreadable files` in a dry run	nothing will fail	`--dry-run` never reads file contents, so it cannot hit a bad sector
`.count() == 0` on a login form	the login succeeded	it's an SPA — the form stays in the DOM, hidden
`Result=success`	the backup worked	the wrapper exited 0

Eight instances. One week. One shape.

Why this trap works so well

Checking the signal feels like checking the thing. That's the entire mechanism.

And the more official the signal looks, the more completely it substitutes. A status code, a green check, a PASSED — these are the artefacts of diligence. Consulting one discharges the feeling of having verified. You did a check. You have a result. It's green.

This isn't an AI problem; the SMART one fools humans as a genre. But an agent hits it faster and with more confidence, because it can check twelve things in the time you'd check one and reports each in the same even tone.

An agent never gets the feeling that something is off. That vague unease is doing more work in your debugging than you probably credit.

The pool had it too — and this is the version I like best

Remember the three spare identities. All three read free.

They weren't the same. One of them differed in Discord, and the reason is a two-line asymmetry:

register set the channel permission overwrite and left it in place
release removed it

So a spare that had never been claimed and a spare that had been claimed and released both reported free — while having different live permissions. free was a word in a status column. It described two different states.

The fix was to make register prove the overwrite can be set — that's the step onboarding exists for — and then remove it again. Parked means parked.

But the interesting part is how it was found.

The diagram caught it, and that's not a coincidence

The bug surfaced while regenerating the fleet topology diagram. The commit message records it plainly: found because a diagram drew all three spares as identical, and Discord disagreed.

Drawing a thing asserts that it is uniform. Three rows in a status table can quietly differ — they're three separate readings that happen to share a word. Three identical boxes in a picture make a claim about all three at once, and a claim is falsifiable.

There's a related one worth carrying: earlier, "we tested the bot pool OK" meant one spare was tested. That was reasonable — the mechanism was what was under test. But a pool's entire promise is that its members are interchangeable, and testing one member is precisely the test that cannot check that.

The uniformity did hold. It just wasn't what the test showed.

The counter isn't scepticism

"Be more sceptical" is useless. Sceptical about what? You can't distrust everything.

What works is narrower: prefer signals nobody chose to write.

A status code is a statement a system makes about itself. Someone wrote that line, and it was true of something — maybe not of your question. Compare:

the file existing on disk, with a size
the reply actually arriving
a diff against the repository
a wrong password being rejected

That last one I'd put on a wall. The agent verified a stored credential by logging in successfully. Green tick. But the service had subnet-based auth enabled — any password would have worked. The credential was never tested at all.

"Verified" now means a deliberately wrong password got rejected. A successful login proves the door opens. Only a failed one proves there's a lock.

Three rules that came out of it

1. Read the value back, by a different path than the one that wrote it.

Setting a field and getting 200 means the request was accepted. When the agent fixed metadata on my media server, the write returned success — and the value silently reverted on the next refresh, because it also needed a lock flag. Only the read-back caught it.

2. Match the scope of the claim to the scope of the check.

The agent once reported "there's no permissions block in any settings file" after checking three of nine locations. Every reading was accurate; the sentence was false. A conclusion built from true measurements is more convincing than a guess, not less — it has evidence attached.

3. A check that cannot fail is worse than no check.

I hit this verifying a copy off a failing drive. I compared total byte counts on both sides. It said IDENTICAL. Twice.

The first version printed sums through awk's default format: 6.31658e+11. I was comparing six significant figures — blind to a megabyte of difference in 631 GB.

So I rewrote it with printf "%d". It saturated at 2147483647. All three directories now "matched" at INT32_MAX — which is not a size, it's a ceiling.

I only caught it because I recognised the number.

A broken check doesn't merely fail to catch things. It removes the impulse to look, which was the part that would have saved you.

(The real verification was a per-file comparison of every path and size — 138,172 files, zero differences. Stronger than any total, because a sum can be right for the wrong reasons.)

If you're building one of these

The change that mattered most wasn't technical. It was one line in the ops agent's operating instructions:

A representation of a state is not the state.
When a cheap direct check exists, take it over an authoritative-looking report.

Concretely:

Verify by read-back, by a different path than the write
Treat "no error" as "no error was reported" — not as "it worked"
Make the agent state what it checked, not only what it concluded
Prefer artefacts to reports — a file, a diff, a byte count, a live URL
Draw the system occasionally. A picture asserts uniformity that a table lets you skip past

And one that's easy to miss: ask the human. My agent spent an hour and a dozen requests diagnosing why an API returned 403 on every endpoint. It produced a genuinely good analysis proving the fault wasn't at our end.

The answer was that API access requires a paid plan. One question would have got there in seconds.

The rule about preferring cheap direct checks over process archaeology applies to people too. A person is a signal with no story attached.

Would I do it again

Yes. Eight agents, a management bot, a pool for the short-lived ones, and one that does the work nobody schedules: audit the backups, chase the failing drive, fix the metadata nobody noticed was wrong.

It found a three-week backup failure I had no idea about. It also confidently told me a dying drive was healthy, because smartctl said PASSED.

Same system, same day. The difference was entirely whether the thing it consulted was a fact or a report about a fact.

If you point an agent at your infrastructure, teach it that distinction before any tool, any permission, any clever topology.

Your monitoring has been making claims at you for years. An agent just believes them faster.

Making a fleet of Claude Code agents talk to each other over Discord

Vladimir Nemergut — Mon, 03 Aug 2026 21:10:48 +0000

I run several long-lived projects, each with its own Claude Code session and its
own accumulated context. Different stacks, different databases, no overlap. The
sessions are useful precisely because they don't share context — but that
also means each one lives in its own terminal window, and I can only talk to
them from my desk.

So I gave each session a Discord bot identity. Now I can ask the one that knows
the .NET codebase about a billing edge case while standing in a queue.

That part took an evening. Then I tried to make them talk to each other, and
that took considerably longer — and turned up a security bug I'd written
myself.

The shape of it

Claude Code's Discord channel plugin lets a session send and receive Discord
messages. Run one session per project, give each its own bot token, and each
project gets an addressable identity.

The one non-obvious bit: the plugin reads its state directory from an
environment variable, so every bot needs its own.

$env:DISCORD_STATE_DIR = "$env:USERPROFILE\.claude\channels\bot-a"
claude

Each state directory holds an access.json — who may talk to this bot, in which
channels, and whether a mention is required.

{
  "dmPolicy": "allowlist",
  "allowFrom": ["<my-user-id>"],
  "groups": { "<channel-id>": { "requireMention": true, "allowFrom": [] } }
}

A channel that isn't in groups is dropped before the model ever sees it. This
is the first thing to check when a bot sits silently in a channel it was clearly
invited to.

The wall: bots cannot trigger bots

With four bots in one channel, the obvious next step is to have one coordinate
the others. Mine posted a message mentioning another bot directly. Nothing
happened. No error, no rejection, no log line — the mentioned bot simply carried
on as if nothing had been said.

I assumed a permission problem and spent a while in the Discord developer
portal. It wasn't that. Here is the controlled comparison that settled it:

Time	Author	Mentions	Response
10:28:06	bot	`<@bot-b>`	none
10:33:14	me (human)	`<@bot-b>`	replied in 36 s

Same channel, same config, same mention text. Only the author differed.

The cause is four lines in the plugin's server.ts:

client.on('messageCreate', msg => {
  if (msg.author.bot) return
  handleInbound(msg).catch(...)
})

Discord delivers the gateway event perfectly. The plugin discards it before
access control and before the model. No permission grant can change that, and
there is no setting for it — the server reads exactly three environment
variables, none of which touch this.

That's worth sitting with for a second, because it's a general lesson about
debugging integrations: the layer that delivers the event and the layer that
acts on it are not the same layer, and configuration only ever fixes one of
them.

Why that line is load-bearing

The obvious fix is to delete the guard. Don't.

That single line is doing two jobs. It blocks other bots, and it stops a bot
reacting to itself. Remove it without a replacement and any bot that ends up
allowed to hear itself will answer its own reply, forever, at API speed.

This is not hypothetical. Before I understood the constraint, I sent one message
that mentioned all four bots at once. Three of them replied within six seconds.
Had they been able to hear each other, that would not have stopped.

So the replacement has to separate the two concerns:

// unconditional, not configurable
if (msg.author.id === client.user?.id) return

…and the other bots decision moves somewhere it can be reasoned about.

The patch

I added an allowBots: string[] to access.json, defaulting to [] — which is
exactly the old behaviour, so an unconfigured bot is unchanged.

Two details mattered more than the feature itself:

It's checked in gate(), not in the event handler. gate() re-reads
access.json on every message, so the allowlist can be edited while four
sessions are running, without restarting anything.

Nothing is enforced centrally. Each bot's own config lists who may interrupt
it. There is no global registry, which means no single file whose corruption
opens everything at once.

Topology: hub and spoke, not mesh

My first design put a token-bucket rate limiter in the server, because a mesh of
four agents loops quadratically and I wanted a backstop.

Then I changed the topology instead, and most of the problem evaporated.

One agent is the hub. Spokes list only the hub in allowBots; the hub lists all
spokes. Spokes cannot wake each other. The only possible cycle is hub ↔ one
spoke, and the hub is on every path by construction — so a loop-breaker in
the hub covers the entire system, and I don't have to trust a rate limiter to be
the thing standing between me and a runaway.

I kept a server-side limiter anyway, as cheap insurance. A prompt-level rule can
be rationalised past; server code cannot. But it stopped being load-bearing, and
that's the difference between a safeguard and a hope.

One property worth being precise about: the isolation is on invocation, not
visibility. Spokes can still read each other's messages whenever they happen
to be awake. They just can't wake each other. That's the property being bought,
and it's smaller than it first sounds.

The bug I found reviewing my own patch

This is the part I'd most like someone else to learn from.

The plugin has an intercept for permission prompts: when a tool needs approval,
you can reply y <request_id> in Discord instead of going back to the terminal.
The code around it carried a comment saying, in effect, anything that cleared
gate() is in allowFrom, so this sender is the user.

That was true when it was written. My allowBots change made it false — a bot
now clears gate() without appearing in allowFrom.

Which means an agent could have approved a tool permission on my behalf.

Exploiting it needed the five-character request_id, which is only ever
broadcast to allowlisted DMs, so this was defence in depth rather than an open
door. The fix is two lines: check allowFrom explicitly and exclude bot authors
outright, mirroring what the button handler had always done.

The shape of the mistake is the interesting part. The patch was correct in
isolation and wrong in combination, because it invalidated an assumption
recorded a hundred lines away as a comment. No type checker catches that. The
only thing that caught it was reading the rest of the file after I thought I was
done.

Four things that cost me hours

A BOM silently reset every bot's config. On Windows PowerShell 5.1,
Set-Content -Encoding utf8 writes a UTF-8 BOM. JSON.parse throws on it, and
the server's response to unparseable JSON is to rename the file
access.json.corrupt-<epoch> and start from defaults — no allowlist, no channel
registration. The bot keeps running and simply stops hearing anything. All four
were hit within ten minutes of each other.

Check the bytes, not the values, because every JSON parser except the one that
matters tolerates a BOM:

Get-ChildItem "$env:USERPROFILE\.claude\channels\*\access.json" |
  ForEach-Object { (Get-Content $_.FullName -AsByteStream -TotalCount 3) -join ' ' }

239 187 191 is a BOM. Write these files with
[System.IO.File]::WriteAllText, which is BOM-free on every PowerShell version.

A forked plugin isn't on the approved-channels allowlist. Once I installed
my patched fork, outbound messages kept working perfectly and inbound
notifications were dropped before the session saw them. The bot looks
unresponsive rather than broken, and nothing in its config shows it. The
fingerprint is replies land, nothing arrives, and the answer is a log line in
the MCP logs plus a launch flag naming the fork explicitly.

An agent is never told its own user ID. It sees the sender's, never its own.
So it cannot verify that a mention was meant for it — and when I sent one
message mentioning four bots and saying "you are the orchestrator", each of them
had to guess which clause was theirs. Three guessed wrong. One of them started
signing messages with another bot's name, having inferred its identity from
channel history.

The fix is prose, not code: address bots by name, and state IDs explicitly
when assigning a role. An ID alone is something the recipient cannot check.

Silence is ambiguous in both directions. A busy agent and an undelivered
message look identical. So do an agent that ignored your report and one that
acted on it silently. I had both failure modes in one week — a message that
never arrived and was read as refusal, and an orchestrator that quietly acted on
a correction without acknowledging it, which from the sender's side is
indistinguishable from being ignored.

If you build one of these, make acknowledgement a rule for anything that changes
shared state. It costs one message and removes an entire category of confusion.

Splitting Liquibase changelog? No problem.

Vladimir Nemergut — Tue, 24 Mar 2020 11:03:17 +0000

What is Liquibase?

As Wikipedia says:

Liquibase is an open-source database-independent library for tracking, managing and applying database schema changes. It was started in 2006 to allow easier tracking of database changes, especially in an agile software development environment.

I find Liquibase as a neat tool to migrate your database automatically. DB migration itself is a very complicated topic anyway, outside the scope of this article.

Liquibase can run as a standalone tool or it can be integrated into your application. It's easy to add it to Spring context.

Spring Boot makes it even easier. Liquibase is autoconfigured if you enable it in the properties file, you have Liquibase in the classpath and you have DataSource in the context.

liquibase.change-log=classpath:changelog.xml
liquibase.enabled=true

Liquibase makes the MockMvc testing very simple, too. One can configure it to create the H2 database for the testing purposes.

How Liquibase works?

Liquibase reads the xml changelog file and figures out what changesets it needs to apply. It uses the DATABASECHANGELOG table in your DB (DataSource) for this purpose. The DATABASECHANGELOG contains the list of changesets that are already applied with their ID, FILENAME, MD5SUM, AUTHOR and few other properties.

The logic is relatively simple. Just by comparing the changelog with the table Liquibase knows what changesets it needs to apply. There are, however, few gotchas ...

Liquibase can only take one changelog file
Liquibase determines the list of changesets to apply before applying them
the actual DB can get out of sync with the DATABASECHANGELOG table. E.g. if you manually modify database, or so on
if Liquibase fails to apply a changeset, it fails immediately and won't continue with next datasets
if Liquibase is running in Spring app as a bean, it executes during application startup, hence if it fails, then the application won't start
changesets are not atomic. It can happen that part of the changeset passes, it modifies the DB properly, and next part fails. The changeset record won't go into DATABASECHANGELOG table. Hence it leaves the DB in the state that requires manual repair (e.g. reverting the part of the changeset and letting Liquibase to run again)
the changesets can't be modified. If you modify changeset after it was applied in your db, then Liquibase fails stating that the MD5SUM doesn't match.
The ID is not the unique identifier of the changeset. It is in fact the combination of ID, FILENAME and AUTHOR
the changesets that are in the DATABASECHANGELOG and are not in the changelog files are ignored

Of course, Liquibase has much more functionality. Just read the documentation. This is also out of scope of this article.

The changelog file grows over time

Yep, if you don't define some strategy at the beginning, your changelog file will just grow bigger and bigger. On a large project it can be a couple of thousands of lines long with hundreds of changesets. There is a high code churn on the changelog file, too, so it will cause you some merging effort.

There are a few alternatives that you should consider early on to avoid this.

Define multiple changelog files

.. and <include> them in the master changelog, e.g.:

<?xml version="1.0" encoding="UTF-8"?>
<databaseChangeLog xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
                   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                   xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog dbchangelog-3.5.xsd">

    <include file="feature1.xml" relativeToChangelogFile="true"/>
    <include file="feature2.xml" relativeToChangelogFile="true"/>
    <include file="feature3.xml" relativeToChangelogFile="true"/>
</databaseChangeLog>

The benefit of this one is obvious - less code churn, better organization. The problem comes if there are any logical dependencies between changesets across the files - e.g. if there are any relations defined between tables of multiple files. Since the Liquibase executes the changesets in sequence, it starts with feature1.xml, continues with feature2.xml.

Perhaps you can find out a better split key - based on target releases perhaps?

    <include file="release_0.1.0.0.xml" relativeToChangelogFile="true"/>
    <include file="release_0.1.0.1.xml" relativeToChangelogFile="true"/>
    <include file="release_1.0.0.0.xml" relativeToChangelogFile="true"/>

Configure multiple Liquibase runs

Since one run can only take one changelog file, just define multiple changelog files and let the Liquibase run multiple times.

In your Spring (Boot) app just define multiple liquibase beans:

import liquibase.integration.spring.SpringLiquibase;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.DependsOn;

import javax.sql.DataSource;

@Configuration
public class MultipleLiquiaseConfiguration {

    @Bean
    public SpringLiquibase liquibaseRelease1(DataSource dataSource) {
        SpringLiquibase liquibase = new SpringLiquibase();
        liquibase.setDataSource(dataSource);
        liquibase.setChangeLog("classpath:release_v1.xml");

        return liquibase;
    }

    @Bean
    public SpringLiquibase liquibaseRelease2(DataSource dataSource) {
        SpringLiquibase liquibase = new SpringLiquibase();
        liquibase.setDataSource(dataSource);
        liquibase.setChangeLog("classpath:release_v2.xml");

        return liquibase;
    }
}

Both beans will be created in the context, hence 2 Liquibase runs will be performed. If you rely on the Spring Boot's autoconfiguration, your entityManager bean will force you to have one bean called liquibase. This is easy to do. Also, if your changelogs need to run in a certain order, you can solve this with @DependsOn:

@Configuration
public class MultipleLiquiaseConfiguration {

    @Bean
    public SpringLiquibase liquibaseV1(DataSource dataSource) {
        SpringLiquibase liquibase = new SpringLiquibase();
        liquibase.setDataSource(dataSource);
        liquibase.setChangeLog("classpath:release_v1.xml");

        return liquibase;
    }

    @Bean
    @DependsOn("liquibaseV1")
    public SpringLiquibase liquibase(DataSource dataSource) {
        SpringLiquibase liquibase = new SpringLiquibase();
        liquibase.setDataSource(dataSource);
        liquibase.setChangeLog("classpath:release_v2.xml");

        return liquibase;
    }
}

Note, that the last to run is called liquibase (which your entityManager depends on) and it points to the previous-to-run with @DependsOn annotation.

How to deal with long changelog?

If you haven't applied any strategy early on, or you just joined a running project with legacy code, your changelog is already too big. Now, how to reduce it?

You might say - well, I just split it to multiple files and use either of the 2 strategies as mentioned above. Well, not so fast! :) I mentioned earlier that the filename is important as it is used to determine if a changeset was applied or not. If you simply move existing changesets to another file, Liquibase would think that those changesets were not applied and in fact will try to apply them again. And it will fail as the DB already contains the changes.

To describe the issue a bit better, just imagine a model situation having this changelog.xml:

<?xml version="1.0" encoding="UTF-8"?>
<databaseChangeLog xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
                   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                   xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog dbchangelog-3.5.xsd">

    <changeSet author="me" id="changeset1">
        <createTable tableName="TABLE1">
            <column name="COLUMN1" type="VARCHAR2(10)"/>
        </createTable>
    </changeSet>

    <changeSet author="me" id="changeset2">
        <createTable tableName="TABLE2">
            <column name="COLUMN1" type="VARCHAR2(10)"/>
        </createTable>
    </changeSet>
</databaseChangeLog>

And you do move the second changeset to changelog2.xml and include changelog2.xml in the changelog.xml. Starting your app will fail with similar exception:

Table "TABLE1" already exists;

Ok, it will work just fine in your unit tests, since the DB is created from scratch, but will fail if you run Liquibase to migrate the DB of your deployed instance. We all agree that this is bad ;)

Luckily, we still have a few options left ;)

Change the `logicalFilePath`

Liquibase allows you to define so called logical file path of your changelog. This allows you to fake Liquibase that the changesets actually come from the same file. Imagine the changelog2.xml would look like this now:

<?xml version="1.0" encoding="UTF-8"?>
<databaseChangeLog xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
                   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                   xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog dbchangelog-3.5.xsd"
                   logicalFilePath="classpath:changelog.xml">

    <changeSet author="me" id="changeset2">
        <createTable tableName="TABLE2">
            <column name="COLUMN1" type="VARCHAR2(10)"/>
        </createTable>
    </changeSet>
</databaseChangeLog>

Note the loficalFilePath value there. Yes, this will work, Liquibase will treat this changeset2 as if it was previously defined in changelog.xml. Perfect.

Actually, this approach has also a few drawbacks that might (but might not) stop you from applying. If you don't store your changelog in the resources, but rather elsewhere in filesystem, your DATABASECHANGELOG will contain the full path to the file. If you then have multiple environments where you want to migrate DB and your changelog file location vary, you have no way how to set the logicalFilePath. Remember that it must match the previous value.

Another issue is that this approach is not the best if your intent to split the changelog is to move the part of it to another package, module, and so on.

Use intermediate changelog

If you intend to move part of your changelog to another module (e.g. you finally want to break that nasty monolith of yours to a few microservices having their own database), this approach might suite you the best. It contains some intermediate and temporary steps, but the outcome is what you want :)

The first step is to move all the relevant changesets to another file elsewhere. In our example above we just move the changeset2 to changelog2.xml. Now we need to fake Liquibase that those changesets didn't change. We do it by modifying the FILENAME value in the database as part of the Liquibase changelog itself ;)

Create one more (intermediate/temporary) changelog (let's call it tmp-migration.xml) with just this one changeset:

<?xml version="1.0" encoding="UTF-8"?>
<databaseChangeLog xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
                   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                   xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog dbchangelog-3.5.xsd">

    <changeSet id="moving changesets from changelog to changelog2" author="Maros Kovacme">
        <sql>
            UPDATE DATABASECHANGELOG
            SET
            FILENAME = REPLACE(FILENAME, 'changelog.xml', 'changelog2.xml'))
            WHERE
            ID IN (
            'changeset2'
            );
        </sql>
    </changeSet>

</databaseChangeLog>

This changeset will replace the FILENAME column value in the DB from classpath:changelog.xml to classpath:changelog2.xml. When we then run Liquibase with the changelog2.xml, it will think that all changesets are already applied. It is not possible to use just 2 changelog files for this purpose. Liquibase first calculates the list of changesets to be applied (per changelog file) and only then it will apply them. We need to modify the FILENAME before it processes the second file.

The last step we have to apply is to define the corresponding beans in our context in the right order:

@Configuration
public class MultipleLiquiaseConfiguration {

    @Bean
    public SpringLiquibase liquibaseChangelog(DataSource dataSource) {
        SpringLiquibase liquibase = new SpringLiquibase();
        liquibase.setDataSource(dataSource);
        liquibase.setChangeLog("classpath:changelog.xml");

        return liquibase;
    }

    @Bean
    @DependsOn("liquibaseChangelog")
    public SpringLiquibase liquibaseMigration(DataSource dataSource) {
        SpringLiquibase liquibase = new SpringLiquibase();
        liquibase.setDataSource(dataSource);
        liquibase.setChangeLog("classpath:tmp-migration.xml");

        return liquibase;
    }

    @Bean("liquibase")
    @DependsOn("liquibaseMigration")
    public SpringLiquibase liquibaseChangelog2(DataSource dataSource) {
        SpringLiquibase liquibase = new SpringLiquibase();
        liquibase.setDataSource(dataSource);
        liquibase.setChangeLog("classpath:changelog2.xml");

        return liquibase;
    }
}

The changelog.xml will run first. The changeset2 exists in the DATABASECHANGELOG but not in the file, hence it is ignored. Then the tmp-migration.xml runs and changes the FILENAME column. The last will run the changelog2.xml, but Liquibase will treat the changeset2 as already applied.

Some time later (when you believe that all affected databases are already migrated) you might remove the tmp-migration.xml together with it's bean. The changeset will stay in the DATABASECHANGELOG table but that's just a minor thing I believe.

And then the next step could be to move the definition of beans to the contexts of your concrete microservices.

Conclusion

There is always some way ;)

Reducing the maven build execution time

Vladimir Nemergut — Mon, 16 Mar 2020 10:47:09 +0000

Current state

We have a multi-module Spring Web MVC 4 application with about 100k lines of code. We use Maven and Azure DevOps. A simple build pipeline builds and runs all the unit tests - about 2.8k of them. Well, honestly, I would call most of them component tests or even integration tests. Let's clear out some definition at the beginning.

What is a unit test?

There are many definitions of a unit test. I like the one by Roy Osherove:

“A unit test is an automated piece of code that invokes the unit of work being tested, and then checks some assumptions about a single end result of that unit. A unit test is almost always written using a unit testing framework. It can be written easily and runs quickly. It’s trustworthy, readable, and maintainable. It’s consistent in its results as long as production code hasn’t changed.”

There is a nice article about unit tests in Martin Fowler's bliki so I won't go into much details.

Let's say that we expect the following from a unit test:

fast feedback (has to run fast)
short (small amount of code)
tests one thing (one logical assert)
fails for a good reason

What is a component test?

Again a nice article in bliki.

Since our application uses Spring Data JPA, we have lots of repositories. Many of them contain custom query methods. We use Spring Test DBUnit to help us out testing them. The idea is simple - you setup a mockup database (H2), import some test data before a test, run a test, assert and then cleanup the DB end the end. Since this approach creates the DB and some (small) spring context with all necessary beans, I treat these tests as component tests.

These tests run much longer than unit tests.

What is an integration test?

The bliki defines the purpose of integration tests as:

The point of integration testing, as the name suggests, is to test whether many separately developed modules work together as expected.

In our case, we have the the Spring Web MVC controllers. And those controllers expose RESTful API. For those we usually create a full spring context (with mocked H2 database) and use MockMvc to examine them. The beauty of such tests is that you test the API, including all underlying services (business logic) and repositories (persistence). The huge drawback is the execution time. It takes really long time to create Spring context (depends on how large your app is). And in many cases the context is destroyed and created fresh. This all adds up to the total build time.

What tests do I need?

If you are now biased to lean towards just unit tests, as they are fast to run, and want to ditch your component/integration tests as they are slow - hang on ... not so fast. You need The tests at all levels. In a reasonable ratio. In large applications you might split the test runs to fast/slow to speedup the feedback loop. You also need UI driven tests, system tests, performance tests ... Ok, the whole topic about high quality testing of a software is very complex and you might start reading here if you are interested. Let's get back to our topic for now.

The path to improve the situation

There is a nice article written about what you can do. It tells you to re-think whether you need that many integration tests and rewrite them with unit tests. As described above, do it with caution.

It's important to define the test strategy early on. Otherwise the issue will just grow too big to finally conclude that you deal with lots of legacy tests that is expensive to change now. In companies and large projects this happens too often to simply ignore it. Legacy or not, there is still something we can do with reasonable effort and positive return of investment.

Spring context

Due to many reasons our spring context is created in total 17 times during the test run. Every time it takes about 30 seconds to start. It is 8.5 minutes added to the test run. You can make use of the advice from the article above to make sure that the context is only created once. We could save 8 minutes here.

DBUnit

If using DBUnit, each test starts with importing data to DB and ends with cleanup. There are several strategies for this. The Spring Test DBUnit expects these annotations on test method:

@DatabaseSetup(value = "data.xml")
@DatabaseTearDown(value = "data.xml")

The default strategy is CLEAN_INSERT for both Setup and Teardown which in fact deletes all data from affected tables and insert them again. With pure DBUnit, you configure it explicitly, e.g.:

AbstractDatabaseTester databaseTester = new DataSourceDatabaseTester(dataSource);
databaseTester.setSetUpOperation(DatabaseOperation.CLEAN_INSERT);
databaseTester.setTearDownOperation(DatabaseOperation.DELETE_ALL);

Consider changing these strategies to something faster, e.g. instead of CLEAN_INSERT for Setup, use just INSERT. Of course, this will cause problems if your tables are not cleaned up properly by previous tests. The CLEAN_INSERT of Teardown can be replaced with just DELETE_ALL or even faster one TRUNCATE_TABLE. There is one catch with TRUNCATE_TABLE on H2 database, more details in the appendix.

We have about 1200 tests that are running with DBUnit. Just imagine if we saved 100 milliseconds for setup/teardown. It would be 4 minutes saved with relatively low effort.

Build time analysis

However, our situation was a bit more complicated. Our build execution time grew from about 9 minutes to something about 25 minutes in avarage over the past 6 months.

(the sudden peaks are caused by the fact that we recycle the build agents every now and then and hence the maven cache is gone).

Moreover, we experience some builds taking as long as 45 minutes and some of them even more then 1 hour.

We did a simple analysis of the build logs. The tests are configured to output to the console with TRACE level. There we could see that some operations took suddenly much longer than usual. For instance the DB teardown:

2020-03-13T12:59:21.9660309Z INFO  DefaultPrepAndExpectedTestCase - cleanupData: about to clean up 13 tables=[...]
2020-03-13T12:59:29.0545094Z INFO  MockServletContext - Initializing Spring FrameworkServlet ''

The first message is the Teardown of DB from one test, the second message is starting a new spring context for another test. Note the timestamps. 7 seconds to cleanup the DB! Remember we do it 1200 times. After analyzing one build output (the build took 45 minutes) we calculated the total time spent on cleaning up the DB - 1800 seconds. 2/3 of the build spent on this. It surely can't take that long.

Of course we already checked the infrastructure. The build agents are pretty decent t2.large EC2 instances with 2 vCPUs and 8GiB RAM powering Ubuntu. Should be OK.

I like quick and easy solutions for complex problems. We analyzed the build output using simple tools - shell and excel. Let's do it step by step:

extract the timestamps when cleanup DB starts:

cat buildoutput.log | grep "DefaultPrepAndExpectedTestCase - cleanupData" | sed 's/\([0123456789T.:-]*\).*DefaultPrepAndExpectedTestCase.*/\1/g' > cleanup_start.out

(grep for the lines matching the substring, then using sed to only output the timestamp)

extract the timestamps of the next messages - this will roughly be the end time of the cleanups:
```
cat buildoutput.log | grep "DefaultPrepAndExpectedTestCase - cleanupData" -A 1 | grep -v "\-\-" | grep -v "DefaultPrepAndExpectedTestCase" | sed 's/\([0123456789T.:-]*\).*/\1/g' > cleanup_end.out
```
(grep for the lines matching the substring and 1 line after, grep out the first line and the line with '--', then using sed to only output the timestamp)
Load both files to Excel
- choose Data -> From Text/CSV (Alt+A, FT)
- select the first file cleanup_start.out
- Source and Change type steps should be added automatically, add a few more:
- the new Added column contains Time.Hour([Column1])*60*60+Time.Minute([Column1])*60+Time.Second([Column1])
- the Removed columns step just deletes the Column1
- now the excel contains the amount of seconds (incl. fraction) elapsed from the day start
- repeat the same for second file cleanup_end.out
- add diff column
Now the diff column contains roughly the duration of the DB cleanup. I simply put a sum at the end of the column to calculate the 1800 seconds mentioned above.
plot a chart
- select the whole third column
- add a new line chart
The chart now shows how log it took the DB cleanup during the build. You can see important information here. It was all fine and then we see sudden blocks of peaks. The pattern looks very suspicious.

Short stare at the chart, some experience and the 'blink' moment - the Garbage Collector!!! This explains why it takes 7 seconds to cleanup the DB. Because the full GC runs in the background.

Improving the situation

We checked the POM of the affected module and ... guess the surefire plugin configuration:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <version>${surefire.version}</version>
        <configuration>
            <argLine>${surefireArgLine} -Dfile.encoding=UTF-8 -Xmx1024m
            </argLine>
        </configuration>
</plugin>

Yep, the -Xmx1024m. The maximum Java heap space explicitly set to 1024m. This is not enough for out tests any longer. Even though we set the Xmx in the build pipeline, it is not picked up. This explains everything - when the heap space is reaching the maximum, the full GC is invoked to free up some memory. This radically slows down the test execution.

First thing we did - increase the Xmx to at least 2048m (or leave it empty to default to 1/4 of RAM, whichever is smaller).

Since we are here, let me show you one more thing. If your tests are good enough to run in isolation properly, and you have the HW that is powerful enough, play around with this configuration of surefire:

<configuration>
    <forkCount>2</forkCount>
    <parallel>classes</parallel>
    <threadCountClasses>2</threadCountClasses>
</configuration>

This settings means, that there will be 2 forks of surefire booter (this guy actually executes the tests in isolated JVM), it will use 2 threads to run the tests and it will break down the test suite to classes to distribute them over. Important to remember here is that the threadCountClasses is used within the same (forked) JVM, meaning that there will be 2 threads per fork. So if you run into race conditions, just decrease the threadCountClasses to 1. The tests will still run in parallel, but in 1 thread per fork. And because H2 is typically 1 per JVM, you should be fine. Unless you have another kind shared data outside JVM.

The outcome that we reached - down to 11 minutes build time from 25 minutes, which is only 44% of previous duration.

Conclusion

Understand the tools that you use (e.g. MockMvc, DBUnit, Spring Test DBUnit, Surefire plugin). Analyse your build outputs. Do not close the case with just stating that your application grows, your test suite grows, hence the build time gets longer. Sure this is true, but you should always reserve some time for refactoring and improving your code.

Appendix

DBUnit H2 `TRUNCATE_TABLE` operation

Truncate table is usually faster operation than delete from. There are some catches though. Read more about it here. H2 won't allow you to truncate table if there are foreign keys to that table. However, there is a special H2 syntax that you can make use of. We implemented the following DBUnit operation:

class H2TruncateOperation extends org.dbunit.operation.AbstractOperation
{
    private final Logger logger = LoggerFactory.getLogger(H2TruncateOperation.class);

    @Override
    public void execute(IDatabaseConnection connection, IDataSet dataSet)
            throws DatabaseUnitException, SQLException
    {
        logger.debug("execute(connection={}, dataSet={}) - start", connection, dataSet);
        IDataSet databaseDataSet = connection.createDataSet();
        DatabaseConfig databaseConfig = connection.getConfig();
        IStatementFactory statementFactory = (IStatementFactory) databaseConfig
                .getProperty("http://www.dbunit.org/properties/statementFactory");
        IBatchStatement statement = statementFactory.createBatchStatement(connection);

        try
        {
            int count = 0;
            Stack<String> tableNames = new Stack<>();
            Set<String> tablesSeen = new HashSet<>();
            ITableIterator iterator = dataSet.iterator();

            String tableName;
            while(iterator.next())
            {
                tableName = iterator.getTableMetaData().getTableName();
                if(!tablesSeen.contains(tableName))
                {
                    tableNames.push(tableName);
                    tablesSeen.add(tableName);
                }
            }

            if(!tableNames.isEmpty())
            {
                statement.addBatch("SET FOREIGN_KEY_CHECKS=0");
            }

            for(; !tableNames.isEmpty(); ++count)
            {
                tableName = tableNames.pop();
                ITableMetaData databaseMetaData = databaseDataSet.getTableMetaData(tableName);
                tableName = databaseMetaData.getTableName();
                String sql = "TRUNCATE TABLE " +
                             this.getQualifiedName(connection.getSchema(), tableName,
                                                   connection) +
                             " RESTART IDENTITY";
                statement.addBatch(sql);
                if(logger.isDebugEnabled())
                {
                    logger.debug("Added SQL: {}", sql);
                }
            }

            if(count > 0)
            {
                statement.addBatch("SET FOREIGN_KEY_CHECKS=1");
                statement.executeBatch();
                statement.clearBatch();
            }
        }
        finally
        {
            statement.close();
        }
    }
}

An example SQL statements:

SET FOREIGN_KEY_CHECKS=0;
TRUNCATE TABLE table1 RESTART IDENTITY;
TRUNCATE TABLE table2 RESTART IDENTITY;
SET FOREIGN_KEY_CHECKS=1;

You can use pass this operation to plain DB unit configuration like this:

AbstractDatabaseTester databaseTester = new DataSourceDatabaseTester(dataSource);
databaseTester.setTearDownOperation(new H2TruncateOperation());

or implement a new DB Operation lookup if you are using Spring Test DBUnit:

public class H2SpecificDatabaseOperationLookup extends DefaultDatabaseOperationLookup
{
    @Override
    public org.dbunit.operation.DatabaseOperation get(DatabaseOperation operation)
    {
        return operation == DatabaseOperation.TRUNCATE_TABLE ?
               new H2TruncateOperation() :
               super.get(operation);
    }
}

and use annotations:

@DbUnitConfiguration(databaseOperationLookup = H2SpecificDatabaseOperationLookup.class)

Zorin OS for a web developer? Yes please!

Vladimir Nemergut — Fri, 10 Jan 2020 16:41:12 +0000

Background

I've been a developer for many years. My beginnings were on Windows 95. Since then my primary development environment has been Windows OS based.

Some time back I had an opportunity to switch my work web development environment to MacOS. MacOS is great for that. Switching back to Windows was not necessarilly any comparable experience 😄

So what about trying something else, shall we?

In this chapter you get some introduction and I share my feelings and struggles.

Zorin OS?

When searching over a dozen of modern Linux distributions, I suddenly found Zorin OS hidden behind Ubuntu, Debian, Mint (from a popularity perspective) and elementary OS (from UX perspective). It's great looking desktop OS. Very visually appealing. There are a few editions you might choose from:

Ultimate
Core
Lite
Education and Education Lite

All but Ultimate are free of charge.

I've got a spare Fujitsu Celsius H710 laptop, so I decided to install Core edition on it.

Installation

I don't want to go through the whole installation procedure. You can find many tutorials on internet. After burning the installation DVD of Zorin OS 15.1, I just followed the installation steps. It is super easy (if you manage to boot from DVD).

Struggle with NVidia drivers

The first installation went through, all fine. Then I wanted to tweak the OS a little bit, that was a mistake 😄. The H710 has hybrid graphics. So actually 2 graphics adapters - Intel and NVidia. I tried to install the additional NVidia drivers from the Software & Updates

Neither of the NVidia drivers worked perfectly. With the proprietary driver you get this annoying NVidia logo while booting and the booting seemed to take longer. Both proprietary and open source NVidia drivers caused the brightness adjustment to stop working. I mean the functional key still work, but the brightness always stays on 100%. Switching back to Nouveau drivers was a nightmare, too. The screen resolution went to 640x480 and no way to change it properly.

If you now get a cool idea to switch to Intel graphics adapter ...

sudo prime-select intel

... don't do this 😏. At least I was not able to make it work with Intel adapter. Mind you, even reinstall won't help as you need to switch back to NVidia.

The only remedy that worked for me was:

switch to NVidia

sudo prime-select nvidia

uninstall NVidia drivers completely:

sudo apt purge nvidia-*

After that Zorin OS experience was back to the top level:

Zorin OS logo when booting in high-res
resolution back to screen native

Laptop power management

Power management is surely important if you want to use your laptop on battery. The most advanced tool is the TLP.

Just install TLP on your laptop:

sudo add-apt-repository ppa:linrunner/tlp
sudo apt update
sudo apt install tlp tlp-rdw

You might find TLP GUI handy:
sudo add-apt-repository ppa:linuxuprising/apps sudo apt update sudo apt install tlpui`

Then go to Software app and search for TLP Extension. This will give you an easy way to switch between power management profiles without opening a terminal window and typing in something.

The TLP installs the default TLP profile under /etc/defaults/tlp. You might follow these steps to create your custom profiles.

copy the /etc/defaults/tlp to ~/.tlp/<Your profile name> as many times as many profiles you want
open the respective profile file in TLP GUI
make all modifications to your liking and save it

What next?

In the next chapters I'd share my impressions of using Zorin OS as web developer.

DEV Community: Vladimir Nemergut

I gave an AI agent my infrastructure instead of my codebase. Every bug was the same bug.

First, the parts that made growth cheap

Then the fifth kind of agent

Three weeks of successful backups, backing up nothing

Then it kept happening

Why this trap works so well

The pool had it too — and this is the version I like best

The diagram caught it, and that's not a coincidence

The counter isn't scepticism

Three rules that came out of it

If you're building one of these

Would I do it again

Making a fleet of Claude Code agents talk to each other over Discord

The shape of it

The wall: bots cannot trigger bots

Why that line is load-bearing

The patch

Topology: hub and spoke, not mesh

The bug I found reviewing my own patch

Four things that cost me hours

Splitting Liquibase changelog? No problem.

What is Liquibase?

How Liquibase works?

The changelog file grows over time

Define multiple changelog files

Configure multiple Liquibase runs

How to deal with long changelog?

Change the logicalFilePath

Use intermediate changelog

Conclusion

Reducing the maven build execution time

Current state

What is a unit test?

What is a component test?

What is an integration test?

What tests do I need?

The path to improve the situation

Spring context

DBUnit

Build time analysis

Improving the situation

Conclusion

Appendix

DBUnit H2 TRUNCATE_TABLE operation

Zorin OS for a web developer? Yes please!

Background

Zorin OS?

Installation

Struggle with NVidia drivers

Laptop power management

What next?

Change the `logicalFilePath`

DBUnit H2 `TRUNCATE_TABLE` operation