DEV Community: Jilles van Gurp

Escaping DevOps hell with Codex

Jilles van Gurp — Thu, 12 Mar 2026 14:35:39 +0000

If you are a developer, you are probably well aware of all the AI goodness that has been happening. I won't bore you with the hyperbole.

My weapon of choice is Codex. Other AI coding tools are available, but debating which one is best at what is not a very interesting conversation to me. What I want to focus on here is DevOps.

I'm the CTO of a small company, which means I switch hats a lot. I used to call myself a backend developer, but these days I do everything vaguely tech-related in our company, plus marketing, sales, and a bunch of other supporting roles. There are only a few of us in the company. If it needs doing, one of us needs to do it. ChatGPT and Codex allow us to get shit done.

DevOps sucks. It really does.

One of the hats I end up wearing is DevOps. I hate doing DevOps. I've been exposed to this stuff for well over two decades. I know how to do it properly. I've done everything from uploading zip files over ISDN lines and remotely restarting Tomcat, as you did in the early 2000s, to automating deployments with Puppet, using CloudFormation in AWS, faffing about with Kubernetes, Docker Swarm, Terraform, and much more. Lately, my weapons of choice are Ansible, Docker, and Docker Compose. I've invested countless quantities of time in learning all that stuff and trying to apply it.

Why do I think DevOps sucks? To me, it feels like dropping out of warp, to use a Star Trek analogy. You have all these grand plans to get some big feature out, and then you find yourself micromanaging some insanely arcane shit in Linux to get it to tell the time correctly, deal with some convoluted networking thing, or whatever. You get blocked for weeks on end. All that to solve the age-old problem of "put this fucking thing over there and run it!" (pardon my French). I call this problem inception. You start with a grand vision: "Our shiny new backend is ready to go, let's deploy it and announce it to the world." Somehow, that escalates into: "I need to figure out how to set up a bastion and private networking so I don't expose my database to the public internet." One thing leads to another, and before you know it, you've sunk three months into the project.

DevOps should be simple but isn't

DevOps is supposed to be about automating what should be automated. So, why does DevOps still feel so manual? The answer is that this stuff is genuinely complicated, and over decades we have built systems full of bear traps with terrible failure modes: data loss, security breaches, downtime, and worse. There is just a lot of stuff that a DevOps person needs to know and do. Taking shortcuts can lead to disaster. That's why it often ends up being a full-time job.

Every once in a while, I get sucked into doing a stretch of DevOps that makes me feel stupid, because it should be simple. Instead, I end up pulling my hair out for days trying to solve weird shit that refuses to work without ritualistic bullshit, magical command-line incantations, and configuration files that need to be exactly right. I know some truly excellent operations and DevOps people and, honestly, I suffer a bit from imposter syndrome whenever I have to do this stuff myself. I'm skilled enough to be dangerous, and I know it.

Codex takes the pain away

I got pulled into the latest round of this two weeks ago. We've been eyeing our setup in Google and concluded that we're spending about 10K per year on hosting. It works great, but it's a lot, and we'd prefer to give ourselves a little raise instead of donating to Google. So, I embarked on the plan to migrate to Hetzner.

But I used Codex to do it. We also have a second setup that, for customer reasons, runs in Telekom Cloud, which is basically an OpenStack-based environment. I already had a lot of Ansible scripts to provision that.

I started by telling Codex to refactor and modernize that codebase and set up a new inventory for my brand-new Hetzner setup. I created a few VMs in Hetzner, a private network, and a load balancer. One of the VMs acts as a bastion so you can SSH into it to reach the other ones that don't have a public IP address.

In small steps, I fixed, upgraded, and modernized the Ansible scripts, using the new Hetzner setup as the test bed. I let Codex do all the work. I got it to fix the Ansible code and drive the provisioning through the tools on my laptop and over SSH. I set up skills and guardrails around the process.

When the Ansible scripts failed, I got it to debug why and implement fixes. I got it to research workarounds. A lot of this was me guiding it along the way. I was leaning on my 2+ decades of experience, but I did not touch a single line of code.

This is actually key. As you go, you will see it struggle and figure things out. When that happens, you just ask it to record what it just did as a skill. You read over the markdown it generates, suggest further improvements, and the next time it does better. This feedback loop between you and the AI is super important. It's the difference between doing things your way, i.e. the right way, and doing it in whatever way it dreams up. Knowing what the right way is, is an experience-driven thing. And this is now our added value as software makers.

After 3 days, I switched over the DNS and shut down the last of our Google VMs. That's 3 days from creating the Hetzner account, not counting the extra day I spent getting them to raise my VM limit because I was still in my first month.

Doing rolling restarts with Elasticsearch

Part of our setup is Elasticsearch. If you've never used it, it's a clustered search engine that achieves high availability through node and cluster coordination. To upgrade it, you have to do a so-called rolling restart. You can't just yank the plug and upgrade all nodes because that would make the cluster unavailable and cause all sorts of issues with split brains, data loss, and worse. This is exactly the kind of thing you normally want an experienced DevOps person to handle responsibly. DevOps is full of tasks where experience matters, because the wrong shortcut can quietly turn into an outage or data loss.

The XXX Elasticsearch cluster upgrade is complete.

All three nodes are now on 9.3.1:

es1 9.3.1
es2 9.3.1
es3 9.3.1
Final cluster state:

green
3 nodes / 3 data nodes
0 relocating shards
0 initializing shards
0 unassigned shards
Notable preflight/result details:

Latest 9.3.x release verified: 9.3.1
Snapshot health passed before rollout; latest successful snapshot was 2026_03_12_05_33
System feature migration was required and completed before the first node restart
I did the rollout one node at a time in this order:

es3
es2
es1
No code changes were needed in the repo.

This morning, I asked Codex to reprovision the cluster in our old data center. I last did this in September, manually. It first did a quick inventory check and a dry run on one of the nodes. It verified the cluster was in a healthy state (green) and that the latest snapshot was verified. Then it went ahead and executed the rollout. It followed the plan to the letter. The block above is the report it gave me when it finished. The whole thing took about 20 minutes. I was on standby giving permission the few times it came back to ask for permission for the next step, as my skill for this specifies.

Before doing this, I iterated with it on writing a skill for doing this as part of my Hetzner migration. The skill covers what an experienced DevOps person would normally do: preflight checks before kickoff, confirmation gates that ask me for permission, and guidance for "what if this happens" scenarios. There's plenty of advice on the internet, and I even wrote about this exact topic years ago in Running Elasticsearch in a Docker 1.12 Swarm. Writing blog articles was something I used to do more regularly as a way of saying, "I should remember this weird thing I just spent 2 days figuring out so I don't have to spend that time again." It's the pre-AI way of creating and recording skills.

If you are interested, you can find the skill file I used here.

What's next?

Over the past few weeks, I've been planning and executing a ridiculous amount of work. I've launched two new websites via Cloudflare. I've created a few new OSS projects. I've done some major surgery on our two live deployments. I've also shipped a few major new features. Somehow, I have also found some time to try out OpenClaw and play with some new AI stuff. I've compressed months of work into a few weeks. I'm not going to lie: I'm exhausted, but I'm also energized. This is crazy fun.

Next on my agenda for modernizing DevOps bullshit that I don't want to deal with is getting some world-class AI monitoring and alerting in place. I need telemetry, logging, and all the rest. I have some of that already, but having it and actually using it are two different things. I want an AI to handle the operational discipline part: checking uptimes, verifying backups, watching resource usage, and making sure everything is working as it should. I want it to give me daily reports, summarize what matters, and escalate issues. I don't want to take a sabbatical to set all this up manually. I just want to get this shit done.

If this sounds like something your team needs

One of the other things I did with Codex recently was launch our AI services and consulting site: formationxyz.com.

The pitch is simple. A lot of companies can see the opportunity with AI, but they struggle to turn that into practical systems, useful workflows, and actual leverage for their teams. That is exactly the gap we want to help close.

At FORMATION XYZ, we help small teams automate repetitive work, build practical AI systems, and put agentic workflows in place that reduce manual effort and create more capacity. If the kind of work I described above sounds interesting to you, and you want help applying AI inside your company in a pragmatic way, we can help.

Kts Scripting of Yaml & Json Dialects

Jilles van Gurp — Tue, 09 Aug 2022 11:14:00 +0000

I've been using Kotlin for quite a few years now. And while I've been using Gradle with the Kotlin scripting support, I've not done much else with Kotlin's scripting ability until fairly recently.

Kotlin scripting (kts) allows you to write scripts with a slightly unfortunate ending of .main.kts that can be interpreted by the Kotlin compiler on the command line. Adding #!/usr/bin/env kotlin to your script tells your shell to use kotlin to execute the script. Any dependencies needed by the script are cached after the first use. So, running these script is generally pretty quick once you have all the jars you need.

Scripting Github Actions

One of my team members, Nikky, got annoyed with the verbosity and insane amount of copy-paste reuse needed to drive Github Actions. And true to her nature, promptly fixed it by using and contributing to GitHub Actions Kotlin DSL

This Kotlin DSL allows you to script github actions using kts. The idea is you write your actions as a .main.kts file, give it execute permissions and then it spits out Yaml files when you run it (one for each workflow that you configure). All the repetitive stuff? Use functions or variables or constants. So much nicer.

Here's a short example:

#!/usr/bin/env kotlin
@file:DependsOn("it.krzeminski:github-actions-kotlin-dsl:0.21.0")

import it.krzeminski.githubactions.actions.appleboy.SshActionV0
import it.krzeminski.githubactions.domain.RunnerType
import it.krzeminski.githubactions.domain.triggers.Push
import it.krzeminski.githubactions.domain.triggers.WorkflowDispatch
import it.krzeminski.githubactions.dsl.expressions.expr
import it.krzeminski.githubactions.dsl.workflow
import it.krzeminski.githubactions.Yaml.writeToFile

val branch = "production"
val environment = "our-environment"

val appServers = mapOf(
    "app1" to "192.168.0.152",
    "enrich1" to "192.168.0.248",
    "app2" to "192.168.0.43",
)
val bastionIp = "xxxxxxxx"
val deployKey = expr("secrets.DEPLOY_KEY")

fun sshAction(host: String, script: String): SshActionV0 {
    return SshActionV0(
        host = host,
        port = 22,
        username = "ubuntu",
        key = deployKey,
        proxyHost = bastionIp,
        proxyPort = 22,
        proxyUsername = "ubuntu",
        proxyKey = deployKey,
        script = script
    )
}

workflow(
    name = "Deploy to $branch",
    on = listOf(
        Push(branches = listOf(branch)),
        WorkflowDispatch()
    ),
    sourceFile = __FILE__.toPath(),
    targetFileName = "deploy_${branch}_$environment.yml",

    ) {

    job(
        id = "publish-telekom-$branch",
        runsOn = RunnerType.UbuntuLatest
    ) {

        appServers.forEach { (name, ip) ->
            uses(
                name = "restart-$name-$branch-$environment",
                action = sshAction(ip, "/opt/formation/bin/restart")
            )
        }
    }
}.writeToFile(true)

Note how we use a loop to call the sshAction function three times. This script generates a Yaml file which is much longer and repeats the action three times. Not DRY and super brittle. And that's on top of the complete lack of type checks, auto-completion, etc. Doing this with kts is so much nicer, much less error prone, and much quicker to figure out.

The little writeToFile(true) spits out the Yaml and tells it to add a consistency check that ensures the committed Yaml matches the script output. So, modifying the kts script and then forgetting to run it will fail the action.

There is much more to this library of course. A lot of github actions are supported out of the box (see here for an overview) and you can add support for additional extensions pretty easily; or just pass in a map.

Also, you can of course produce more than one action from a single kts script. We use this to configure actions for e.g. pull requests and merges to master. The latter has some continuous deployment related actions but of course they share a lot of code. Likewise, most of our actions include a slack notification and share a lot of configuration. With Yaml, you end up with a lot of duplication. With kts, you can get rid of all that duplication.

Finally, not having to deal with Yaml's weird syntax is a big plus. Manually editing Yaml seems very brittle, verbose, and error prone.

Applying my new knowledge to kt-search

I've been working on kt-search, my Kotlin Multi-Platform client for Opensearch and Elasticsearch for a while. Somehow, it never occurred to me that using that in combination with kts is such an obvious thing to do.

So, that was easily remedied and I now have a companion library that combines that with kotlinx-cli to make writing scripts very straightforward.

Here's a little script that checks status of your Elasticsearch/Opensearch cluster:

#!/usr/bin/env kotlin

@file:Repository("https://jitpack.io")
@file:Repository("https://maven.tryformation.com/releases")
@file:DependsOn("com.github.jillesvangurp:kt-search-kts:0.1.3")

import com.jillesvangurp.ktsearch.ClusterStatus
import com.jillesvangurp.ktsearch.clusterHealth
import com.jillesvangurp.ktsearch.kts.addClientParams
import com.jillesvangurp.ktsearch.kts.searchClient
import com.jillesvangurp.ktsearch.root
import kotlinx.cli.ArgParser
import kotlinx.coroutines.runBlocking

val parser = ArgParser("script")
val searchClientParams = parser.addClientParams()
parser.parse(args)

val client = searchClientParams.searchClient

// now use the client as normally in a runBlocking block
runBlocking {
    val clusterStatus=client.clusterHealth()
    client.root().let { rootResp ->
        println(
            """
                Cluster name: ${rootResp.clusterName}
                Search Engine distribution: ${rootResp.version.distribution}
                Version: ${rootResp.version.number}
                Status: ${clusterStatus.status}
            """.trimIndent()
        )
    }
}

Several things that are happening here. It pulls in the kt-search-kts jar and it's dependency kt-search as well as a lot of other dependencies.

Then it creates a kotlinx-cli parser and adds the parameters we need to be able to set host, port, and other settings we need to get a search client. And then it calls the searchClient extension property on that to create the client with those settings.

And then we use it. You can do whatever you want with this:

Run some queries
Do some bulk indexing
Use the snapshot APIs
Configure cluster settings
Index management

I've seen other scripting languages being used for this. Python and go seem to be popular options for this. IMHO, this is nicer. More type safety, less guessing, less verbosity.

How to run these scripts

To run these scripts, you need to install kotlin 1.7.x via your package manager of choice. Homebrew works, there's a snap package, there's an arch package, etc. Whatever OS and package manager you use, you can probably make it run. And of course you can also use docker for this.

After that, just make sure the shebang is set correctly ('#!/usr/bin/env kotlin') and give your script execute permission:

chmod 755 myscript.main.kts
./myscript.main.kts

Is it perfect?

Of course this is far from perfect. In my opinion, Jetbrains can and should make a big effort to make this way nicer.

The .main.kts filename ending is silly. This should be just .kts. I'm sure there's a reason for this but I doubt it's a very good reason.
Kotlin has a native compiler now that works on Linux, Mac, and Windows. And a growing ecosystem of multi-platform libraries. Pre-compiling scripts to binaries would be a nice option. Even pre-compiling them to executable jar files might be a nice thing and it would simplify the run-time dependencies.
The import mechanism is a bit flaky and brittle and doesn't understand multi-platform dependencies. If you look at the build file of kt-search-kts, you will see it depends on kt-search-jvm. Adding the -jvm forces it to depend on the jvm variant. Reason: the dependency resolution in kts is not smart enough to add this by itself.

Final verdict

These are just two examples of what you can do with kts. Out of the box, you can use the full Java standard library and as I show above, adding some additional dependencies is pretty easy. One of Kotlin's killer features is defining so-called internal Domain Specific Languages (DSLs). Basically, you abuse the Kotlin syntax to turn whatever framework you have into a mini DSL. My search client has DSLs for querying, index mappings, bulk indexing, etc. And the Github action library I use of course provides a DSL for Github actions.

Whatever you are dealing with, you can create a Kotlin DSL for it. If you have any Json dialect, checkout my JsonDsl library, which is part of kt-search. With that you can create simple Kotlin classes to model your DSL using type safe properties and have a run-time modifiable map to add anything it doesn't model. Creating a Yaml version of this is very straightforward and likely something I might do at some point (pull requests welcome).

Once you have that, you can script whatever: Amazon Cloudformation, Ansible, Elasticsearch Queries, etc. So, while kts still has some rough edges, it is so much nicer than writing Yaml, ansible, or whatever other type unsafe, not quite-a-scripting-language, you are using currently. Minimal verbosity, maximum gains.

Docker over SSH & Qemu : Replacing Docker for Mac

Jilles van Gurp — Wed, 01 Sep 2021 06:40:05 +0000

Yesterday, Docker announced that Docker for Mac is going to require a paid account for large companies soon. While this does not immediately impact me, I have been relying on docker desktop for mac for a while and that annoys me for several reasons:

it's somewhat flaky. I've had loads of issues over the years where I had to wipe it out and reinstall.
the update process is flaky
it barfs a lot of stuff all over the file system making cleanup a PITA

The Alternative: DOCKER_HOST & ssh

Docker is a simple binary that acts as a client to the docker daemon. Normally it connects to that via a socket that the locally running docker daemon creates.

However, you can easily make it connect to a remotely running docker daemon by either using the -H option or setting the DOCKER_HOST environment variable. One of the supported protocols is ssh.

So, I installed qemu via homebrew, and created a vm running a linux distribution (Manjaro), installed ssh & docker on that, and set up an authorized key so I can ssh into that from my mac terminal. I configured the networking to forward port 5555 to the ssh port 22 on the vm.

Then I simply set this environment variable on my mac:

export DOCKER_HOST=jilles@localhost:5555

After this all my docker commands (using the client that came with docker for mac; but of course you can install that via homebrew as well), go to the remote host, which is where all the docker containers run.

Docker Port Forwarding

The point of running docker is launching things like databases, web servers, etc. with the goal of actually connecting to these things. These things have ports that you might want to talk to. Normally what you do is forward those ports with the -p option. For example,

docker run -p8080:80 nginx

would run nginx and allow you to connect to that from localhost via port 8080. One minor problem: those ports will be inside the linux vm and not on your mac's localhost.

There are various ways to address this. An easy one is to use ssh for this. To forward a port, you can use the -L option with ssh:

ssh -L 8080:localhost:8080 -p 5555 jilles@localhost

After this, you can access it in your browser.

Installing Qemu on a Mac

You can make this work with any remote ssh; including cloud based options for this. But I wanted a locally running vm. There are ways to do this on mac. A nice OSS and lightweight option for this is qemu. But of course, you can probably make this work with parallels, virtualbox, vmware, or whatever else.

You can install qemu via homebrew (or whatever else you prefer). Also make sure to install libvirt. I was not able to get networking going without it.

brew install qemu libvirt

When that has finished and you've started the libvirt service (see instructions it dumps in your terminal when it installs), you can create a disk image and start qemu with a linux iso of your choice (or whatever OS you prefer).

For reference, here's a script that I use to start qemu:

#  qemu-img create -f qcow2 manjaro.qcow2 30G
#   -cdrom  manjaro-xfce-21.1.1-210827-linux513.iso \

qemu-system-x86_64 \
  -m 4G \
  -vga virtio \
  -display default,show-cursor=on \
  -usb \
  -device usb-tablet \
  -machine type=q35,accel=hvf \
  -smp 2 \
  -drive file=manjaro.qcow2,if=virtio \
  -cpu Nehalem \
  -device e1000,netdev=net0 \
  -netdev user,id=net0,hostfwd=tcp::5555-:22 \
  -soundhw

The commented first line is the command you use to create a disk image. The line below that is the command line option for qemu you use to mount the linux iso. After you've installed to your disk image, you can remove that and boot from the disk image. I went with manjaro, which was pretty hassle free to get going. The networking options are important as you need port forwarding for ssh.

Benefits & Down Sides

On the plus side:

I have a nice sandboxed linux vm that contains all my docker stuff. I can shut it down, upgrade it, wipe it out, etc.
Docker command line works as normally and things like docker-compose work as well
qemu is reasonably fast and uses a similar virtualization strategy as docker for mac uses
I can uninstall docker for mac.

Downsides:

This stuff needs memory and cpu. I noticed qemu struggling a bit with some of the things I normally run in docker for mac.
While command line docker works fine, other things like some of our gradle build files assume docker is running locally. I may have to investigate forwarding a socket over ssh. Likewise docker port forwarding needs some manual work as well.
You need some level of skills on the command line, setting up linux, getting qemu going. If you have that; great.

There are all sorts of ways to make this more seamless but it works quite nicely like this already. Adding stuff like kubernetes support might be relevant for some. Making the port mapping less tedious might be doable with e.g. sshuttle or setting up an ssh proxy.

Let me know on twitter (@jillesvangurp ) what you think about this.

UPDATE, after running this for a week, I figured out that I needed to disable app nap, which is slowing down applications that you aren't looking at, like qemu when you are using it via ssh. Really annoying feature. You might not want to do this on a laptop. I added two aliases for this to my .zshrc:

alias appnapoff="defaults write NSGlobalDomain NSAppSleepDisabled -bool YES"
alias appnapon="defaults write NSGlobalDomain NSAppSleepDisabled -bool NO"

Also, kind is a nice option if you want to add Kubernetes to this mix.

Improving Build Speeds.

Jilles van Gurp — Wed, 24 Mar 2021 09:01:54 +0000

I wrote a way too long HN comment this morning and realized that I probably should turn that into a proper article. The article that triggered me was a pretty old one on the importance of keeping builds fast. I could not agree more. And I have lots of wisdom to share on that front from having worked to keep builds fast for most of my career. Even though I develop mostly Kotlin these days, I also work with other tech stacks and pretty much all of the advice applies to almost any tech stack.

Build Performance Matters

I've always been aggressive on trying to keep my Java and lately Kotlin builds fast. Anything over a few minutes in CI becomes a drain on the team. Basically, a productive team will have many pull requests open at any point and lots of commits happening on all of them. That means builds start piling up. People start hopping between tasks (or procrastinating) while builds are happening. Cheap laptops become a drain on developer productivity. Etc. All of this is bad. Maintain the flow and keep things as fast as you can. It's worth investing time in.

Some of the overhead is unavoidable unfortunately. E.g. the Kotlin compiler is a bit of a slouch despite some improvements recently. Many integration tests these days involve using docker or docker compose. That's better than a lot of fakes and imperfect substitutes. But it sucks up time. A lot of Kotlin and Spring projects involve code generation. This adds to your build times. Breaking builds up into modules increases build times as well. Be mindful of all this.

The rest of this article is a series of performance tips not covered in the hacker news article. Most of it should apply to any tech stack; though some may have limitations with e.g. concurrency.

Run Tests Concurrently

Run your tests concurrently and write your tests such that you can do so. Running thousands of tests sequentially is stupid. When using Junit 5, you need to set junit.jupiter.execution.parallel.enabled=true in platform.properties (goes in your test resources). Use more threads than CPUs for this as your tests will likely be IO limited and not CPU limited. Use junit.jupiter.execution.parallel.config.dynamic.factor=4 to control this in Junit 5.

If you are not maxing out all your cores, throw more threads at it because you can go faster. If your tests don't pass when running in parallel, fix it. Yes, this is hard but it will make your tests better.

No Database or Other Expensive Cleanup

Don't do expensive cleanup and setup in tests. Set it up once. Doing repeated cleanup and setup takes time. Also integration tests become more realistic if they don't operate in a vacuum: your production system is not an empty system.

To enable being able to do this, randomize test data so that the same tests can run multiple times even if data already exists in your database. Docker will take care of cleaning up ephemeral data after your build. This also helps with running tests concurrently.

Tests are Either Unit or Integration/Scenario Tests

Distinguish between (proper) unit tests and scenario driven integration tests as the two ideal forms of a test. Anything in between is going to be slow and imperfect in terms of what it does. This means you can either improve test coverage (of code, functionality, and edge cases) by making it a proper integration test or faster by making it a proper unit test (runs in milliseconds because there is no expensive setup).

Integration Tests Should be Scenario Driven

With integration tests, add to your scenarios to make the most of your sunken cost (time to set up the scenario). Ensure they touch as much of your system as they can to do this. You are looking for e.g. feature interaction bugs, Heisen-bugs related to concurrency, weird things that only happen in the real world. So make it as real as you can get away with. A unit test is not going to catch any of these things. That's why they are called integration tests. Make it as real as you can.

Fix Flaky Tests

Fix flaky tests. This usually means understanding why they are flaky and addressing that. If that's technical debt in your production code, that's a good thing. Flaky tests tend to be slow and waste a lot of time.

Fail Fast

Separate your unit and integration tests and make your builds fail fast. Compile + unit tests should be under a minute tops. So, if somebody messed up, you'll know in a minute after the commit is pushed to CI.

Don't Sleep

Get rid of sleep calls in tests. This is an anti pattern that indicates either flaky tests or naive strategies for dealing with testing asynchronous code (usually both). It's a mistake every time and it makes your tests slow. The solution is polling and ensuring that each test only takes as much time as it strictly needs to.

Use More Threads Than CPUs

Run with more threads than your system can handle to flush out flaky tests. Interesting failures happen when your system is under load. Things time out, get blocked, deadlocked, etc. You want to learn about why this happens. Fix the tests until the tests pass reliably with way more threads than CPUs. Then back it down until you hit the optimum test performance. You'll have rock solid test that run as fast as they can.

Keep Your Tools up to Date

Keep your build tools up to date and learn how to use them. Most good build tools work on performance issues all the time because it's important. I use Gradle currently and the difference between now and even two years ago is substantial. Even good old Maven got better over time.

Get Fast Build Machines

Pay for faster CI machines. Every second counts. If your laptop builds faster than CI, fix it. There's no excuse for that. I once quadrupled our CI performance by simply switching from Travis CI to AWS code build with a proper instance type. 20 minutes to 5 minutes. Exact same build. And it removed the limits on concurrent builds as well. Massive performance boost and a rounding error on our IT cost.

Conclusion

Most of this advice should work for any language. Life is too short for waiting for builds to happen. With all of this, do as I say and not as I do. I am always battling slow builds in any project I join. Some of these things tend to be controversial in some teams. People get obnoxious and religious about using docker (or not), using in memory databases (or not). Adapt to your team. If you want fast, builds, understand why they are slow and how you can fix it. The above advice is just a range of tools you can use. Or not. Make using them a conscious choice at least. It's better than being fatalistic about accepting slow builds as a de-facto reality.

Reactive Security Filter with Spring & Kotlin

Jilles van Gurp — Tue, 25 Aug 2020 10:39:40 +0000

Over the years, I've had to implement security filters a couple of time. Recently I had to add JWT token based API authentication to a Spring project.

Some complicating factors:

It's the reactive variant of Spring, aka. Flux.
Flux has a very complicated API surface.
To not have to deal with that we use Kotlin & Co-routines. This too has a few rough edges still as it is very new.
Servlet Filters don't work when using Flux because they are inherently synchronous. So we have to do things the Flux way.

So, I spent a good amount of time to figure out how to do this correctly and this is another instance of me documenting by blogging so I don't have to google my way through the maze of cryptic documentation again. Also, I hope others might find this useful.

High-level design

The design for my solution is fairly simple. I never really liked Spring Security as it mainly gives me headaches. Also I have custom requirements now and more coming that just won't fit in what it does that easily (I speak from experience). But I imagine it does something similar.

So instead:

We use simple JWT tokens that are signed, have a payload with things like a userId, and need to be checked as part of our Authentication and Authorization logic. Standard stuff these days.
Any request that includes an Authorization header, we want to grab the JWT token from there, validate it, and create a SecurityContext object. This object forms the basis for our authorization logic.
The Authorization logic lives in an AuthorizationService that is called to run checks from our business logic. When that happens, it needs to grab the SecurityContext check whether we authenticated, grab the userId, and figure out the set of roles and privileges (beyond the scope of this article) the principal has.

So, in short, we need something that creates the security context and stuffs it in a place where the AuthorizationService can grab it. Since we use co-routines on top of Flux, that place is the Flux Reactor Context and we want to get to that via the coroutineContext that is part of the co-routine scope all our logic executes in.

The filter

Spring Flux offers two ways to implement something similar to the good old ServletFilter, which is what you'd use when we were all still doing synchronous IO with Tomcat. One of these is called WebFilter, this appears to be the most useful of the two, since crucially it returns something called a ServerWebExchange, which in a somewhat convoluted way gives us access to the request Flux and the ability to interact with the Spring Reactor Context. The best way to think of that is as a ThreadLocal like construct for Flux where we can park custom data and access it downstream. Via the coroutines-reactor library, we gain a few feature to access this via the co-routine scope.

The other way to filter is via HandlerFilterFunction which looks like it's a bit more limited as it does not provide an obvious way to do anything with Flux (correct me if I'm wrong) but would be a better fit if you use the Spring's router DSL.

@Component
class AuthorizationWebFilter(val tokenService: TokenService): WebFilter {
    override fun filter(exchange: ServerWebExchange, chain: WebFilterChain): Mono<Void> {
        val jwtToken = tokenService.parseHeader(
            exchange.request.headers["Authorization"]?.firstOrNull())
        val context = tokenService.createSecurityContext(jwtToken)

        return chain
            .filter(exchange)
            .subscriberContext {
                it.put(FormationSecurityContext::class.java, context)
            }
    }
}

This is where Spring's API gets a little weird. This reads different then what it actually does and this threw me off for an while. The key here is that you call subscriberContext on the return value of .filter(exchange). To me this reads like: first do the request logic and then mess with the context. Luckily, what it does is different and the context gets modified before the logic kicks in. Just a bit of API weirdness.

The put method is weirder. Especially in combination with how we are getting values from the reactor Context. Intellij suggests a type of Any for both key and value. This is a lie and just where the Java type system fell a bit short, I guess. The correct types for this are Class<T> and T. So, it's a map indexed by the class of the value. In our case that would be FormationSecurityContext.

A final gotcha is that unlike most Map implementations, put does not manipulate a Context but creates a new one. I initially had this because I assumed put did not have a return value.

 subscriberContext {
    // this is wrong!
    it.put(FormationSecurityContext::class.java, context)
    it
}

So, that looks like a deceptively easy bit of code but it was made hard by a lack of documentation, and Spring not following the principle of the least amount of surprise, which makes all this hard to discover.

Getting the value out on the other side

Now that we have our security context, we want to use it. For this I implemented a simple DSL to check auth in places where we need that. This is the Kotlin way and I prefer it over annotations and/or AOP based madness.

// ReactorContext is still experimental
@ExperimentalCoroutinesApi
@Component
class AuthorizationService(val roleRepository: RoleRepository) {
    /**
     * Runs the block if the authorization checks succeeed or throws a `NotAuthorizedException`.
     */
    suspend fun <T> authorize(privilege: Privilege,ownerId: String, block: suspend ()->T):T {
        val reactorContext = coroutineContext[ReactorContext]
        val securityContext = reactorContext?.context?
            .get(FormationSecurityContext::class.java)
        if(securityContext == null) {
            // should not happen; means our AuthorizationWebFilter is broken
            throw IllegalStateException("no context")
        } else {
            if(!securityContext.isAuthenticated) 
              throw NotAuthorizedException(AuthProblemCode.JWT_MISSING)
            // additional auth checks beyond the scope of this article
            checkAuth(privilege,ownerId,securityContext.user
              ?: throw IllegalStateException("no user"))
            return block.invoke()
        }
    }
}

To use this, you simply do something like this:

suspend fun getUserProfile(userId: String): UserProfile =
    authorizationService.authorize(UserProfilePrivilege.GET_USER_PROFILE, userId) {
        userRepository.findUser(userId)?.toUserProfile() ?: throw NotFoundException(userId)
    }

It's a suspend function because we are using this from Flux based flow. In this case, we are actually using the Expedia GraphQL integration for Spring, which is definitely beyond the scope of this article but quite easy to set up.

But if you weren't, you could do something like this to create an endpoint:

coRouter {
    GET("/user/{userId}") {
        ok().contentType(MediaType.APPLICATION_JSON)
            .bodyValueAndAwait(userProfileSerice.getUserProfile( it.pathVariable("userId")))
    }
}

The bodyValueAndAwait extension function takes our suspending function and turns it into a Spring Mono, so Spring Reactor does the right things with Flux.

Publish Kotlin multiplaform maven repo

Jilles van Gurp — Tue, 11 Aug 2020 12:55:59 +0000

A recurring topic on many projects for me is publishing private packages of some sort so that they can be used in other private projects. Historically this has always been a PITA to setup and involves either paying someone to provide you some SAAS solution or reserving a chunk of time to do devops. It's a reason I often dodge the issue by simply not bothering with maven repositories.

I code a lot of Kotlin in the last few years and that means I get to deal with gradle a lot. On many projects people dodge the need for private repositories by doing mono repositories with multi module gradle or maven repositories. I've always hated dealing with multi module gradle projects because things slow down a lot when you are waiting for builds to happen. As soon as you have n modules, everything now happens n times (compile, test, package, etc). And while build tools are great for making things repeatable, they do have the potential to suck the life out of you in terms of consuming all your time. Three minutes may sound like nothing but repeat that 10-20 times in a day and you have just lost an hour.

So, especially for things that don't change a lot wouldn't it be nice if you could park them in a separate project and just download the already compiled binary. The short answer to this completely rhetorical question is "well duh". That's where private maven repositories come in.

Multiplatform

Recently, Kotlin multiplatform became a thing. With Kotlin multiplatform, you cross compile Kotlin code to multiple platforms like IOS, Android, Linux, WebAssembly, Javascript, etc. using the Kotlin multiplatform gradle plugin. This makes a lot of sense if you are trying to reuse code between different platforms and a natural fit for this is extracting common code into a multiplatform library that you then need to put somewhere so you can actually use it. Somewhere like a private maven repository.

The last two weeks or so, I've been working on lots of new things since I joined Formation. One of those things is exactly this. We have android code and some server code written in Kotlin and are currently writing a lot more code. So, obviously Kotlin multiplatform has our interest since we want to do IOS soon as well. When I say "has our interest".

What I really mean is I was stuck in Gradle hell for the past week trying to figure out this stuff from scraps of misleading, outdated, incomplete, or flat-out wrong documentation, stack overflow posts, etc. This stuff is very new and immature and technically only available in Beta so far. So, this is not unexpected.

I've been trying to figure out an easy strategy to deploy multiplatform artifacts via a private repository. We started with Github Packages because we are on Github and are making full use of their freemium layer; which is actually really great these days. We pay 0$ for a Github organization with as many private repositories as we need, we get CI/CD via Github actions and I've even managed to use Github Packages. Sadly, Github packages has evaded all my attempts to make it work with Kotlin multiplatform. Given the awesome price tag, I still think it's pretty nice but it seems this is a dead end (for now) for multiplatform at least. I also tried Jitpack with a public repository and had issues with that as well.

GCS based repository

We currently deploy some server stuff in Google Cloud which provided me with another false start in the form of Google Artifacts, which is a package repository with maven functionality that is currently in some alpha release and therefore not something you can actually use yet. That's a bit of a bummer because presumably Google is going to be all over Kotlin multiplatform given their Android involvement (or they should be, big corporations have trouble with doing logical things like this).

So, two potential candidates for a private repository down, I was getting a bit frustrated until I remembered that I got some mileage out of setting up maven repositories via ssh some years ago (while I was still using maven) and more recently using an s3 bucket (also on a maven project). That led me down a rabbit hole of "I wonder if I can do something with Google Storage for this ...".

In short, yes. Google storage is a lot less popular than S3, which caused me a few headaches piecing together what I actually needed to do. But eventually I figured it out:

plugins {
    // we're using the multiplatform plugin
    kotlin("multiplatform") version "1.3.72"
    // we want to publish our jars
    id("maven-publish")
}

// first you need set up your publishing repository
publishing {
    repositories {
        maven {
            url = if(publishLocal.toBoolean()) {
                // great for testing
                // gradle publish -PpublishLocal=true -Pversion=0.42
                uri("file:///${projectDir}/build/localrepo")
            } else {
                // this is what we do in github actions
                // GOOGLE_APPLICATION_CREDENTIALS env var must be set for this to work
                // either to a path with the json for the service account or with the base64 content of that.
                // in github actions we should configure a secret on the repository with a base64 version of a service account
                // export GOOGLE_APPLICATION_CREDENTIALS=$(cat /Users/jillesvangurp/.gcloud/jvg-admin.json | base64)
                uri("gcs://insert-your-bucket-name-here/releases")
            }
        }
    }
}

repositories {
    // if you want to depend on jars from your repo, add it like so
    maven { url = uri("gcs://insert-your-bucket-name-here/releases") }
    mavenCentral()
}

kotlin {
    jvm {
        // ...
    }
    js {
        // ...
    }
}

Just add that to your build.gradle.kts file. The multiplatform plugin actually integrates well with the publishing plugin so things should just work without further configuration. You may need to set artifactId, groupId and version somewhere of course.

The tricky bit is getting credentials to the plugin. Basically, see the comments for that. You need a service account credentials file stored locally and the GOOGLE_APPLICATION_CREDENTIALS environment pointing to that. I did this via the console but you should be able to do this via the command line if you prefer.

For good measure, I also added a local repo so I can test it actually does the right things before creating a mess in my bucket.

Setting up CI to do this automatically

Then the next issue is setting up a github actions workflow to do this for us:

name: Publish package to GitHub Packages
on:
  release:
    types: [created]
jobs:
  publish:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - uses: actions/setup-java@v1
        with:
          java-version: 1.8
      - name: Save google token
        run: echo "${{ secrets.GOOGLE_CLOUD_KEY }}" | base64 -d > ${{ github.workspace }}/google_tok.json
      - name: Publish package
        run: gradle -Pversion=${{ github.event.release.tag_name }} build publish
        env:
          #          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          GOOGLE_APPLICATION_CREDENTIALS: ${{ github.workspace }}/google_tok.json

Basically, the way this works is that whenever I tag a release via Github's releases feature, it triggers this workflow to publish a set of jars to my gcs bucket. I simply added my token in base 64 as a secret to the github project:

cat mytoken.json | base64 | pbcopy

Then the first workflow step unpacks that and puts it into a file in the workspace. The second one adds the GOOGLE_APPLICATION_CREDENTIALS variable.

Conclusions

Setting up a private repository like this is easy and cheap. It's was a PITA to piece together the relevant bits of documentation; so I wrote up an article documenting this both for my future self (I'll likely use this again) and for others to benefit from. I've been using this on two internal projects for a few weeks and we are currently using the artifacts in our Android project. Soon, we'll likely start experimenting with IOS as well.

The same instructions should also work for S3, although the credentials for that are a bit easier to deal with (just grab your key and secret and put them into Github secrets and use them as is).

Using Pandoc to create a Website

Jilles van Gurp — Mon, 25 May 2020 19:16:50 +0000

I think I created my first web page around 1997. I was studying at the University of Utrecht at the time. There was a brief period around 1994 where the using the World Wide Web meant queueing for the single unix terminal with a gateway to the internet & Mosaic installed after lectures. This of course soon escalated. By 1997, all computer lab terminals (a mix of Sun and HP unix machines, and later some windows PCs) had a browser. It was around then that I put up a little HTML page on the faculty web server announcing my presence to the world.

In the years after, this thing migrated to different places and around 2002 I registered my domain jillesvangurp.com (I was already too late to register jilles.com). Around the same time, I got into blogging. First with something called pivot, which I swapped out for Wordpress a couple of years later. For the past 16 or so years I've been using that to host my website. Recently, somebody pointed out that my web page was broken. Basically, my hosting provider did some infrastructure changes and this messed up https & http. Kind of embarrassing and it annoyed me that I had to spend time on fixing it.

So, I decided that enough was enough and it was time to retire Wordpress. This had been on my TODO list probably for the last five years or so. But it was one of those things that I never got around to. Over time, I've grown more uncomfortable with the notion of running a mess of php that is regularly in need of security updates and generally a vector for having your website defaced. Also, I like to write in markdown and Wordpress seems to insist on not storing that natively and/or mangling it.

After a survey of different tools for static website generation, I decided to keep it simple. There are a lot of these tools out there but they all seem to be a combination of opinionated and convoluted for what I want. And I got annoyed with their documentation. A problem I see with this type of tools is that the happy path of using them is well documented but basically only does less than half the job that you need done. Piecing the rest together has about the same complexity as just rolling your own scripts.

So, I picked the simplest tool that gets the job done: pandoc. Pandoc is a nice command-line tool to convert different text file formats to other formats. I've used a few months ago on my Elasticsearch Kotlin Client to generate an epub version of the manual. Crucially, it supports the Github flavor of markdown as input and html5 as the output. Even better, it can process code samples, do syntax highlighting, and some simple templating. That's all I need. I can do the rest with bash.

So, I started hacking a few bash scripts together that I've been adding features to in the last weeks. At this point, it's good enough for what I need. Of course the whole set up is highly tailored to my needs but that might be good enough for others as well. So, I decided to share the source code on Github. If not, the scripts are simple enough to figure out that you can probably fix it to do whatever you need. Feel free to fork and adapt.

Code

Here's a script that I wrote to convert markdown:

#! /usr/bin/env bash
for page in $(ls pages); do
  pandoc --from markdown_github+smart+yaml_metadata_block+auto_identifiers "pages/$page" \
    -o "public/$(basename $page .md).html" \
    --template templates/page.html\
    -V navigation="$(cat navigation.html)" \
    -V footer="$(cat footer.html)"
done

This simply generates html for each of the pages in the pages directory and uses the page.html template. This in turn has a few variables that it mostly gets from the markdown meta data section and I've added a few on the command line. Nice and simple.

I have a similar script that processes articles that I migrated from my old Wordpress setup (in the articles directory). Fixing the exported markdown was hard as Wordpress export makes a mess of that. But I got it done in the end with a lot of patience (and some regex replacing).

Since running the above script takes quite long on 300+ articles, I created another script with a bit of hackery to fork processes with bash:


for blogpost in $(ls articles); do 
  export publishdate=$(echo $blogpost | grep -oE '[0-9]{4}-[0-9]{2}-[0-9]{2}')
  pandoc --from markdown_github+smart+yaml_metadata_block+auto_identifiers "articles/$blogpost" \
    -o "public/blog/$(basename $blogpost .md).html" \
    --template templates/article.html   \
    -V publishdate="$publishdate" \
    -V navigation="$(cat navigation.html)" \
    -V footer="$(cat footer.html)"&
done
for pid in $(jobs -p); do
    wait $pid
done

Note it has a little & on the pandoc command. This causes bash to fork a process. Then after all processes are forked, I wait for all the jobs to finish. Takes about 10 seconds to process everything. Good enough.

Also note the -V options for setting variables, those are used in the template. In this case, I've added the publication date, which is parsed from the file name using grep.

For the sitemap, I simply gobbled together some nice find command and a few echo commands.

#! /usr/bin/env bash
sitemap="public/sitemap.xml"
baseurl="https://www.jillesvangurp.com"
timestamp=$(date -u +"%Y-%m-%dT%H:%M:%SZ")

function url() {
  echo "<url><loc>${baseurl}/$1<loc><lastmod>$timestamp</lastmod></url>"
}

robots=`cat <<EOF
User-agent: *
Allow: *
Sitemap: $baseurl/sitemap.xml
EOF
`
echo "$robots" > public/robots.txt

header=`cat <<EOF
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
EOF
` 
echo "$header" > $sitemap

for file in $(find public -name "*.html" | sed -e 's/public\///'); do
  echo $(url $file) >> $sitemap
done

printf "</urlset>\n" >> $sitemap

I also needed an index page for my articles:

#! /usr/bin/env bash
rm -f _index.md
export HEADER=`cat <<EOF
--------
title: Article Index
author: Jilles van Gurp
--------

Intro text omitted ...
EOF
`

echo "$HEADER" > _index.md

current_year="-"
for name in $(find articles -type f -exec basename {} \; | sort -ur | sed 's/\.md//'); do
  year=$(echo $name | sed -E 's/([0-9]{4})-([0-9]{2})-([0-9]{2})-(.*)/\1/')
  month=$(echo $name | sed -E 's/([0-9]{4})-([0-9]{2})-([0-9]{2})-(.*)/\2/')
  day=$(echo $name | sed -E 's/([0-9]{4})-([0-9]{2})-([0-9]{2})-(.*)/\3/')
  title=$(echo $name | sed -E 's/([0-9]{4})-([0-9]{2})-([0-9]{2})-(.*)/\4/')
  nice_title=$(echo "$year-$month-$day - $(echo ${title:0:1} | tr  '[a-z]' '[A-Z]' )${title:1}" | sed -e 's/-/ /g')
  if [ $year != $current_year ]; then
    echo -e "\n## $year\n" >> _index.md
  fi
  current_year=$year
  echo "- [$nice_title](/$year/$month/$day/$title)" >> _index.md

done

pandoc --from markdown_github+smart+yaml_metadata_block+auto_identifiers "_index.md" \
  -o "public/blog/index.html" \
  --template templates/article.html \
  -V navigation="$(cat navigation.html)" \
  -V footer="$(cat footer.html)"\
  -V year="$(date +%Y)"

# cp _index.md generatedindex.md
rm -f _index.md

This is the most complicated script so far; mainly because I wanted to support the old Wordpress link structure of year/month/day/title so as to not break external URLs. I have an .htaccess file with some redirects.

There's another very similar script that generates an atom feed for those people still using feed readers. I used atom for mostly nostalgic reasons as I was following the standardization of that while I was in Nokia Research. I think RSS sort of stayed more dominant and then both are sort of equally irrelevant nowadays. But it's still around and I guess they are still uses by various news aggregators so having a feed probably is a good idea from an SEO perspective.

Finally, I have a Makefile that invokes all my little scripts in the right order and all the output goes to a public folder. The last step uses rsync to copy everything over to my hosting provider. The whole process takes about 30 seconds; which is a bit on the slow side but acceptable.

Conclusion

Is this setup better than other tools? Probably not. But I have full control over what happens and it's simple enough that I can use and maintain it. It's good enough for me; and it gets the job done.

Note, I've added links to the actual files in my git repo. Inevitably, there will be changes on master but all the links point to a tag created for this article.

Using Cloudfront, S3, and Route 53 for hosting

Jilles van Gurp — Wed, 17 Jul 2019 15:41:06 +0000

Basic setup
Updating content
Cloudfront invalidations
Url rewrites using S3
Conclusion

In the past few months, I've retired our nginx server and replaced it with Cloudfront, s3, Route 53, and amazon issued certificates to simplify hosting www.inbot.io. This has been a bit more painful than I liked and there have been a few issues. This is mainly because the AWS documentation sucks and is fragmented over different products and generally represents a maze of misdirection.

Anyone from Amazon reading this, a helpful guid for somebody looking to combine your products for a basic use case like "Given a completely bog standard one page javascript website, this is how you host it on AWS" would be extremely helpful. Instead you have bits and pieces of documentation for each of your products leaving all the other bits and pieces as an exercise to the reader. I had to refer to Stackoverflow because every documentation page you land to seems to be lacking some crucial details. I wasted some serious amounts of time figuring out this most basic of uses for these products.

The good news is that it does work once you figure out all the workarounds in each of their products. The benefit is that it simplifies your infrastructure for a simple one page app. We have no self hosted bits and pieces. Additionally, a CDN ensures that your users have a good experience downloading your website from a fast CDN instead of hitting your poor web server on the wrong side of the globe. The bad news is that the AWS UI is a bit lacking in usability and flexibility and doing some common things that you would do in e.g. nginx are not that easy.

So, to avoid me having to google this together again, I'm documenting what I had to do to make this work. Also, others may find this useful.

Basic setup

First, thanks to Tiberiu Oprea for this extremely helpful overview. This will get you pretty far. I'm not going to repeat everything that he says there and will instead focus some extra stuff I had to figure out separately. AWS people reading this, use this website as a reference on how to properly document your product.

Assuming you followed his instructions to the letter, you would end up with a Cloudfront + s3 setup (just replace inbot.io with your own domain):

https certificates for inbot.io and www.inbot.io in the AWS certificate manager. One gotcha here is that it takes ages for Cloudfront to actually see those. Allow a few hours for this to be picked up. If you get it wrong, delete and try again. I lost half a day over this where I thought I was doing it wrong when in fact Cloudfront just is a bit stupid when it comes to actually asking for the current list of certificates. Eventually they showed up in the relevant drop down. Give it some time.
An s3 bucket with the domain www.inbot.io.s3-website-eu-west-1.amazonaws.com. This is where you deploy your static content. I suggest using AWS cli for this.
Another s3 bucket with the domain inbot.io.s3-website-eu-west-1.amazonaws.com. This bucket is configured to redirect to www.inbot.io. It has no content.
Two matching Cloudfront setups for both buckets. Make sure you redirect http to https in Cloudfront.
Two A records in Route 53 pointing for the domains with and without www at both Cloudfront setups.

Note that the bucket names are important for getting Route 53 to do the right things. So match the domain name in the bucket name and don't get creative here.

You can verify that everything redirects as required with a few simple curl commands:

# Cloudfront redirects to https
curl -v http://inbot.io 
# Cloudfront hits the S3 bucket and S3 redirects to https://www.inbot.io
curl -v https://inbot.io

If all this works alright, your browser will redirect http://inbot.io/whatever/path to https://www.inbot.io/whatever/path via two permanent redirects. So users typing in inbot.io end up on your website instead of a blank page, or worse, an S3 403 XML page.

This seems like the AWS product people needs to get spanked a little. This is literally one of the most basic use cases a user of their products might have which is to host a simple website and it totally sucks. There's literally no website in this world that would not want this out of the box without having redirects for https and handling domains with and without www. This should not require two separate buckets and Cloudfront setups. This is madness. But at least it works.

Updating content

One problem with Cloudfront is that TTLs are pretty long. This is helpful for caching things in a globally distributed CDN, but annoying when you need to fix a bug fix in your website and it takes hours or days for your users to actually get the fix. Addressing this requires some planning.

If you use something like Webpack, you should ensure the file names are hashed so this does not matter for most files. This leaves you one file that is still a problem: index.html. Since in our case this file is small, I ended up disabling TTL for this.

In our CI build we use AWS cli to interact with AWS. We use this command to upload our index.html like this:

aws s3 cp ./build/index.html ${S3_BUCKET} \
  --acl public-read \
  --cache-control max-age=0,no-cache,no-store,must-revalidate

S3 will respect these headers and start serving your new index.html as soon as you fix something. Cloudfront passes these headers to the browser as well.

For the other files, we use a reasonable TTL. We have a bunch of static files that rarely change and lots of webpack hashed artifacts that generate new files.

One gotcha here is to not use aws sync --delete. The problem here is that if you delete a file from s3 and the user still has an old index.html pointing to the old file hash, they will now run into 404s until they force reload the page in their browser. And of course assuming they even know how to do this; this poses some unique challenges on mobile.

Cloudfront invalidations

You can force Cloudfront invalidations when you deploy new content. The main file to invalidate is index.html. This ensures that Cloudfront updates the CDN nodes world wide within a few minutes instead of doing it much slower after the next user tries to load the file for the first time. So, within minutes after updating this any browser that reloads our page, will hit Cloudfront to get the latests because it forwards the max-age=0,no-cache,no-store,must-revalidate and then get the latest version.

aws Cloudfront create-invalidation \
  --distribution-id ${CLOUDFRONT_ID} \
  --paths / /index.html /app/index.html

Of course keeping your index.html small will be helpful since loads from s3 are going to be a bit slower than cache hits from Cloudfront. This is OK since the bulk of our content is in the javascript and other files.

Url rewrites using S3

We had a few url rewrites in our nginx. These caused us some headaches until I figured out how to use S3 Routing Rules. Part of the problem here is that we had some links that we distributed to users that broke without us knowing because things got lost during the redirects. For example, we have a reward program for referring new users. They click on a link that used to go to http://inbot.io/join/XYZ (we've since fixed that to at least go to https) where XYZ is a referral code that needs to be passed into the javascript that constructs the signup form on our website so it can be passed to our server. To redirect this, I added a rule to our S3 bucket's static content properties:

<RoutingRules>
  <RoutingRule>
    <Condition>
      <KeyPrefixEquals>join/</KeyPrefixEquals>
    </Condition>
    <Redirect>
      <Protocol>https</Protocol>
      <HostName>www.inbot.io</HostName>
      <ReplaceKeyPrefixWith>app/index.html#/join/</ReplaceKeyPrefixWith>
    </Redirect>
  </RoutingRule>
</RoutingRules>

Note, the protocol and hostname are essential, otherwise S3 will happily redirect you to the bucket url without SSL.

You can also use RoutingRules to fix nicer http errors for e.g. 403s that S3 throws when it can't find an object (how is that not a 404?). One of these days, I'll probably invest some time in using AWS cli or Cloudfront to do this in one command but I ended up clicking all of this together in the AWS ui.

So this fixed our issues. Note we are redirecting to app/index.html#/join/XYZ. I decided to get rid of our subdomain for the web app and simply host everything under www.inbot.io. Inside our web app, we use # paths (aka anchors) and everything is handled by our javascript.

CORS header for our stellar.toml

Another issue we had that was that we have one file on our site that needs to have CORS headers set correctly. The file in question is the stellar.toml file that Stellar uses to figure out meta data about our cryptocurrency, the InToken. The how and why is not important but setting up CORS is a common requirement and something that is straightforward in nginx. This one had me pondering for a while. S3 does not provide a good solution for this. However, it turns out that in Cloudfront you can configure so-called cache behaviors for specific paths. In our case we added a behavior for /.well-known/stellar.toml, with the custom header. This location is where Stellar clients expect to find the meta data file. And since a lot of Stellar clients are browser applications, they need the CORS headers to be set correctly. If you want to read more about this, refer to the Stellar documentation.

Conclusion

This setup is working pretty OK. We have a simple CI job that does most of this. I did use the AWS UI to click together the buckets and Cloudfront setup. You could probably use e.g. Cloudformation or some aws cli incantations for this but since this is a one time thing, I did not bother to automate this. The flip-side of that is that it is kind of a long process with many steps and quite easy to mess up.

Also, I'm not completely happy about having to do a lot of things in the AWS UI (painfully unusable). However, I can't bring myself to automating this hopefully one time setup. There are diminishing returns when it comes to automating this stuff.

Libra, blockchains, and the meaning of it all

Jilles van Gurp — Tue, 25 Jun 2019 12:43:47 +0000

Recently, Facebook announced their plans for launching their own blockchain platform and crypto currency, the Libra. As I have done the technical work for Inbot to launch our own token on the Stellar blockchain, I'm of course very interested in what they are doing, why, and how.

What problem does Libra solve?
Byzantine Consensus vs. PoS & PoW
Smart Contracts
Libra consortium
Privacy, KYC, and Financials
Why is Facebook creating a new Blockchain?
Monetization
What happens next?

In this article I want to explore what I think is happening and what it all means. As such, it is severely at risk of being both late and somewhat redundant as world plus dog seems to have an opinion on this. In writing this article, I want to reflect a bit on the bigger picture and also intend to distance myself a bit from the alarmist and populist hyperbole that has been circulating since Facebook announced Libra.

This is going to be a longish article. In short, I'm cautiously optimistic that Facebook is serious about Libra and is doing exactly the kind of things technically, legally, and practically that they ought to be doing to make this a success for them. Of course, I cannot vouch for their motivations; though I would assume it involves enriching themselves in some way.

First, let's get a few things out of the way. In debating this with others, I've noticed a lot of negative sentiment around the topic of crypto currencies, blockchains, and Facebook. A lot of this is for valid reasons and I partially share those sentiments. I'm a natural skeptic and despite my technical involvement with a blockchain based product, I'm not actually active as a crypto investor. I tend to look at this topic from both a technical and pragmatic angle. I don't consider myself a dreamer, utopian, or otherwise idealistic person.

Another part of this negativity is simply ignorance and people assuming they have a firm grip on this when arguably they don't. Luddites have been there at every step of the way from the invention of the wheel, to the industrial revolution (where the term Luddites originates), the invention of radio, television, the internet, etc.

It's good to be skeptical and conservative but it would be foolish to dismiss Libra as just a fad. It won't blow over and whether you like this or not it looks like blockchains are here to stay in one form or another. For us, developers, technologists, etc. the main thing is to make sense of what is happening and upgrade our skill sets so we can make ourselves useful gluing together all the new bits and pieces of technology.

I think of the Libra announcement as a symptom of the entire market maturing and a necessary step for crypto currencies to become more mainstream. It's a combination of necessary, not unexpected, and inevitable for this to happen. More companies will follow and probably very soon now that there is a sense of urgency to not let Facebook 'win'.

Disclaimer I use the term blockchain loosely in this article and some people might take offense with that. Blockchains or blockchain like platforms share a few useful properties of having some cryptographically way of storing and protecting a record of transactions. For the purpose of this article I'll use the word blockchains to refer to that type of system.

What problem does Libra solve?

Simply put, the existing financial system has been around since the late 1600s. It has served us well and it is ingrained in our legal systems and we're used to it.

However, it has issues. One of those issues is the level of control that resides in the hands of the banks and governments that control them and occasionally abuse the powers given to them. Another is the built in inefficiencies through our reliance on mechanisms that either predate the invention of computers (e.g. paper work) or date back to the early days of the invention of that. Additionally, poorly aligned incentives to be either cheap or fast cause banks to profit from delaying transactions and allow them to exploit their controlling position by e.g. charging excessively for transactions, denying service to some, or otherwise benefit from their exclusive position. Banks make stupendous amounts of money for essentially obstructing financial traffic. A notable thing about the Libra announcement is that banks do not feature at all in the consortium. This is not an accident. Libra is intended to make them redundant.

Blockchains provide a solution for administering ledgers, i.e. a record of transactions, that is by design impossible to cheat (assuming no bugs, compromised algorithms, etc.). Using a blockchain, much of the need to have banks goes away. Ledgers are of course nothing new. The ancient Sumerians had them and much of their remaining written material consists of ledgers. Same for the Egyptians, Romans, etc. Ledger technology is ancient.

The modern financial system invented in the 17th century introduced the notion of banks, central banks, stock exchanges, etc. that are powered by ledgers. Additionally, it introduced the notion of bits of paper to represent money, an IOU by a bank or central bank that the bit of money is being administered correctly. The integrity of currency is core to our modern financial system and for the past century it has relied on governments and central banks overseeing the way people administer their ledgers. Necessarily, this involves independent bookkeeping, elaborate checks and balances when transacting, and rules and laws for conflict resolution when somebody inevitably cheats. Blockchains internalize much of the manual overhead by making it impossible to cheat. It removes the need for people to keep independent ledgers and thus reduces cost and friction associated with financial transactions.

Contrary to the popular belief, a simple database is not a drop in replacement. We've already had those for the last few centuries. First in paper form and for the last century or so in mechanical (the M in IBM stands for machines) and then digital form (i.e. a database). Blockchains have one key feature: they allow mutually distrusting users to have confidence that none of them is able to cheat or commit fraudulent transactions.

Not only is it cheaper to use a block chain but it is also faster. A good blockchain platform can transact in seconds or even faster. After that, the transaction is guaranteed to have happened. Additionally, because the cost is so low, it is possible to do very small transactions. This enables a whole range of use cases that is currently mostly outside the financial system.

Indeed, as Facebook notes in their marketing material, there are still billions of people that lack a bank account. Most of their business is conducted using cash, bartering, or a range of non block chain based payment solutions that have emerged to address this.

Byzantine Consensus vs. PoS & PoW

From reading the whitepaper, it looks like Facebook is broadly copying the approach used by the likes of Stellar (which I'm very familiar with), Ripple, and a few others. Stellar is using a consensus model where validators agree on which transactions to accept. This process is fast because it does not involve e.g. mining and it awards a special status to the owners of these validators in the sense that they control what happens in the network.

Libra is implementing a so-called permissive variation of Byzantine Fault Tolerant (BFT) algorithms. Permissive BFTs are emerging as a third alternative to the more established permission less BFTs like Bitcoin and Ethereum. There are two variants of this: Proof of Work (aka. mining) or Proof of Stake based networks. Respectively the three approaches accept transactions based on either who you are, what you did (PoW.), or how much you own (PoS.).

The way consensus works in permissive BFTs like Stellar is that if you choose to run a validator, you must configure a list of other validators that you trust and define a quorum of hpow many of those need to agree for a transaction to be valid for your node. Transactions on your validator are only accepted if there is consensus with the nodes you list and the validators they trust. This extended consensus network means that e.g. 50% attacks common in permission less network are very hard to perform unless you manage to hijack existing validators that are already trusted. It also means that as the number of validators that trust each other directly or indirectly, the network becomes more decentralized and it becomes harder to cheat. Facebook is following a similar model.

Like Stellar, Libra will launch with a consortium of validators and they intend to open up to more third party validators later. As the number of validators grows, the network will be more resilient against attempts by any of them to control the network. While anyone can launch a validator, the only validators that matter are those that are trusted by the existing ones. This is what it means to be permissive. Think of it as an invitation only kind of thing where whether you get an invite will be based on your merit, reputation, and trustworthiness. In the Stellar network an important driver for getting others to trust you are also factors such as your uptime, business relations with other validator owning organizations, etc.

So, it would be wrong to characterize this as centralized where e.g. Facebook decides on who gets to join their network. This would be an oversimplification that seems to come up regularly in discussions of how Stellar and Ripple work and how Libra will work. E.g. Stellar actually has a growing number of validators that have to agree with each other. When there is no agreement (i.e. consensus), the network stops making progress.

This actually happened recently when several validator operators decided at the same time to do some maintenance and took down some of their nodes. As a result, the remaining nodes failed to find enough nodes to agree with and stopped accepting new transactions until the nodes came back online. While this sounds bad, this is actually a good safety feature. Stellar will choose halting over forking. Nobody lost their money. In their response to this incident, the Stellar Foundation actually made clear that they want to reduce the importance of their own validators in the network and identified this as a root cause for the failure. Their stated goal is to be able for their own validators to be taken offline without halting the consensus.

Similar Incidents with e.g. Ethereum instead resulted in unintentional forks or partitions (as opposed to intentional forks that also happen) where transactions on the wrong branch of the fork were lost. To mitigate against this, it is common to wait for several blocks before assuming a transaction has happened. Consequently, transactions can take very long to complete in Ethereum and Bitcoin.

In CAP theorem terms, permissive Byzantine consensus favors Consistency and Partition Tolerance over Availability where permission less PoS and PoW based systems tend to sacrifice consistency instead. It's the key reason why Facebook picked this mechanism because it is both safe and fast and therefore meets their requirements of being suitable for their intended use case. Also, the notion of not having their network invaded by hostile validators is likely more of a feature for them and not a bug.

In terms of the existing financial system, what Facebook is doing is essentially a more efficient digital equivalent where the same types of entities responsible for making transactions happen now facilitate this by running validators that do this automatically. The same properties that made that good enough, also make permissive BFTs good enough provided the network has enough independent validators to be resilient against some of them misbehaving. Inevitably, there are going to be a lot of politics around this topic; just like in the existing financial system.

IMHO there will remain a role for permission less systems as stores of value and the interaction between different blockchain networks and diversity of platforms that mutually integrate with each other is going to be key to the long term robustness of the ecosystem.

Smart Contracts

Unlike Stellar and Ripple, Facebook has smart contracts. These are similar to what you'd find in e.g. Ethereum (i.e. solidity). However, there are some differences. Hackernoon has written a great overview of the essentials. Move, the language and vm in Libra provides all the essentials to provide strong guarantees about correctness and integrity.

The language, vm, and module system are designed to facilitate formal verification. This is very important because it allows for contracts and modules that are provably correct. This greatly simplifies auditing.
Unlike Solidity, the main smart contract language in Ethereum and similar blockchains, it does not support dynamic functions. This further simplifies reasoning about and auditing Move contracts.
Auditing has proven to be a significant hurdle with Ethereum. When Inbot was preparing an ICO early 2018, we were indeed preparing an ERC20 token. Ultimately we were to late and the market had already crashed by the time we were getting ready. Part of our concern was significant cost for auditing and a nagging concern that there might be bugs in our contracts still. Smart contract bugs are no joke and can have significant financial consequences.

The lack of smart contracts in Stellar is a feature, not a bug. Instead of smart contracts, Stellar (and Ripple) provide a set of primitive transaction types that you can combine to build financial products. However, it is inherently more limited than a full blown contract language. Libra actually makes a nice compromise here. By providing a language and module system with strong verification mechanisms, it allows for an ecosystem of verified modules to emerge that users may combine to build smart contracts. So, auditing should be similarly simple as Stellar for the basic use cases while still allowing for more complicated contracts to be written.

Libra consortium

A blockchain is only as good as the people backing it. Facebook has gathered a strong consortium with major companies in it. Particularly, the presence of major existing payment providers like Visa, Mastercard, and Paypal, suggests that adoption could be quite rapid. Another thing worth pointing out is that these companies are competitors and that the financial stakes are very high for them. This further strengthens the argument that the consensus model that Libra is taking here is not about centralizing power but about facilitating mutually competing entities like this to collaborate.

I imagine many governments currently considering Facebook's actions regarding this, will be scrutinizing these arrangements. I think it is safe to assume that Facebook has considered this and that much of the technical architecture is actually intentionally trying to provide strong guarantees here and aimed to preempt a lot of the concerns.

The stated intention of Facebook is to create a legal entity called Calibra that will presumably control the platform. Crucially, this legal entity is a direct subsidiary of Facebook and will presumably control the financial reserve, intellectual property (i.e. patents), and technical direction. I am somewhat surprised that Facebook has chosen this path instead of setting up a foundation that is controlled by its members (like e.g. Stellar, Ethereum, and other blockchain platforms have done). This suggests that Facebook will have a special position in the platform.

Privacy, KYC, and Financials

Facebook is under a lot of pressure from the public, press, and increasingly legal authorities over their lack of regard for their own user's privacy. A blockchain is by its nature very public and it is unclear how they are addressing privacy; despite their solemn promises that user transactions will be private.

Potentially all the transactions of all their users would end up being public domain information and certainly Facebook would be gathering a lot of financial data about their users. This obviously raises some red flags for those concerned with their privacy. The ability to trace back user payments to ads is potentially a gold mine for Facebook.

To comply with legal requirements, Facebook will have to apply AML and KYC policies. Arguably, they already know a lot about their users as it is today, which gives them a strong starting position. Requiring documentation in the form of selfies, copies of identity documents, and proof of address, will put them in a unique position of controlling a vast amount of very private data on their billions of users worldwide.

Worse, a centrally controlled platform like this is also of interest to intelligence agencies; and potentially a tool that may be used to control citizens. E.g. WeChat is a very popular payment platform in China and the Chinese government routinely punishes citizens by banning people from it; which severely limits their ability to do financial transactions in China because WeChat has largely replaced the use of traditional money. In creating Libra, Facebook seems to want to compete directly with platforms like that.

Why is Facebook creating a new Blockchain?

Many users of existing blockchains may have raised a few eyebrows about Facebook building their own platform. However, I think I have a very rational explanation: none of the existing platforms has a level of maturity that would make it suitable for operating at the scale Facebook needs it to run.

Billions of users using Libra means an absolutely huge transaction volume. I doubt many of the established blockchains are anywhere near ready for such volumes. Both Ethereum and Bitcoin are as of yet still mining based and limited to a world wide transaction volume that ranges in the single, or at best, double digit numbers of transactions per second. Existing traditional payment platforms can handle thousands or tens of thousands of transactions.

Additionally, these are still limited to relatively large transactions as there are transaction fees involved. Facebook seems to intend to have a very low transaction cost in order to facilitate very small transactions and micro payments as well as traditional payments. E.g. they seem to highlight the lack of good payment solutions for third world countries where many people do not have banks or credit cards. All, this suggests that Facebook has a need to support a very high volume of transactions at a very low cost.

So, clearly there is more going on here than just a not invented here syndrome. Facebook inventing their own platform of course gives them control and as pointed out there are plenty of reasons to not trust them with that level of control. But it is not like there is anything out there that they could have just taken and used.

Monetization

Obviously, Facebook is a very valuable company and they are looking to make money through their platform. By design, transaction fees are going to be low. So, it would be wrong to assume that is going to be the primary revenue driver. Instead, it is likely that Facebook is looking to leverage their position as the controlling entity of Calibra to find alternative revenue streams.

They are creating an exclusive ecosystem that generates a lot of data. Access to this ecosystem will likely require buying into it. Additionally, controlling the large reserve that is needed to stabilize the Libra means an opportunity to make lots of money from simply investing these funds. Finally, day to day fluctuations in exchange rates and conversions to and from existing currencies create very lucrative opportunities to make money.

Finally, the value of Facebook as a holder of KYC and AML information means that they can provide guarantees to others about whom they are dealing with. As governments will inevitably want to crack down on money laundering, financing of terrorism and tax dodging, access to this information is going to be crucial for anyone looking to do business via their platform. The alternative of doing KYC and AML in house is likely to not scale and Facebook is uniquely positioned to provide very high quality information through their existing relationship with their billions of users.

What happens next?

I expect Libra will trigger a lot of scrutiny. But, it will also act as a wakeup call. So far, blockchains have been operating in the margins of the financial and legal system. A combination of libertarian utopianism, greed, anarchism, etc. seems to be what has been driving things forward. Lately, there have been some increasingly serious attempts by e.g. IBM with World Wire, Bit Bond, WireX, and similar efforts to use blockchains for payments, securities, etc. However, most of these products are still relatively new and limited to low transaction volumes.

With Libra, Facebook will overnight create a new reality with potentially hundreds of millions of people getting involved directly or indirectly. This will represent a sharp increase in the use, and importance of blockchains for the financial plumbing of the world. We're now about to hit the elbow in the exponential curve where this will go from relatively small numbers to massive numbers.

I expect that Facebook will get some competition very shortly and fully expect most of the tech giants are already in an advanced stage of planning counter moves. I'm talking about the likes of Apple and Google, both of which operate mobile phone based payment platforms; and Microsoft, Amazon, traditional banks, and in some cases small governments of nations not so eager to have their financial infrastructure be hijacked by any of these.

In short, I expect that we will have a frenzy of investments, speculation, and R&D activity around this topic over the next year. Anyone that matters in this space is going to want in on the action.If they were interested before, they will now be accelerating their attempts. None of these companies will be happy to sit back and watch Facebook eat their lunch.

Additionally, I expect e.g. the US and EU to have a thing or two to say about financial oversight. Short term, Libra seems to be well positioned to dodge some of this. However, clearly, there is already widespread support for legal action against Facebook and they risk further scrutiny with this announcement. In a way, they are forcing the agenda early, which I think is actually smart because it catches everyone unprepared.

My impression is that governments are too slow to respond in a timely fashion and that by the time they will be ready to legislate, there will be several too big too fail type platforms already as well as powerful lobbies that will aim to prevent them blocking this. Time is running out quickly and inaction just means the existing laws continue to apply.

Assuming, Libra gets through this initial phase, we will likely have several new blockchain based payment platforms starting to compete with each other within the next 1-2 years, each with large volumes of transactions and vast amounts of money flowing around.

One interesting thing here is that this will inevitably also create a need for inter blockchain traffic via exchanges. E.g. the Stellar platform seems to be already emerging as one of the platforms of choice for this kind of traffic. IBM's world wire is using it, several exchanges connect to it (or rather make use of its built in decentralized exchange), and several fintech companies are operating stable coins for both crypto and traditional currencies on it. The simplistic view of "there can be only one" is in my view not justified. There will be at least several platforms and they will inevitably be highly integrated with each other.

In summary, I believe that Facebook's moves with Libra are logical and very smart. For their level of ambition, Libra is the right platform to be building technically and they seem to have a strong consortium with which they are launching this.

I also believe that this actually strengthens the blockchain ecosystem and that it will lead to a lot of new things. Whether Facebook's Libra emerges as the dominant solution in this space is very much undecided. If that were to happen it would be because of a lack of initiative and vision by their competitors. Personally, I don't believe in this doom scenario and instead they will be one of several platforms that remain standing after the dust settles on this in a few years.

Explaining Stellar using Cliste

Jilles van Gurp — Wed, 17 Oct 2018 17:35:03 +0000

Origins of Cliste

A few months ago, we decided to create a token on Stellar at Inbot. So, I read up, explored the sdks and started trying to figure out how everything works in the Stellar world.

Since I primarily use Kotlin these days, I decided that I wanted to adapt the official stellar sdk for Java and make it a bit more kotlin friendly. Kotlin and Java play really nice together so this basically boiled down to me creating a new Github project called inbot-stellar-kotlin-wrapper.

Initially I was just fooling around and trying to figure out different parts of the Horizon API. I figured out how to run a standalone chain and wrote some code, added some tests, etc. After a few days I realized that

this stuff is easy
I really would like to use the command line instead of writing code and tests all the time.

So, I googled for a suitable command line argument parser for Kotlin and stumbled on com.xenomachina.kotlin-argparser. It's a really nice kotlin dsl for picking apart commandlines.

Next steps were basically, coming up with a nice name: Command Line Interface for STellar, aka. cliste. At this point, I've been working on this project on and off for about 2 months. It's getting to a state where it is pretty useful.

Explaining Stellar using Cliste

Now that I have cliste, I can show some simple examples to explain how stellar works. Instead of giving you the usual marketing cliches, bad metaphores, and other verbose ways of communicating stuff, I'll highlight some key features using cliste:

Running a standalone stellar

If you want to fool around a bit, firing up a standalone chain is the best way.

docker run --rm --name stellarstandalone -p "8000:8000"  stellar/quickstart --standalone

It will take a second to start. Since we pass --rf all your data will be lost if you kill the container. So, you can use this as a sandbox. You can swap out --standalone for --testnet or --public to target the testnet or public stellar net instead.

Creating accounts

Lets create an account and give ourselves some XLM. In stellar, accounts must have a minimum balance of 0.5 XLM. On the standalone chain in the quickstart image it is still 20 XLM though. So, to be safe. We give alice plenty.

Since we are on a standalone network, we'll have to create an account from nothing. This works as follows.

$ ./cliste createAccount alice 1000
17:35:56.715 [main] INFO io.inbot.kotlinstellar.KotlinStellarWrapper - using standalone network
17:35:58.166 [main] INFO org.stellar.sdk.KotlinExtensions - 7 07067baf191a516c6434c84c6232c3672dd179883451641b417fb3090bc82d08 success:true fee:100 CREATE_ACCOUNT
17:35:58.166 [main] INFO io.inbot.kotlinstellar.KotlinStellarWrapper - created GBJR3JH4ZC5LZKGM2PPSVMJFPPWREVEKD4TQJL2RMUHI75P35N4JVABC
created account with secret key SCBB6IXATC2RFKCFIFUUIXGD5QDHRYAEQZBDDDMPRO3PVYXEMETCC67F

What just happened here:

we generated a new key pair
it was stored in keys.properties under the key alice. This allows us to use that as an alias in future commands.
we gave ourselves some XLM. We can do this because we know the seed of the chain.
it logged some details, like the private key

This won't work on testnet but you can use the friendbot instead. On the public net, the only way to fund new accounts is through an exchange or via an existing account.

Alice can create more accounts now

Now that alice has a valid account, she can create an account for bob without and fund the base XLM balance for bob herself.

$ ./cliste -a alice createAccount bob 50
17:40:07.888 [main] INFO org.stellar.sdk.KotlinExtensions - 57 e580f9e898a7151000e0228500d17846165aebcfed405411167cf5b1f41dd6b4 success:true fee:100 CREATE_ACCOUNT
17:40:07.892 [main] INFO io.inbot.kotlinstellar.KotlinStellarWrapper - created GAURP3FQ56BOF2PXF5DCLFS4QOJPTO5XM62G2AAMAQXB4PZWYNSS4RKA
created account with secret key SCYD6JC5GHFW63T5KTNVOT5FFJIW2R67VBEHFYIQZCMS2VLLBAKUI4YW

So we simply add alice's key to the command with -a and define a new key.

Managing accounts

If you want to know what keys you have you can list them as follows:

$ ./cliste listKeys
Defined keys (2):
alice: secretKey SCBB6I.... accountId: GBJR3JH4ZC5LZKGM2PPSVMJFPPWREVEKD4TQJL2RMUHI75P35N4JVABC
bob: secretKey SCYD6J.... accountId: GAURP3FQ56BOF2PXF5DCLFS4QOJPTO5XM62G2AAMAQXB4PZWYNSS4RKA

The secret keys are abbreviated here.

Note, you can also manually add keys in keys.properties. You can also mix public keys and private keys here. So, you can use this like an address book for public keys that you car about. You only need private keys if you are going to do transactions for the account.

Be careful storing private keys you care about here. This tool is intended as a development tool only. I may add some more suitable protection in the future but right now it is all plain text. In a nutshell, don't do anything with this that I would not do either.

Payments and balances

Lets give bob some more XLM:

$ ./cliste -a alice pay bob 10 XLM
17:44:12.535 [main] INFO org.stellar.sdk.KotlinExtensions - 106 703c34d98162f5223805cdd54a48261ae73e0dae8d5fccc76caba59d492f228a success:true fee:100 PAYMENT

Bob and alice can each check their balance:

$ ./cliste -a alice balance
accountId: GBJR3JH4ZC5LZKGM2PPSVMJFPPWREVEKD4TQJL2RMUHI75P35N4JVABC subEntryCount: 0 home domain: null

thresholds: 0 0 0
signers:
    GBJR3JH4ZC5LZKGM2PPSVMJFPPWREVEKD4TQJL2RMUHI75P35N4JVABC 1
authRequired: false
authRevocable: false

Balances:
XLM b:939.9999800 l:- - sl: - - bl: -

$ ./cliste -a bob balance
accountId: GAURP3FQ56BOF2PXF5DCLFS4QOJPTO5XM62G2AAMAQXB4PZWYNSS4RKA subEntryCount: 0 home domain: null

thresholds: 0 0 0
signers:
    GAURP3FQ56BOF2PXF5DCLFS4QOJPTO5XM62G2AAMAQXB4PZWYNSS4RKA 1
authRequired: false
authRevocable: false

Balances:
XLM b:60.0000000 l:- - sl: - - bl: -

This returns a bit of meta data and a list of balances that currently only includes XLM.

So, Bob, now has exactly 60 XLM. Alice has payed 50 and 10 XLM as well as some fees for both transactions: 2x 100 stroops. A stroop is 1/100000th of an XLM. This means you can fund 100K transactions with a mere 1 XLM, which at the time of writing is roughly 20 euro cents.

Issuing your own token

To issue a new token, we need an issuing account. Lets create that as well as a distribution account that we will use for distributing our FOO token. Alice gets to pick up the bill for funding these accounts as well.

$ ./cliste -a alice createAccount issuing 100
$ ./cliste -a alice createAccount distribution 100

Token management

Now lets define our token:

$ ./cliste defineAsset issuing FOO

$ ./cliste listAssets
Defined assets (1):
FOO     GBSR46WQNCDK7SPUW3XR3C663AE3QDG3MGG5S66GI6XQINQYGAZH7CF5

This does nothing else than add another alias to a file called assets.properties. In stellar assets are always identified by their code + the account that issued it.

We will issue FOO from our issuing account to our distribution account.

Trust lines

For this to work there needs to be a trustline. In stellar, you can only hold tokens that you trust. So for our distribution account to be able to hold FOO, it needs to trust FOO

$ ./cliste -a distribution trust FOO
17:58:07.565 [main] INFO org.stellar.sdk.KotlinExtensions - 273 f5ad81be61734ebfaef89233e5203a9f24cad1538312b17607b047da9de15a62 success:true fee:100 CHANGE_TRUST

If we now check the balance for the distribution account we'll see a new entry under balances:

$ ./cliste -a distribution balance
accountId: GADRJ4IDLVMOFE5DNQF7UZGWAUABKO7MYM2FAKGEGVPZA2W4RFQFF6ZQ subEntryCount: 1 home domain: null

thresholds: 0 0 0
signers:
    GADRJ4IDLVMOFE5DNQF7UZGWAUABKO7MYM2FAKGEGVPZA2W4RFQFF6ZQ 1
authRequired: false
authRevocable: false

Balances:
FOO (GBSR46WQNCDK7SPUW3XR3C663AE3QDG3MGG5S66GI6XQINQYGAZH7CF5) b:0.0000000 l:922337203685.4775807 - sl: - - bl: -
XLM b:99.9999900 l:- - sl: - - bl: -

So, our distribution account 'trusts' FOO to the extent of Long.MAX stroops of a FOO or about 922337203685.4775807 FOO. This is the maximum value you can put in stellar's 64 bit balance. If you want you can actually limit your trust to something smaller.

Magic happens ...

Now lets do a magic trick and make our distribution account owner a billionaire (in FOO):

$ ./cliste -a issuing pay distribution 10000000000 FOO
18:01:57.355 [main] INFO org.stellar.sdk.KotlinExtensions - 319 11d1f4533656ee7b7e009f6db8ad9c61159bd4012790ffc46eff7ed03000a85d success:true fee:100 PAYMENT

Simply paying some FOO from the issuing account actually causes the FOO coin to come into existence. If we check the balance, we'll see simply paying from issuing to distribution created the token:

$ ./cliste -a distribution balance
accountId: GADRJ4IDLVMOFE5DNQF7UZGWAUABKO7MYM2FAKGEGVPZA2W4RFQFF6ZQ subEntryCount: 1 home domain: null

thresholds: 0 0 0
signers:
    GADRJ4IDLVMOFE5DNQF7UZGWAUABKO7MYM2FAKGEGVPZA2W4RFQFF6ZQ 1
authRequired: false
authRevocable: false

Balances:
FOO (GBSR46WQNCDK7SPUW3XR3C663AE3QDG3MGG5S66GI6XQINQYGAZH7CF5) b:10000000000.0000000 l:922337203685.4775807 - sl: - - bl: -
XLM b:99.9999900 l:- - sl: - - bl: -

Multi signatures

Of course in practice you might want to lock things down a bit. For this we can modify the account options. A good practice is to protect important accounts with mutliple signatures.

Adding signees to the issuing account

Lets add alice and bob as a signees to the issuing account.

$ ./cliste -a issuing setOptions --signer-key alice --signer-weight 5
18:07:07.987 [main] INFO org.stellar.sdk.KotlinExtensions - 381 78d4572ef51900f088babf87a4e878771391950a24ca6f710a1deacde0967ad7 success:true fee:100 SET_OPTIONS

$ ./cliste -a issuing setOptions --signer-key bob --signer-weight 5
18:07:17.933 [main] INFO org.stellar.sdk.KotlinExtensions - 383 5c7b0ee0747c57fbe456781888b322ef7d93270af9db8daf010c263ad64b5c38 success:true fee:100 SET_OPTIONS

This adds alice and bob as signers. Their keys have a weight of 5. If you look at the balance above, you'll see it defaults to 0. So either alice or bob have enough weight to do everything.

Managing thresholds and weights

Lets lock down the issuing account:

$ ./cliste -a issuing setOptions --low-threshold 8 --medium-threshold 8 --high-threshold 8 --master-key-weight 0
18:16:02.379 [main] INFO org.stellar.sdk.KotlinExtensions - 488 f1a347698c2b8ee6d9c891d6da7ae1c86fc367785b4f0dc1e71c54d78a7726e9 success:true fee:100 SET_OPTIONS

$ ./cliste -a issuing balance
accountId: GBSR46WQNCDK7SPUW3XR3C663AE3QDG3MGG5S66GI6XQINQYGAZH7CF5 subEntryCount: 2 home domain: null

thresholds: 8 8 8
signers:
    GBJR3JH4ZC5LZKGM2PPSVMJFPPWREVEKD4TQJL2RMUHI75P35N4JVABC 5
    GAURP3FQ56BOF2PXF5DCLFS4QOJPTO5XM62G2AAMAQXB4PZWYNSS4RKA 5
    GBSR46WQNCDK7SPUW3XR3C663AE3QDG3MGG5S66GI6XQINQYGAZH7CF5 0
authRequired: false
authRevocable: false

Balances:
XLM b:99.9999500 l:- - sl: - - bl: -

So, this confirms our issuing account now has 3 keys. We set the master key to 0; so it can no longer be used to issue FOO:

$ ./cliste -a issuing pay distribution 10000000000 FOO
java.lang.IllegalStateException: failure after 0 transaction failed tx_bad_auth - null
    at org.stellar.sdk.KotlinExtensionsKt.doTransactionInternal(KotlinExtensions.kt:180)
    at org.stellar.sdk.KotlinExtensionsKt.doTransaction(KotlinExtensions.kt:142)
    at io.inbot.kotlinstellar.KotlinStellarWrapper.pay(KotlinStellarWrapper.kt:322)
    at io.inbot.kotlinstellar.KotlinStellarWrapper.pay$default(KotlinStellarWrapper.kt:314)
    at io.inbot.kotlinstellar.cli.CommandsKt$doPay$1.invoke(Commands.kt:194)
    at io.inbot.kotlinstellar.cli.CommandsKt$doPay$1.invoke(Commands.kt)
    at io.inbot.kotlinstellar.cli.CommandContext.run(CommandContext.kt:54)
    at io.inbot.kotlinstellar.cli.CliSteMainKt.main(CliSteMain.kt:88)
com.xenomachina.argparser.SystemExitException: Problem running 'pay'. failure after 0 transaction failed tx_bad_auth - null
    at io.inbot.kotlinstellar.cli.CommandContext.run(CommandContext.kt:62)
    at io.inbot.kotlinstellar.cli.CliSteMainKt.main(CliSteMain.kt:88)
cliste: Problem running 'pay'. failure after 0 transaction failed tx_bad_auth - null

Signing transactions

To make this work, both Alice and Bob need to sign the transaction. So lets create an unsigned transaction:

$ ./cliste -a issuing preparePaymentTX distribution 10000000000 FOO
Transaction envelope xdr:
tx hash: zf+uG/7ePiTLuoqLbqgMMyQDq+PlxEsJkVEKq/jEixs=
tx envelope xdr: AAAAAGUeetBohq/J9LbvHYve2Am4DNthjdl7xkevBDYYMDJ/AAAAZAAAAMYAAAAGAAAAAAAAAAAAAAABAAAAAAAAAAEAAAAABxTxA11Y4pOjbAv6ZNYFABU77MM0UCjENV+QatyJYFIAAAABRk9PAAAAAABlHnrQaIavyfS27x2L3tgJuAzbYY3Ze8ZHrwQ2GDAyfwFjRXhdigAAAAAAAAAAAAA=

This creates a transaction but instead of submitting it to stellar, it outputs the serialized binary representation in a format called XDR. This is actually what stellar stores internally.

Adding signatures

Stellar requires two signatures for this transaction because we configured this in our previous step. So both alice and bob must add their signatures before we can submit this transaction.

First alice signs the transaction:

$ ./cliste -a alice signTx AAAAAGUeetBohq/J9LbvHYve2Am4DNthjdl7xkevBDYYMDJ/AAAAZAAAAMYAAAAGAAAAAAAAAAAAAAABAAAAAAAAAAEAAAAABxTxA11Y4pOjbAv6ZNYFABU77MM0UCjENV+QatyJYFIAAAABRk9PAAAAAABlHnrQaIavyfS27x2L3tgJuAzbYY3Ze8ZHrwQ2GDAyfwFjRXhdigAAAAAAAAAAAAA=
tx hash: zf+uG/7ePiTLuoqLbqgMMyQDq+PlxEsJkVEKq/jEixs=
tx envelope xdr: AAAAAGUeetBohq/J9LbvHYve2Am4DNthjdl7xkevBDYYMDJ/AAAAZAAAAMYAAAAGAAAAAAAAAAAAAAABAAAAAAAAAAEAAAAABxTxA11Y4pOjbAv6ZNYFABU77MM0UCjENV+QatyJYFIAAAABRk9PAAAAAABlHnrQaIavyfS27x2L3tgJuAzbYY3Ze8ZHrwQ2GDAyfwFjRXhdigAAAAAAAAAAAAH763iaAAAAQKZ+gYmDIqv9hUdZdC9+C4bUuX4RWmT8BnCI9wnb35IZ7IZIg5U8NIMvtodGEr4uv3NNB5/tbABEaNtDygihcws=

Notice we get back a different XDR. It now includes the signature from Alice. One signature is not enough. So, Alice can now send her signed XDR to Bob via email/slack/etc. who can sign it as well:

./cliste -a bob signTx AAAAAGUeetBohq/J9LbvHYve2Am4DNthjdl7xkevBDYYMDJ/AAAAZAAAAMYAAAAGAAAAAAAAAAAAAAABAAAAAAAAAAEAAAAABxTxA11Y4pOjbAv6ZNYFABU77MM0UCjENV+QatyJYFIAAAABRk9PAAAAAABlHnrQaIavyfS27x2L3tgJuAzbYY3Ze8ZHrwQ2GDAyfwFjRXhdigAAAAAAAAAAAAH763iaAAAAQKZ+gYmDIqv9hUdZdC9+C4bUuX4RWmT8BnCI9wnb35IZ7IZIg5U8NIMvtodGEr4uv3NNB5/tbABEaNtDygihcws=
tx hash: zf+uG/7ePiTLuoqLbqgMMyQDq+PlxEsJkVEKq/jEixs=
tx envelope xdr: AAAAAGUeetBohq/J9LbvHYve2Am4DNthjdl7xkevBDYYMDJ/AAAAZAAAAMYAAAAGAAAAAAAAAAAAAAABAAAAAAAAAAEAAAAABxTxA11Y4pOjbAv6ZNYFABU77MM0UCjENV+QatyJYFIAAAABRk9PAAAAAABlHnrQaIavyfS27x2L3tgJuAzbYY3Ze8ZHrwQ2GDAyfwFjRXhdigAAAAAAAAAAAAL763iaAAAAQKZ+gYmDIqv9hUdZdC9+C4bUuX4RWmT8BnCI9wnb35IZ7IZIg5U8NIMvtodGEr4uv3NNB5/tbABEaNtDygihcws2w2UuAAAAQOboMFKz6sOnFPio17cuaOBLrHYN7k/DpFSGAaYVgYKg25YCMqZug2brTkh7LXaubChpFBYJHkF4vN/tUQNSCwM=

Bob gets back an even bigger XDR.

Examining the XDR

Before submitting it, it might be a good idea to check what is inside:

$ ./cliste txInfo AAAAAGUeetBohq/J9LbvHYve2Am4DNthjdl7xkevBDYYMDJ/AAAAZAAAAMYAAAAGAAAAAAAAAAAAAAABAAAAAAAAAAEAAAAABxTxA11Y4pOjbAv6ZNYFABU77MM0UCjENV+QatyJYFIAAAABRk9PAAAAAABlHnrQaIavyfS27x2L3tgJuAzbYY3Ze8ZHrwQ2GDAyfwFjRXhdigAAAAAAAAAAAAL763iaAAAAQKZ+gYmDIqv9hUdZdC9+C4bUuX4RWmT8BnCI9wnb35IZ7IZIg5U8NIMvtodGEr4uv3NNB5/tbABEaNtDygihcws2w2UuAAAAQOboMFKz6sOnFPio17cuaOBLrHYN7k/DpFSGAaYVgYKg25YCMqZug2brTkh7LXaubChpFBYJHkF4vN/tUQNSCwM=
850403524614 operations:
source account: GBSR46WQNCDK7SPUW3XR3C663AE3QDG3MGG5S66GI6XQINQYGAZH7CF5
10000000000.0000000 FOO to GADRJ4IDLVMOFE5DNQF7UZGWAUABKO7MYM2FAKGEGVPZA2W4RFQFF6ZQ
Signatures:
pn6BiYMiq/2FR1l0L34LhtS5fhFaZPwGcIj3CdvfkhnshkiDlTw0gy+2h0YSvi6/c00Hn+1sAERo20PKCKFzCw==
5ugwUrPqw6cU+KjXty5o4Eusdg3uT8OkVIYBphWBgqDblgIypm6DZutOSHstdq5sKGkUFgkeQXi83+1RA1ILAw==

Submitting the signed transaction

Now lets submit the transaction:

$ ./cliste submitTx AAAAAGUeetBohq/J9LbvHYve2Am4DNthjdl7xkevBDYYMDJ/AAAAZAAAAMYAAAAGAAAAAAAAAAAAAAABAAAAAAAAAAEAAAAABxTxA11Y4pOjbAv6ZNYFABU77MM0UCjENV+QatyJYFIAAAABRk9PAAAAAABlHnrQaIavyfS27x2L3tgJuAzbYY3Ze8ZHrwQ2GDAyfwFjRXhdigAAAAAAAAAAAAL763iaAAAAQKZ+gYmDIqv9hUdZdC9+C4bUuX4RWmT8BnCI9wnb35IZ7IZIg5U8NIMvtodGEr4uv3NNB5/tbABEaNtDygihcws2w2UuAAAAQOboMFKz6sOnFPio17cuaOBLrHYN7k/DpFSGAaYVgYKg25YCMqZug2brTkh7LXaubChpFBYJHkF4vN/tUQNSCwM=
OK

And check our distribution account again:

$ ./cliste -a distribution balance
accountId: GADRJ4IDLVMOFE5DNQF7UZGWAUABKO7MYM2FAKGEGVPZA2W4RFQFF6ZQ subEntryCount: 1 home domain: null

thresholds: 0 0 0
signers:
    GADRJ4IDLVMOFE5DNQF7UZGWAUABKO7MYM2FAKGEGVPZA2W4RFQFF6ZQ 1
authRequired: false
authRevocable: false

Balances:
FOO (GBSR46WQNCDK7SPUW3XR3C663AE3QDG3MGG5S66GI6XQINQYGAZH7CF5) b:20000000000.0000000 l:922337203685.4775807 - sl: - - bl: -
XLM b:99.9999900 l:- - sl: - - bl:

Alice and Bob can now start using FOO

$ ./cliste -a alice trust FOO
$ ./cliste -a bob trust FOO
$ ./cliste -a distribution pay alice 1000000 FOO
$ ./cliste -a alice pay bob 100000 FOO

Post Agile: embracing asynchronous processes

Jilles van Gurp — Mon, 13 Aug 2018 14:56:58 +0000

Like many people my age in this industry (software development, if that was not clear), I remember how things worked before the internet and before agile. Programming in these days meant using expensive tools that came with stacks of books that sat on your desk. Google and Stackoverflow did not exist. Having internet in the workplace was not very widespread. I learned to program using the Basic manual that came with my commodore 64. Agile was not a thing.

Things have changed since then and mostly for the better. Kent Beck's Extreme programming was a book and idea I was very enthusiastic about when it came out. Around that time I was doing a Ph. D. in the field of Software Engineering. I even referenced it in my 2003 Ph. D. thesis. The agile movement (Kent Beck was of course one of the Agile Manifesto signees), revolutionized development in many ways.

Bad Agile

Everybody is claiming/pretending to be agile these days. So, the word has become somewhat meaningless. Every bank, insurer, and shitty little software shop is doing Agile. Capital A Agile of course because they are doing things "by the book". Anyone who knows anything about Agile would know this is exactly the wrong thing to do. I've been in the same room with some of the manifesto signees at various conferences, lectures, workshops, etc. and I've heard them spell this out. Agile is a set of tools and a way of working that you use and adapt to your needs. Use the books as a starting point, not the end state. If you know nothing, you might as well start by doing this.

I've grown somewhat tired of the "you're doing it wrong" camp that snuffs out any criticism of things like Scrum, which I would argue has become somewhat toxic in our industry. People either hate scrum or love it and mostly for the wrong reasons (either way). Broadly speaking, people dislike the wide spread practice of its mindless application, the constant bickering in its pointless meetings, and the emotional drain and wasted energy over arguments on how to do it "right", whatever that means. Personally, this is a thing that I've witnessed on pretty much any project I've been involved with in the last 15 years. Anecdotical evidence suggests I'm not alone.

For better or worse, Scrum has nominally replaced waterfall as something that is easy enough to grasp for organizations to restyle themselves as Agile without actually changing that much. That doesn't mean that Scrum or Agile are bad but Scrum in particular seems to have become the tool of choice for bad agile implementations. Scrum even has its own term for this: scrumbutt.

There's a lot of bad agile going on in our industry. Most software developing companies are just as boring, stupid, and ineffective as 20 years ago. Government IT projects still go spectacularly wrong. Banks still sink tons of money in misguided projects. Companies like Lidl still allow themselves to get ripped off by companies like SAP (to the extent of a sweet 0.5 Billion). For the record, I blame both companies for this.

Everybody prays to the church of Agile now and there are hordes of self styled Agilists/Agile coaches/etc to help you figure out how to do it right. A lot of companies employ some of those people pre-emptively to ensure they do things "by the book", which of course defeats the purpose.

Whatever your point of view on this, a trend in our industry is that things are changing again and people are looking beyond agile at different and relatively new ways of organizing software development. If only to distinguish themselves from all the people doing bad Agile.

Agile is close to 20 years old now. You are not going to improve things by not changing things and shaking things up a little. Some people have started referring to this as post agile. Whatever it is, scrum is definitely not a part of it.

History and evolution of Agile

When the Agile manifesto was signed, most people were in fact not doing Agile or anything remotely close to it. Agile was a new thing then and somewhat controversial. People were doing all sorts of things and were generally confusing and conflating processes and modeling techniques and requirements engineering methodology and tools. Universities mostly taught waterfall then.

In the late nineties and early 2000s people attempted to standardize modeling languages. The result, UML, was widely popular for a while and companies like Rational (later acquired by IBM) tried to make it the center-point of the development process. The result, Rational Unified Process was was considered modern and hip then and given the timing a bit of a counter move against Agile. Rational and RUP ended up in the hands of IBM; arguably one of the least agile companies in existence at the time (and today). Blue suits/white shirts were very much still a thing at IBM then. Standards, certification, training, and related consulting business were booming.

UML and RUP perpetuated the dogmas of waterfall, which was to first do detailed designs (using UML, of course) after doing requirements specifications (also using UML, how convenient) and before doing implementation work and before testing would commence. Briefly that too was supposed to be done using UML with something called model driven development. Thankfully, MDA and the associated range of (now) abandonware to do that no longer comes up a lot in serious discussions about software development these days.

But RUP was also a stepping stone towards Agile. Rational unified tried to be iterative as well. This really meant you got to do waterfall multiple times in a project; once a quarter or so. Rational's premise was that this required tools, lots of tools. Very complex tools that required lots of consulting. This is why IBM bought them and they made a lot of money getting organizations to implement RUP, training software architects and selling expensive licenses for software. Back in those days, any self respecting software architect had some boxes and books with the Rational logo prominently in sight. They'd be wielding impressive looking diagrams and there would typically be a lot of architecture and design documentation with more diagrams.

Agile was hugely disRUPtive, mostly in the sense that it popped that bubble. Sorry about the bad pun. It just melted away in the space of about 5 years. Between 2000 and 2005, UML slowly disappeared from our lives. I haven't used UML tools in ages and can't say I miss them.

Iterative development is of course as old as waterfall. The original paper by Royce on waterfall Managing the development of large software systems from 1970 is actually still a pretty good read and was soon complemented by papers on spiral and iterative development.

Feedback loops are a good thing; every engineer knows this. In fact, Royce brought up iterations in that paper! Literal quote from the paper: "Attempt to do the job twice - the first result provides an early simulation of the final product". Royce was trying to be Agile in 1970. Of course his work got dumbed down to: lets first do requirements; set those in stone and turn them into design and implementation and maybe do a bit of testing/bugfixing before we throw it over the wall and walk away. Fun fact, the word waterfall does not appear in Royce's paper! The original paper on waterfall does not mention waterfalls, at all. Don't blame Royce for waterfall. Waterfall was never a thing, from day 1.

What the agile manifesto and movement accomplished was that the waterfall bubble was burst and iterative development became the norm. Having a lot of design documentation in UML slows you down when iterating and makes it hard to do that. If iterating is your goal, you can't lose time doing that. People figured out that the added value of this typically incomplete and out of date documentation was questionable.

The result was that UML became a thing for whiteboards and from there an entirely optional thing that these days is not a topic that comes up in software planning at all. Same with requirements documentation. This was a bit of a black art to begin with. With agile people figured out that it's much easier to specify small deltas of what you want changed against what you have right now instead of specifying the whole thing up front. Which you inevitably would get wrong anyway.

Getting rid of project bureaucracy like that allowed for shorter cycles and faster iterations and focused development around working prototypes. Extreme programming was about taking that to the (at the time) extreme of doing sprints that were as short as a few weeks. This was unheard of in an industry where projects could spend months or years without even producing working code.

Agile brought a revolution in tools

Tools such as issue trackers empowered this. Bugzilla was the first popular one that got a lot of traction. This happened roughly around the time that Agile became a thing. Issue trackers turned requirements into a new way of working. Instead of writing specifications, you instead specify change requests in the form of issues that are tracked. Initially this was used for bugs but very soon this expanded to track essentially all sorts of changes. Similarly, wikis took the place of design and product documentation.

Agile spawned the adoption and development of a lot of new tools; many of which had their origins in open source communities. These days any decent project has an issue tracker, some kind of decentralized version control system (usually Git), CI tooling, wikis, communication tools like irc or slack, etc. These tools are essential and they continue to change how people are working. Some of these tools are run in the cloud by companies like Atlassian, Gitlab, and Github. The recent acquisition of the latter by Microsoft shows how important these tools have become.

The open source world was always distributed and could never rely on meetings. Consequently they adopted tools and processes that supported their way of working. Early projects used things like mailing lists and news groups; and version control systems like CVS, RCS, and later things like Subversion and Git. Likewise, Irc predates Slack by several decades and continues to be the preferred way of communicating in some projects. The practice of pull requests on Github/Gitlab that is now common in enterprise projects emerged out of the practice of exchanging patches via mailing lists and news groups. Eventually Git was created to make this process easier and Github created a UI for it.

Many of these tools were not common (or even around) when the Agile manifesto was signed. However, Agile people embraced these tools and also spawned the Devops movement which integrated operations experience and responsibilities into teams. Devops brought even more tools to the table. Things like deployment automation, docker, kubernetes, PAAS, SAAS, chatops, etc.

All these tools and their associated practices are slowly resulting in a post agile world.

Meetings are synchronization bottlenecks

Agile replaced waterfall with structured processes to organize development in an iterative way. The above mentioned tools mostly came later. An unintended side effect of agile was lots of new meetings. Talk to any engineer about things like Scrum and you'll essentially be ticking off what meetings they have to do: sprint plannings, estimations, retrospectives, reviews, and of course standups. The schoolbook implementations of Scrum are essentially endless cycles of such meetings. Most engineers I know hate meetings and/or see them as a necessary evil at best.

With post agile people are discovering that the added value of meetings is questionable and that tools exist that facilitate getting rid of them. One of my favorite .com era surviving companies, despair.com, still sells a great poster with the following slogan: Meetings. None of us is as dumb as all of us.. I've been to more than a few scrum related meetings that reminded me of that poster.

Meetings are inherently synchronous in both time and (usually) space. They require heads in a room at a specific moment. This is highly disruptive because people have to stop what they are doing, go to the meeting, and discuss and decide things together, and come to a decision. Video conferencing tools kind of suck for this and remote attendees are typically at a huge disadvantage in such meetings.

Going asynchronous

With post agile, people are keeping the tools they adopted from OSS teams and some of the practices of Agile. However they are abandoning meetings and generally eliminating synchronization bottlenecks in their processes. This is similarly liberating as saying goodbye to convoluted out of date UML diagrams, crappy requirements specifications, and the glacial pace of waterfall style development. I have many friends and colleagues that are working in distributed teams that span the globe.

There are a couple of practices that are key to post agile. The key enabler is continuous deployment or continuous releases. Simply put: if stuff is ready, you make it available right away in order to keep feedback loops as short as possible. The OSS community figured this out first. Mozilla nightly builds have been a thing for pretty much as long as their open source products have been around. Frequent releases are essential for gathering feedback via an issue tracker. You don't wait until after the next retrospective in two weeks or whatever arbitrary milestone you have to get that feedback. And you use tools and automated processes to make sure this happens in a controlled and predictable way.

Continuous deployment requires a high degree of automation. It requires continuous integration: aka. automatically assessing whether you are fit to ship right now. Every change triggers a CI build. CD eliminates release management as a role and CI relieves product managers from having to manually test and approve releases.

Continuous integration in turn requires having tests that can run automatically that cover enough of the product that people feel confident that things work as they intend. The goal of automated tests is to prevent having manual testing on the critical path of releasing any software.

Another thing that continuous deployment requires is the ability to branch and merge changes. In order to keep the production branch stable and releasable, it is essential to keep work in progress on branches until it is good enough.

Git is the key enabler for this and another tool that emerged out of the OSS world. Before Git, version control systems were huge organizational bottlenecks. Branching was a royal pain in the ass and sufficiently complicated that subsequent merging required lots of planning. Organizations frequently were bottle-necked on merges. This was mitigated with commit freezes and similar practices. Linus Torvalds has been running the largest OSS project in the world (Linux) for nearly 25 years and all this planning and coordination around branches and merges annoyed him enough that he invented Git. Git codifies and improves the asynchronous change management that the Linux development community has been practicing.

Of course Linux still continues to rely on mailing lists for using git. Linux developers exchange git patches (i.e. textual exports of commits, not just diffs) via email. However, a company called Github emerged that made Git more user friendly for the masses and introduced us to a key tool for post agile: the Pull Request. It's essentially the same thing but the flow of reviewing and merging is supported with a nice web UI.

Pull requests are an asynchronous way to manage changes in software. Back when agile started out version control was done doing CVS (subversion was still in beta) and branching was considered a very dangerous ritual that was best avoided. Entire companies were getting by without either version control systems or branches.

Pull requests, CI, and CD are powerful tools that can remove a lot of synchronization bottlenecks in software development. You initiate work on a topic via an issue tracker, after discussion in that tool, on mailing lists, or slack/skype/irc/whatever, you then create a branch and start work. When done you create a pull request (PR). Relevant stakeholders provide feedback (after being assigned or @tagged). Meanwhile CI confirms things are ready to merge. When the PR is approved, it is merged. This in turn triggers an automated deployment. Start to finish the life of a software change can be completely asynchronous. People synchronize on tickets in issue trackers and pull requests and escalate via asynchronous communication tools. When stuff goes wrong the relevant PR is identified, the git history and issue trackers are used to figure out what happened and if needed a new PR is created to either revert or fix the issue. Tools facilitate decision making, planning, auditing, automation and affect all life cycles of the traditional waterfall model.

Asynchronous enables distributed teams

Going asynchronous can cut down on meetings and eliminates Sprints as a necessity or a meaningful unit of work and allows you to iterate in hours instead of weeks. Asynchronous also enables you to distribute the work geographically. Meetings are an obstacle for this because they require people to be in the same place and timezone. Neither is practical if you have people on all continents. By going asynchronous, people can coordinate work and synchronize via issues, pull request, and slack while the decision making around releases, deployments, etc. is automated.

This enables you to get rid of essentially all Agile related meetings. All of them. This is why I believe that going asynchronous and distributed effectively moves us to a post Agile world. Standups are not practical in a distributed team, so those go. Having lengthy sprint plannings and estimation sessions over video calls are not practical either. Sprints themselves are no longer necessary. And so on.

Should you go post Agile?

Should everybody now jump on the bandwagon and start doing post agile? I'd argue that, no, you should only do things that fit your context. Just like you were supposed to do with Agile. What is helpful though is reflecting on where you are with your organization in terms of what you are doing, what you are trying to accomplish, what tools and processes you have, what your bottlenecks are, and how you are evolving your processes and tools over time. You probably already use a lot of the tools and practices mentioned above.

I see distributed teams and companies as the key drivers of the post agile world. Asynchronous tools and practices are a key enabler for that. There are numerous economical advantages to going distributed that may cause people to be interested in becoming more distributed and less bottlenecked on meetings. For example, hiring is easier when people don't have to move to your central office. You have access to a global pool of talent. Not having to have expensive offices in expensive places like San Francisco or London is also a huge plus. Not wasting double digit R&D budget percentages on meetings and associated traveling is also a big gain. Sticking everybody in meeting rooms for a whole day every two weeks for scrum related planning and ceremony is 10% of your development budget. In places like San Francisco we are talking significant chunks of investor cash being spent on that as engineers are expensive there.

If you want to tap into these benefits, you need to think about how you can integrate asynchronous in your current processes. It's perfectly alright to continue to do Scrum and practice post agile practices at the same time. Many companies do this. It's not about jumping ship but about reflecting on what you are doing and why and how that is working. Personally, I've long preferred Kanban style processes because they are easier to align with doing things asynchronously. Whatever you do, please don't be dogmatic about it.

Streaming results from a JdbcTemplate in Kotlin

Jilles van Gurp — Fri, 15 Jun 2018 13:01:30 +0000

I've been transitioning from using Hibernate to using JdbcTemplate in a Kotlin based Spring Boot 2.x project recently. The why and how of that I wrote down in another article. One of the repository methods that I needed to move over was something like this:

    @Query("...")
    fun streamUserIds(): Stream<String>

One of the nice things with Spring Boot is that it does the right thing with generating a useful implementation that does the right thing. You can do Streams, Optionals, Lists, etc. JdbcTemplate does not really have anything similar.

IMHO a somewhat strange omission in the JdbcTemplate API but easy to fix. After a bit of googling, I found this somewhat helpful page with a solution that nearly worked but not quite. But it put me on the right track.

So, I decided to share my implementation since I think it is a bit simpler and better:

    fun <T> queryStream(sql: String, converter: (SqlRowSet) -> T?, args: Array<Any>): Stream<T> {
        val rowSet = jdbcTemplate.queryForRowSet(sql, *args)

        class RowSetIter : Iterator<T> {
            var current: T? = null

            override fun hasNext(): Boolean {
                if (current != null) {
                    return true
                } else {
                    if (rowSet.next()) {
                        current = converter.invoke(rowSet)
                        return true
                    }
                }
                return false
            }

            override fun next(): T {
                if (hasNext()) {
                    val retVal = current
                    current = null
                    return retVal!!
                } else {
                    throw NoSuchElementException()
                }
            }
        }

        val spliterator = Spliterators.spliteratorUnknownSize(RowSetIter(), Spliterator.IMMUTABLE)
        return StreamSupport.stream(spliterator, false)
    }

All this does is wrap SqlRowSet with a simple iterator. SqlRowSet is a simple wrapper around the JDBC RowSet with sane exception handling that makes the above a bit less tedious.

My implementation fixes a few issues that the code in the linked article has:

the exit condition was wrong and omitted the last row.
I like to do the heavy lifting in the hasNext() method instead of the next() method and make next rely on hasNext(). This also removes the need to call next on the first row. I've implemented some iterators before and this seems a good pattern for iterators.
It was attempting to stream rowset. This doesn't really make sense given that this is some lowlevel object representing a db cursor and all you are doing is returning the same object and calling next() on it. So instead, I'm using a lambda to convert each row to a T. So, whether you are mapping some entity or just extracting strings, it will work. And you can always make T a Unit if you really just want to iterate over the rows and not return anything.

Another gotcha is that you obviously can't close the connection before you stream because it needs to use the database cursor to get results. So, the proper way to solve this this is with a TransactionTemplate so you keep the connection open until after you are done streaming results from the DB:

transactionTemplate.execute {
    dao.queryStream("select user_id from table") { rs, _ ->
        // Map rows to a String
        rs.getString("user_id")
    }.forEach {
        // do something with each user_id
        println("User $it")
    }    
}

This will work whether you have 50 users or 50 million user ids. The code above should be easy to port back to Java if you need it to. I also shared this code as a comment to the Github gist that was linked from the article that inspired this.