DEV Community

Cover image for Bootstrapping the System, Part 3: Making It Run (Chapter 13)
Kamen
Kamen

Posted on Originally published at kamenivanov.substack.com

Bootstrapping the System, Part 3: Making It Run (Chapter 13)

Parts 1 and 2 left the application with every bean it needs and a correct home for every setting it reads. What neither of them touched is the process.

A process has properties that no properties file decides. How many requests it is allowed to hold at once, which is a concurrency model rather than a number. How a container's environment becomes something a JVM will accept, given that nothing hands a properties file to a container. What a log line looks like and where it lands, which no amount of logging.level configuration answers. And what the thing can tell you about itself at three in the morning, when the only question that matters is which version is running.

Those four are this part, and it ends with an image and a compose file, so that from here on the whole project starts with one command. The first of them is a single line that reaches further than anything else in the file.


Virtual threads change what the pool means

It is this one:

spring.threads.virtual.enabled=true
Enter fullscreen mode Exit fullscreen mode

Boot responds by running the servlet container's request handling on virtual threads, so every inbound request gets its own, and the asynchronous machinery inside Spring follows. The change is small to write and large in what it removes, because the thread pool that used to sit in front of everything was doing a job nobody had asked it to do.

A classic Tomcat serves requests from a fixed pool, two hundred threads by default. A request that blocks on a database call holds one of those two hundred for the duration, which means that two hundred was never a thread count so much as a concurrency limit, and the limit was enforced in the one place in the stack that knew nothing about what the requests were doing. Exceed it and requests queue in the connector, which is admission control by accident: nobody chose two hundred, nobody tuned it against the database, and the number is in a different file from everything it affects.

With virtual threads there is effectively no such ceiling, and the consequence is that the first real limit a request meets is the one in Part 2's datasource block. Ten thousand concurrent requests become ten thousand virtual threads, cheaply, and all ten thousand of them arrive at Hikari asking for one of twenty-five connections. Twenty-five proceed and the rest wait. This is the same backpressure as before, relocated to somewhere that actually understands the constraint, which is the right place for it, and it makes connection-timeout=5000 a far more consequential line than it looks. Five seconds is now the application's admission control: a request that cannot get a connection within it fails, visibly, with something the handler from Chapter 11 can turn into a response, rather than sitting in a queue nobody is measuring.

Enabling this two years ago would have been a worse idea than it is now, and the reason is worth knowing because it is the sort of thing that silently undoes the whole feature. A virtual thread that blocked inside a synchronized block used to pin itself to its carrier platform thread, so an application with a synchronized hot path quietly reverted to a small thread pool under load, which is the opposite of what the configuration appeared to request. JDK 24 removed that behaviour, and on the JDK 25 this project targets, a virtual thread blocking on a monitor unmounts like any other blocking operation. What remains is narrow: native frames can still pin, through a native method or the Foreign Function and Memory API calling back into blocking Java. Nothing in this codebase does either.

What does not change is anything about the transaction model, which is worth stating because it is the first thing people assume breaks. A virtual thread is a thread. @Transactional binds its entity manager to the current thread exactly as before, the ThreadLocal machinery underneath it works unchanged, and the service layer cannot tell the difference. The one thing to be careful with is a ThreadLocal used as a cache rather than as context: one that held two hundred copies of something under the old model will hold as many copies as there are concurrent requests under this one. There is no such cache here, which is the only reason this is a footnote rather than a section of its own.


From compose to the JVM

In a container, nothing passes a properties file. The entrypoint script does the translation:

#!/bin/sh
set -e

for var in DB_URL DB_USER DB_PASSWORD ACTIVE_PROFILE; do
    eval "value=\$$var"
    if [ -z "$value" ]; then
        echo "$(date +"%T"): FATAL: $var is not set"
        exit 1
    fi
done

DEBUG_OPTS=""
if [ "${DEBUG:-false}" = "true" ]; then
    DEBUG_OPTS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:8787"
fi

# Strip carriage returns: .env files edited on Windows carry them,
# and a trailing \r makes the JVM reject the last option.
JAVA_OPTS=$(printf '%s' "${JAVA_OPTS:-}" | tr -d '\r')

echo "$(date +"%T"): ===== Environment ====="
echo "$(date +"%T"): ACTIVE_PROFILE=$ACTIVE_PROFILE"
echo "$(date +"%T"): DB_URL=$DB_URL"
echo "$(date +"%T"): DB_USER=$DB_USER"
echo "$(date +"%T"): ======================="

cd /runtime

# $DEBUG_OPTS and $JAVA_OPTS are unquoted: word splitting is how
# whitespace-separated JVM options arrive at java as separate arguments.
# Options whose values contain spaces belong in JAVA_TOOL_OPTIONS instead.
# exec allows Java to receive container shutdown signals directly.
exec java $DEBUG_OPTS $JAVA_OPTS \
    "-Dspring.profiles.active=$ACTIVE_PROFILE" \
    "-Dspring.datasource.url=$DB_URL" \
    "-Dspring.datasource.username=$DB_USER" \
    "-Dspring.datasource.password=$DB_PASSWORD" \
    -Ddebug=false \
    -jar "${JAR_LOCATION:?JAR_LOCATION must be set}"
Enter fullscreen mode Exit fullscreen mode

Spring Boot would have accepted SPRING_DATASOURCE_URL directly, through relaxed binding, and this script could have been one line. Translating by hand buys two things.

The first is the echo block. When a container will not start and the question is whether the variable arrived at all, this is the first thing in the log. With relaxed binding there is no such point: either Spring complains about something downstream, or you go digging through docker inspect.

The second is that DB_URL is a name this project owns. Compose files, deployment manifests and whatever CI does all refer to it, and none of them know that there is a Spring application on the other side. SPRING_DATASOURCE_URL leaks the framework into the orchestration, and the day the framework changes, every one of those files changes with it.

The cost is that the script needs maintaining, and a new setting has to be added to it rather than working automatically. That is fine while the list is short, and the list staying short is itself useful pressure.

The explicit check at the top is there because an unset variable produces -Dspring.datasource.password= and an attempt to connect with an empty password, which fails much further along and much less clearly. exec makes the JVM process 1 in the container, so signals from Docker reach it directly rather than being swallowed by a shell that would have stayed between them.

-Ddebug=false is not redundant either. Spring reads the debug property, and if it is true, every autoconfiguration logger goes to DEBUG regardless of what logback.xml says. Setting it explicitly means nothing in the environment can turn the log into a wall of Spring internals by accident.

Three smaller things in that script are each a bug I would otherwise have hit twice. The debug agent is attached only when DEBUG is true, which keeps the decision in the environment rather than in the image, and makes a remote debugger something you turn on for a container instead of something you rebuild for. JAVA_OPTS is piped through tr -d '\r' because a .env file edited on Windows carries carriage returns and a trailing one makes the JVM reject the last option with a message about a flag that looks perfectly spelled. And $DEBUG_OPTS $JAVA_OPTS are the only unquoted expansions in the file: word splitting is how a whitespace-separated list of JVM options arrives at java as separate arguments, which is also why an option whose value contains spaces belongs in JAVA_TOOL_OPTIONS rather than here. The comments saying so are in the script because this is exactly the kind of line somebody tidies up by adding quotes.

${JAR_LOCATION:?JAR_LOCATION must be set} is the same instinct as the loop above it, applied to the one variable the loop cannot check, because it comes from the image rather than from the environment.


Appenders in XML, levels in properties

Logging is configured in two files, and the division between them is not arbitrary.

<configuration>
    <appender name="console" class="ch.qos.logback.core.ConsoleAppender">
        <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
            <pattern>%d{HH:mm:ss.SSS} [%thread] %highlight(%-5level) %cyan(%logger{36}) - %msg%n</pattern>
        </encoder>
    </appender>

    <appender name="file" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>${LOG_DIR:-.}/advanced.log</file>
        <append>true</append>
        <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
            <pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
        </encoder>
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <fileNamePattern>${LOG_DIR:-.}/%d{yyyy-MM, aux}/advanced-%d{yyyy-MM-dd}.log.gz</fileNamePattern>
        </rollingPolicy>
    </appender>

    <root level="INFO">
        <appender-ref ref="console"/>
        <appender-ref ref="file"/>
    </root>
</configuration>
Enter fullscreen mode Exit fullscreen mode

logback.xml holds structure: which appenders exist, what a line looks like, where the file goes and when it rolls. None of that differs between a laptop and production, and all of it is awkward to express as flat properties, which is exactly the case where a nested format earns its keep. The levels are not here. Every logging.level.* is a property, because a level is a statement about an environment, and an environment is what profiles are for.

That boundary is easy to state and easy to violate, and the reference project I lifted this file from violated it: a committed <logger name="..." level="DEBUG" /> for the application's own package. It works on a laptop, it is invisible in review, and it ships DEBUG to production while application-dev.properties sits there looking authoritative about log levels it no longer controls. Two files disagreeing about the same setting is worse than either of them being wrong, because the losing one still reads as true.

The console pattern is coloured and the file pattern is not. Highlight escape codes are useful in a terminal and are literal bytes in a file, where they turn every grep into a question about whether the match failed or the colour code got in the way. Same pattern otherwise, so a line copied out of one reads identically in the other.

The log file's directory comes from LOG_DIR with the working directory as its fallback, and that indirection exists because of a bug in the file I started from. That project wrote to ./app.log from logback and told the actuator its log file was at /runtime/logs/app.log, so the logfile endpoint returned a 404 for a file that existed somewhere else entirely, and had done for long enough that nobody expected the endpoint to work. One variable, read by both, is the whole fix: logback writes there and management.endpoint.logfile.external-file points at the same place.


An application that knows which commit it is

The first question anyone asks a service that is behaving strangely is which version of itself it is running, and the answer is usually a guess based on when it was last deployed. Making the application answer it costs one resource file and one bean.

Maven writes the file at build time by filtering a template:

buildJdk=${java.version}  
buildJvm=${java.vm.vendor} ${java.vm.name} ${java.vm.version}  
buildOs=${os.name} ${os.version} (${os.arch})  
buildTime=${buildTime}  
builtBy=${user.name}  
implementationBranch=${scmBranch}  
implementationRevision=${scmRevision}  
implementationTitle=${project.artifactId}  
implementationVendor=${project.organization.name}  
implementationVendorId=${project.groupId}  
implementationVersion=${project.version}  
specificationTitle=${project.name}
Enter fullscreen mode Exit fullscreen mode

Those placeholders are three different kinds of thing, which matters when one of them silently stays a placeholder. project.* comes from the POM. java.version and user.name are JVM system properties that the resources plugin resolves while it runs, so they describe the machine that built the jar rather than the machine running it, which is the point. scmBranch, scmRevision and buildTime all come from buildnumber-maven-plugin, under names its configuration chooses rather than names Maven provides, which is why timestampPropertyName is set to buildTime and buildNumberPropertyName to scmRevision. None of the three exists unless the plugin's create goal has already run, so it is bound to generate-resources, one phase before the resources plugin reads the template. Get that wrong and the file ships with ${scmRevision} in it as a literal string, which is the worst outcome available: an endpoint that answers confidently with nothing.

There is one hole in that - revisionOnScmFailure is set to 0, so a build from an exported copy with no git metadata reports a revision of zero instead of failing. That is the same failure the paragraph above warns about, chosen rather than stumbled into: a build that is not from a checkout has no answer to give, and a conspicuous zero is better than a hash that happens to parse.

Exposing it is a MapInfoContributor:

public class BuildInfoContributor extends MapInfoContributor {

    private static final String BUILD_PROPERTIES = "/build.properties";

    public BuildInfoContributor() throws IOException {
        super(readBuildProperties());
    }

    private static Map<String, Object> readBuildProperties() throws IOException {
        try (final InputStream is = BuildInfoContributor.class.getResourceAsStream(BUILD_PROPERTIES)) {
            if (is == null) {
                throw new IllegalStateException(BUILD_PROPERTIES + " is missing from the jar");
            }

            final var productInfoProperties = new Properties();
            productInfoProperties.load(is);

            final var buildProperties = new TreeMap<String, Object>();
            for (final Map.Entry<Object, Object> buildProperty : productInfoProperties.entrySet()) {
                buildProperties.put(buildProperty.getKey().toString(), buildProperty.getValue());
            }
            return buildProperties;
        }
    }
}
Enter fullscreen mode Exit fullscreen mode

The null check is there because the version I started from did not have one, and the failure it produces is instructive. getResourceAsStream returns null for a resource that is not on the classpath rather than throwing, Properties.load(null) throws a NullPointerException, and because this runs in a constructor, Spring reports it as a bean creation failure with an NPE several frames down in a class that has nothing to do with the cause. The actual cause is a resources-plugin configuration that stopped filtering, which is a one-line fix behind twenty lines of stack trace. Replacing it with a thrown IllegalStateException naming the file turns a debugging session into reading the first line of the log.

A TreeMap rather than a HashMap is a smaller decision in the same spirit. The endpoint renders the map in iteration order, and sorted output means two deployments can be compared by eye rather than by reading both twice.


Enough actuator to be useful

The actuator goes on a separate port, bound to loopback:

management.server.port=8081
management.server.address=127.0.0.1
management.endpoints.web.base-path=/
management.endpoints.web.exposure.include=health,info,logfile

management.endpoint.health.show-components=always
management.endpoint.health.show-details=always
management.endpoint.health.group.liveness.include=ping
management.endpoint.health.group.readiness.include=*

management.endpoint.logfile.external-file=${LOG_DIR:.}/advanced.log
Enter fullscreen mode Exit fullscreen mode

The separate port is the part worth copying. On 8080 the health endpoint is reachable by anyone who can reach the application, and show-details=always means it reports which components are up, including the database. On 8081 bound to 127.0.0.1 it is reachable from inside the container and nowhere else, which is exactly who needs it: the Docker healthcheck in the next section, and a developer with a shell.

Liveness and readiness are separated because they answer different questions. Liveness is whether the process should be restarted, and ping is enough: the JVM is up and answering. Readiness is whether it should receive traffic, which includes the database, because an application that cannot reach its database should not be in the load balancer. Conflating them produces a restart loop whenever the database is briefly unavailable.

The logfile endpoint reads the same LOG_DIR the appender writes to, which is the whole reason that variable exists rather than two hard-coded paths that agree until one of them is edited.

Three endpoints is a short list - Prometheus, loggers, heapdump, mappings and the rest are all useful, and every one of them is easier to argue about once there is something running to observe.


The first image, and three things wrong with it

Everything above converges here:

FROM bellsoft/liberica-openjdk-alpine:${liberica.version}

ENV LOG_DIR=/runtime/logs
ENV JAR_LOCATION=/data/advanced-service.jar

WORKDIR /runtime

RUN mkdir -p "$LOG_DIR" \
    && chmod a+rwx "$LOG_DIR"

COPY scripts /scripts
COPY maven/application-${project.version}-exec.jar ${JAR_LOCATION}

EXPOSE 8080 8787

HEALTHCHECK --interval=5s --timeout=3s --retries=20 \
    CMD wget -q -O - -S http://127.0.0.1:8081/health || exit 1

ENTRYPOINT ["/bin/sh", "/scripts/run.sh"]
Enter fullscreen mode Exit fullscreen mode

The base image is pinned to 25.0.4-9, a Java version and a build number rather than the 25 that floats, and the value is a property in the parent POM so that the JDK is declared next to every other version this build pins. Filtering puts it in the FROM line, the same mechanism that puts project.version in the COPY. JAR_LOCATION is the -exec jar from Part 1, which exists as a separate artifact precisely so that the image can name it without ambiguity, and it is copied from maven/ rather than from target/ because that is where the Maven plugin that builds this image places the assembly inside the build context. LOG_DIR is the variable the logback appender and the actuator's logfile endpoint both read, which is why they cannot disagree about where the log is. And the healthcheck reaches 127.0.0.1:8081, which works only because the actuator is bound to loopback and the healthcheck runs inside the container: the one caller that needs it is the one caller that can reach it.

Getting the healthcheck destination right matters more than it looks. The version I started from used 0.0.0.0, which is an address to bind to rather than an address to connect to. On Linux it usually resolves to the local host anyway, which is the worst kind of bug: correct by accident, and silently broken the day something changes underneath it.

This image works. It also has three problems I can name, and naming them now is the right choice than presenting it as finished.

It runs as root, and the chmod a+rwx on the log directory is the tell. A world-writable directory is what you write when the question of which user owns the process has not been asked.

It copies the fat jar as a single layer, so every change to a single service class produces a new layer containing every dependency the application has. Spring Boot can take a jar apart into layers that change at different rates, and this image does not.

And the choice between a JDK, a JRE and a purpose-built runtime has not been made. This image carries a full JDK, which includes a compiler the service never calls, and the obvious correction is the JRE image from the same publisher. It is not quite that simple, because two things I use live in the JDK rather than in the JRE: the JDWP agent the script attaches when DEBUG is true comes from jdk.jdwp.agent, and jcmd, jstack and jfr are what you reach for the first time a production container misbehaves. An image that cannot take a thread dump is a false economy. The version is a jlink runtime with those modules included, which is smaller than either and needs a module list this project has not worked out yet. Until then the JDK pays for itself, and the only reason this is on the list is size, which starts to matter the day the image is pulled rather than built in place.

None of the three is an oversight, and each one needs something this project does not have yet. Layering matters once the build runs often enough for the cache to pay, the runtime image is a module list rather than a one-line change, and the user question is properly answered alongside everything else about running this in an orchestrator. They are a chapter of their own, later, and the shape of it will be the same as every other chapter here: the first version was minimal, and the next one extends it.


Running it, two ways

A fresh clone should reach a running application without a document explaining how. One compose file, and an .env beside it, do it:

services:
  mysql:
    image: mysql:9.6.0
    environment:
      MYSQL_ROOT_PASSWORD: root
      MYSQL_DATABASE: ${DB_NAME}
      MYSQL_USER: ${DB_USERNAME}
      MYSQL_PASSWORD: ${DB_PASSWORD}
    ports:
      - "3306:3306"
    healthcheck:
      test: ["CMD", "mysqladmin", "ping", "-h", "localhost", "-uroot", "-proot"]
      interval: 5s
      timeout: 3s
      retries: 20

  advanced-service:
    container_name: advanced-service
    image: advanced-service:${TAG}
    depends_on:
      mysql:
        condition: service_healthy
    deploy:
      resources:
        limits:
          memory: 512M
    environment:
      DB_POOL_SIZE: 5
      JAVA_OPTS: >-
        -XX:MaxRAMPercentage=55
        -XX:MaxMetaspaceSize=128m
        -XX:ReservedCodeCacheSize=64m
        -XX:MaxDirectMemorySize=32m
        -XX:+UseSerialGC
        -Xss512k
        -XX:TieredStopAtLevel=1
      DEBUG: ${DEBUG}
      ACTIVE_PROFILE: dev
      DB_URL: jdbc:mysql://mysql:3306/${DB_NAME}?sslMode=DISABLED&allowPublicKeyRetrieval=true
      DB_USER: ${DB_USERNAME}
      DB_PASSWORD: ${DB_PASSWORD}
    ports:
      - "8787:8787"
      - "8080:8080"
Enter fullscreen mode Exit fullscreen mode

Four of the variables under environment are the ones the script insists on, spelled exactly as it reads them, and that is the whole point of owning the names: this block is readable without knowing that there is a Spring application behind it. Their values are not here, though. They come from an .env file sitting next to the compose file, which holds the database name, the credentials, the image tag and the debug switch, so the compose file describes the shape of the deployment and the .env holds the things that differ between one machine and another. The debug default there is false, because a file that is convenient to commit should not also be the one that attaches a debugger.

JAVA_OPTS is the longest entry and the one that matters most. The memory limit is declared once, under deploy.resources.limits, and the JVM is told to size itself as a percentage of it rather than given a fixed -Xmx, which is the difference between a container you can resize and a container you have to rebuild. The rest of that block is what a small limit actually requires: a metaspace ceiling, a bounded code cache, the serial collector rather than G1's several heap regions and threads, and a smaller stack. DB_POOL_SIZE is set to five in the same spirit, because the twenty-five from Part 2 is a production number and twenty-five connections are not free on either side of the socket.

depends_on with condition: service_healthy is worth the extra two lines over the bare form, which only waits for the container to be created and not for the database inside it to accept connections. With the condition, compose waits for the healthcheck. Flyway's retry settings from Part 1 then cover what remains, because a database that passes mysqladmin ping is not necessarily a database that has finished initialising, and the two mechanisms protect against different parts of the same gap.

The database port is still published even though the application now reaches MySQL over the compose network, and that is deliberate: it is what lets a client on the host inspect what the schema actually looks like once there is one.

The second way is the one to use while writing code, and the reason is the cycle rather than the debugger. Attaching to the container works - that is what DEBUG and 8787 are for, but every change still costs a rebuild of the jar, a rebuild of the image and a restart, where the same change from an IDE costs a recompiled class and often not even that. Bring up the database alone, copy the committed example file, and run the application from there:

docker compose -f docs/docker/compose.yml up mysql
cp local.properties.example local.properties
Enter fullscreen mode Exit fullscreen mode

The run configuration passes the file in, along with the profile:

--spring.config.additional-location=file:./local.properties --spring.profiles.active=dev
Enter fullscreen mode Exit fullscreen mode

Either way the log shows the same two things: Flyway connecting, finding no migrations to apply yet, and a context with ten beans that came from four configuration classes. curl localhost:8081/health answers UP from inside the container, or from the host when the application runs on it.

That empty migrations directory is the last loose end. Part 1 configured Flyway without showing what it would run, and the schema it is waiting for turns out to carry decisions that three earlier chapters have been quietly depending on, which is why it gets a chapter of its own rather than a paragraph here.


What the three parts add up to

Thirteen chapters of code that could not start now starts, and the shape of how it starts is visible in four places rather than inferred from annotations. The @Import list is the inventory. The four configuration levels each answer one question. The entrypoint script states its requirements and refuses to run without them. The actuator says whether the process is alive, whether it should receive traffic, which commit it was built from, and where to find the log.

Several of the decisions in these three parts cost something, and it is worth naming them. Disabling the component scan means a line of typing per configuration class. Declaring beans by hand means three lines per service instead of one annotation. Translating environment variables by hand means the script needs maintaining and a new setting does not work automatically. In exchange, every one of those things is a place where a reader can see what the system is made of without running it, and ProductsServiceImpl is still a class that a test can instantiate with new.

What the application cannot do is answer a single request. There is no controller, no mapping, no request body, and the exception handler from Chapter 11 has nothing to handle, because nothing can be asked of it yet.


What's Next?

The application starts, and the first thing it does is look for migrations and find none. Chapter 12 argued at length that the only guard against assigning the same product to a category twice is a unique constraint, and that constraint has been a sentence in a paragraph ever since.

Chapter 14 is that file. Why four tables are one migration rather than four, why the identifier column is sixteen bytes in a particular order, what a version column changes about the service layer, and which of the handler mappings from Chapter 11 turns out to be missing once optimistic locking exists.

The three problems with the image stay open until there is a reason to solve them, which is the chapter on deploying this rather than running it.


Codebase & Architecture Blueprint

The entire evolutionary architecture of this project is tracked using strict Git tags. To clone the repository and switch exactly to the state established in Chapter 13, Part 3, use the following link:

Note: All core modules are configured with strict compilation-level boundaries. Compile and run mvn clean install to see the structure in action. Maven version 3.9.* and Java 25 are required.

◀️ Read Chapter 13 Part 2: Bootstrapping the System, Where the Settings Come From
▶️ Read Chapter 14: The Schema Three Chapters Were Already Relying On (Coming soon)


Join the Masterclass Journey

This article is part of an ongoing weekly series. If you want to receive every upcoming chapter directly in your inbox, alongside deep-dives, infrastructure architecture diagrams, and premium insights before anyone else, subscribe to my Substack Newsletter here!


Recommended Resources for Java & Spring Boot Engineers

Disclosure: I earn a commission if you purchase through the links below, at no additional cost to you.

If you are preparing for Senior/Lead Java interviews or looking to solidify your Spring Boot & Architecture skills, check out these highly-rated resources from the Javarevisited publication (Use promo code friends20 for an exclusive 20% discount automatically applied at checkout):

Top comments (0)