DEV Community: David

Expense Tracker: Learning Performance Testing and Continuous Profiling

David — Thu, 09 Apr 2026 07:00:00 +0000

(Originally posted on Medium, Photo by Matheus Bertelli: https://www.pexels.com/photo/close-up-of-cooling-system-inside-gaming-pc-34552800/)

In this article, I learned how to precisely identify CPU bottlenecks in my code using everything I’ve learned so far, including a new tool: Pyroscope.

Introduction

As a brief review, Expense Tracker is my personal solution to recording my expenses, where I took the opportunity to learn about unit testing, continuous integration, and observability. It exists as a standalone Android app, but this series is about the client-server version with Spring Boot. After I was done with the basic features of the app, I decided to find out how it would perform in the real world, and so I came across performance testing. The two types of performance tests I’m concerned with are Load and Stress tests. There are other types like soak and chaos tests, but I chose to focus on the basics. Load tests are to anticipate the resources needed for standard load conditions, while stress tests are meant to find out how much pain a system has to endure before it caves in entirely, and if it does so gracefully.

Load Testing vs Stress Testing: The Main Differences | BrowserStack

Understand the key differences between load testing and stress testing and why both are crucial for optimizing your application's performance.

browserstack.com

TL;DR

The Bottleneck: Identified a critical N+1 query issue causing 80s+ latencies and 4,000+ spans in a single trace.
The Fix: Reduced latency by 97% using database JOINs and in-memory Hash Map caching.
Thread Pool Tuning: Optimized JWT generation by aligning Tomcat thread pools with v-core counts to minimize context-switching.
The Reality Check: Concluded that BCrypt slowness is a non-negotiable security feature, choosing to rate-limit the app rather than compromise hashing strength.
Final Verdict: Stabilized the system at 10 RPS with a 100-thread cap for a 2 v-core staging environment.

Preparation

My initial objective was simple:

What are the limits of my system?

To find that out, I made sure to observe the Grafana dashboards I built when I was learning about metrics, including:

JVM metrics:

MySQL Metrics:

Request, Error, Duration (RED) dashboard

I also followed this article as a guideline and decided to come up with a basic test script based on a typical user flow:

Test Script

Log in
- POST /auth/login
Enter the draft screen
- POST /transactions/draft
- GET /transactions/draft
- GET /categories
Change the account from the default
- GET /accounts
- PUT /transactions/draft
Attach the relevant documents
- POST /transactions/draft/documents
Modify the items
- PUT /transactions/draft/items/{uuid}
Save the draft and view the main page
- POST /transactions
- GET /transactions
View statistics according to certain criteria
- POST /statistics/... (5 different endpoints)
View the filtered transactions themselves
- POST /statistics/transactions

Testing

My tool of choice is Apache JMeter, and for all my tests, I used the Constant Throughput Timer, which essentially dictates my Requests per Second, or RPS, so long as the server can keep up. I was iteratively building my JMeter script as I slowly discovered my system limits each run. Some key points include:

My MySQL never showed any real signs of being anywhere near breaking — the memory, row locks, and slow queries were consistently okay
I never encountered any memory issues for my JVM. The graphs all pretty much looked like this:
Earlier on in my tests, I also noticed that my CPU kept peaking at certain points, but I didn’t think much of it. I later learned why that was important.

VM CPU:

Host CPU:

I noticed various things and changed a number of other things over time, but I’ll leave them out from here for brevity.

Developing Acceptance Criteria

During my tests and research, and after being comfortable with gauges and counters, I discovered the usefulness of the third metric type in Prometheus: histograms. They help to answer the question:

What latency did 90% of my users experience?

The graph above is capped at 10 here because those are the settings prescribed by the OpenTelemetry semantic conventions. Histograms work with buckets, and so precise detail can be hidden — it’s possible that a majority of requests take significantly longer than 10 seconds, or alternatively, it’s likely that a large portion of request durations are towards the tail end of a bucket boundary. Errors of estimation are actually expected because of this, and one can choose their own bucket boundaries to fit their use case. I stuck with the defaults because 10 seconds sounds long enough to me for a bad user experience.

I initially focused on averages, but I later realised why histograms are worth considering. Histogram quantiles are more helpful in my case because of the possibility of outliers skewing the results.

With this new information, I decided to rework my dashboards and make use of Grafana’s thresholds:

Response time p95 latency:
- 0 to 3 seconds: OKAY/green;
- 3 to 7 seconds: FAIR/yellow;
- 7 to 10 seconds: BAD/red
Memory and CPU: 90%
Storage: 70%
Server error rate (HTTP 4xx): 10%
Client error rate (HTTP 5xx): 30%

These numbers come from what I’ve decided to be an acceptable user experience. With the thresholds established, my JVM dashboard looks like this:

And my RED dashboard looks like this:

At that point, my objective changed:

Optimize the bottlenecks

There were three offenders:

Upload Document
Get transactions
Login

The document endpoint was mostly fixed by changing my hardware from an HDD to an SSD, but the others were a bit more involved

Optimizations

GET /transactions

Problem

I was making unnecessary round-trip queries to the database, severely increasing the latency

Solutions

Use a JOIN query to fetch all the needed data at once
For nested entities, also use hash maps to cache them and make “in-memory” joins.
Improvement: my worst-case trace went from 1 minute 21 seconds down to 2.2 seconds (97% improvement)

Details

My test script essentially acted like a mass insertion script, so I wanted to see how the app would handle the roughly 777 transactions I had inserted by that point, but after logging in to the test user account and opening the screen, it crashed. After looking at Tempo, I found out why:

My endpoint took 1 minute and 20 seconds, but the app crashed due to a 10-second timeout, which is OkHttp’s default. It took so much time because I made an excessive number of round-trips to the database to fetch nested entities. That’s why this trace has almost 4,000 spans. On Markus Winand’s website for database indexing, this problem is called the N+1 query problem, where 1 is the initial query to fetch the data and N is the number of rows fetched. The solution is to use a join query. Once I made the change, the time drastically improved — the same endpoint went down in duration by 97% to 2.2 seconds.

POST /auth/login

Problems

JWT generation was stressing the CPU
Password hashing was also stressing the CPU

Solutions

Limit JWT generation to a CPU-sized thread pool
No more software optimizations to be made. Get better hardware
Token generation was improved by 64% from 1 minute 8 seconds to 24 seconds in my worst case, but password hashing still took the bulk of the time

Details

When I checked my traces — which are usually more accurate than histograms due to their different use cases — I saw that the time would get as bad as 37 seconds, so I had a good amount of trouble with this one. I thought that maybe private key and JWT generation were computationally expensive and just took that much time, and I initially didn’t know how to verify that. After adding timer logs and searching with Loki, I saw something interesting:

There’s a point where it suddenly goes from taking over 13 seconds to less than 1 second to generate that JWT. Quite confusing. I didn’t know how to diagnose the problem when the output was so divided.

I eventually discovered profiling as a potential tool to help, and I chose Grafana Pyroscope as my tool. I initially tried VisualVM, but setting each one up required roughly the same amount of effort, so Pyroscope won out since it’s part of the Grafana ecosystem I’m already invested in.

With Pyroscope set up, I searched for my JWT method, createUserToken, and found that it had taken over 1 minute of CPU time out of the 23 minutes that the newly profiled test ran for. After looking into it and asking Gemini, I discovered that context-switching was my main issue: CPU-bound tasks shouldn’t be performed across a lot of threads, as the overhead is high. They should only have as many threads as there are cores. Spring’s default thread count for Tomcat is 200. This pool size isn’t a problem for I/O threads, which in this case is for server requests, but by letting each worker thread call the JWT function, I was slowing everything down.

I further confirmed this suspicion by returning to my CPU graphs

VM:

JVM:

Host (the VM had 4 out of 8 cores):

Once I looked at the timings and compared them with my RPS for the login endpoint, I saw that they were closely aligned, which proved to me that the endpoint’s volume accounted for the increased CPU load.

The improvement was made by delegating the task to a thread pool sized with Runtime.availableProcessors() (link) and verified by testing with JUnit. After deploying my change, I saw that the optimization worked. Here are the before and after screenshots:
Before:

After:

An improvement from 1 minute 8 seconds to 24 seconds. However, the endpoint still had a bad p95, so after more digging and sorting the table by Self instead of Total, and that’s where I found the worst offender:

Password hashing with BCrypt

What I tried:

Creating a custom subclass of the Password Encoder in order to delegate it to a thread pool to achieve the same speed-up effect as JWT generation. After testing it with 5 consecutive JUnit runs, my best time was roughly 26 seconds whereas the target was 10.
Rate limiting with the endpoint with Bucket4J. I started trading latency for throughput and couldn’t really find a good middle ground

After searching around for other ways to optimize my code and not finding anything, I decided to stop for 3 main reasons:

The slowness of password hashing is a security feature. I shouldn’t work against that
My hardware has reached its limit
A massive simultaneous login isn’t a regular user pattern

The Verdict

I had tried various other conditions, including:

Increasing the RPS of the load test
Increasing the Tomcat thread count
Varying the user count in my JMeter script
Using my staging VM to determine my final resources

The VM I chose has 2 v-cores and 2 gigabytes of RAM, and factoring that in, my final decisions include:

Rate limit the app to 10 requests per second
The Tomcat thread count should be capped at 100

Conclusion

The project was a fun learning experience overall, and now I have a good idea of what it’s like to architect an app from end to end with performance, automation, and code stability in mind. It was especially nice learning the hard limits and what I could and could not improve.

Thank you for your time

Learning Full-stack Observability: Tracing

David — Wed, 08 Apr 2026 07:00:00 +0000

(Originally published on Medium, Image by Gerd Altmann from Pixabay)

Introduction

Distributed Tracing serves the purpose of connecting network calls across various services within an organization. In my case, that means I can:

From Android: see the time taken for the request to get sent from the device
From Spring Boot: see the potential error logs a certain user may have
From the database: see how much time is spent to determine the efficiency of my queries

Traces are made of spans, and spans have attributes, and a span may have a parent span. A span without a parent is known as the root span. More about that in the OpenTelemetry documentation

In this example:

The root span is named GET, starting at the Android app.
It’s composed of 5 spans, 3 of which are database queries, and the longest-running one is the root.
The next span after the app starts at the backend
The trace took 1.91 seconds in total.
As mentioned before, my choice of tools for OpenTelemetry will be the ones from the Grafana stack, and so I’ll be using Tempo.

Trace Propagation

Briefly, I’ll go over how traces work. The W3C Trace Context and OpenTelemetry documentation explain in much better detail.

Spans are generated within each service before being sent to the span collector. Spans are able to be created with ancestral information using trace headers.

A trace header is passed from the first component down the service trail. If a service has tracing enabled and doesn’t detect a compatible trace header, it will create one. It comes in multiple formats, but W3C is the main header format.

Since the Android app is instrumented, the trace starts from there; otherwise, it would have started and ended at the backend as a single-service trace

And the resulting trace can be viewed in Tempo

Trace Correlation

This allows one to filter logs by HTTP request instead of having to narrow down the potential time of the request. Correlation connects traces to the other 2* types of OpenTelemetry — logs and metrics — and allows one to jump back and forth between them.

(* profiles are an upcoming 4th signal in OpenTelemetry. As of now, Grafana Pyroscope can be used with traces, but that will be discussed in the performance test)

Log Correlation with Loki

The key attributes here are trace_id and span_id. These have to be set as structured metadata of the log entry. In Spring, these can be extracted using the request headers, such as with RequestContextHolder(link). The request attributes then have to be converted into log attributes that can be parsed using a Promtail pipeline, as I explained in my logging article. With an auto-instrumented app such as one using the Java Agent, these steps can be skipped as the attributes are extracted automatically.

Loki to Tempo

After getting the trace attributes, I configured Loki to use Derived Fields to link to Tempo.

And this is an example of what will appear when properly configured

Note the “tempo” buttons:

Tempo to Loki

grafana.com

Example link:

When configured properly, this “LOG” button, or a chain link icon, will appear on the traces.

Metric Correlation with Prometheus

For Prometheus, it’s slightly more complicated. All labels are indexed, so a high cardinality field like trace_id is out of the question. There’s no such thing as “metadata labels” as there is with Loki, either. Instead, exemplars are needed; the feature is currently available, but still experimental

For some reason, I’ve only gotten it to work with histogram_quantile.

There’s also Traces to Metrics, but I won’t talk about it here.

Custom Trace Attributes

My main use case for this is to allow traces to filtered by a user's device, to act as a very basic form of session replay.

Instrumentation

Because of the fundamental differences between Android’s JVM and the standard JVM, auto-instrumentation is generally not available, so I’ve gone the route of using manual instrumentation.

Retrofit is my library of choice for making HTTP calls, and so I chose to use the OkHttp instrumentation.

This is a summarized version of my Dagger Provides method for the Retrofit object:

@Named("authorized")
fun retrofitAuthorized(buildConstants: BuildConstants, openTelemetry: OpenTelemetry): Retrofit {
    val okHttpClient = OkHttpClient.Builder()
        .addNetworkInterceptor {
            val span = Span.current()

            if(span.isRecording) {
                val userId = dataProvider.userId()
                if (userId != null) {
                    span.setAttribute(
                        Constants.CustomTraceAttributes.UserId.attributeName,
                        userId.toString()
                    )
                }
                val deviceId = dataProvider.deviceId()
                if (deviceId != null) {
                    span.setAttribute(
                        Constants.CustomTraceAttributes.DeviceId.attributeName,
                        deviceId.toString()
                    )
                }
            }
            it.proceed(it.request())
        }
        .build()

    val callFactory =
        OkHttpTelemetry.builder(openTelemetry)
            .build().newCallFactory(okHttpClient)
    val baseUrl = buildConstants.exTrackBaseUrl()

    val retrofit = Retrofit.Builder()
        .baseUrl(baseUrl)
        .callFactory(callFactory)
        .build()
    return retrofit
}

I’m using Span.current() here because the instrumentation already starts a span through a custom interceptor.

And the definition for DeviceDataProvider is simply this:

interface DeviceDataProvider {
    fun deviceId(): UUID?
    fun userId(): UUID?
}

Where I store the device ID in a SharedPreferences file.

And the result is that I can search Tempo with a query like this:

{resource.service.name="expense-tracker-android"} && {span.extrack.deviceId = "8ea68a2d-ffba-42a8-970c-02a397210fb0"}

I also added these attributes to my Loki logs because I could afford to have metadata without indexes, but as discussed in my Prometheus article, I didn’t add these to my metrics because of high cardinality

Conclusion

With tracing in place, I now have a tool which works as a form of "super-logger" to help me narrow down API calls faster.
Now that I’ve concluded the “OpenTelemetry” portion of my ExTrack series, I’ll be discussing how I was able to do performance tests on my server and how all 3 signals provided great value to me. That will be all for now.

Thank you for your time.

Learning Full-stack Observability: Logging

David — Tue, 07 Apr 2026 07:00:00 +0000

(Originally published on Medium, Image by Boskampi from Pixabay)

Introduction

When it comes to logging, there are multiple concerns, some of which I'll highlight for this post:

Retention: to prevent server disk space from becoming scarce
Search: the core use case of logs
Formatting: aids in readability and search

Across different tools and platforms such as programming language and Operating System, these aspects vary significantly. One of the main points of OpenTelemetry is to have a central place to view all those logs, whether they come from:

A file on the local machine, like with my MySQL server

A file on a remote machine, like with my NGINX server

A mobile device, like with ExTrack:

As mentioned in my earlier articles, my tools of choice for OpenTelemetry come from the Grafana stack, so I’ll be using Loki.

Loki

Loki’s main description is:

Like Prometheus, but for logs.

So that’s why I made this article to be after the Metrics article.

Loki mainly relies on labels in order to narrow down logs while searching, similar to Prometheus.

(service_name and level, 2 of the most common labels in my case)

It also supports structured metadata, which is similar to labels, but they’re not indexed, and so they can have high cardinality, such as an IP address or a user ID. The main idea when working with Loki is to filter by a few key indexed labels, and then narrow down with the various search operators.

The 2 main ways I push to Loki are:

Scraping the raw log files with Grafana Promtail*, which then pushes them to Loki
Using the OpenTelemetry Java Agent in combination with Grafana Alloy using OTLP

(*Promtail is at end-of-life, Alloy is the recommended alternative)

Promtail

At a basic level, Promtail uses a pipeline to process log lines into the final output to be sent to Loki. Pipeline consists of a wide variety of stages, which include, most notably in my case, the json stage, since I can use JSON for both the MySQL error log and NGINX access logs.

MySQL Promtail Config

- job_name: mysql_error_logs
  static_configs:
  - targets:
      - localhost
    labels:
      service_name: mysql_error_logs
      __path__: "C:/ProgramData/MySQL/MySQL Server 8.0/Data/LAPTOP-7H9JJDHB.err.00.json"
  pipeline_stages:
  - json:
      expressions:
        time: time
        msg: msg
        prio: prio
        level_derived: prio
        level: label
        err_code: err_code
        err_symbol: err_symbol
        SQL_state: SQL_state
        subsystem: subsystem
  # https://dev.mysql.com/doc/refman/8.4/en/error-log-event-fields.html
  - template:
      source: level_derived
      template: '{{ if eq .Value "0" }}System{{ else if eq .Value "1" }}Error{{ else if eq .Value "2" }}Warning{{ else }}Note{{ end }}'
  - timestamp:
      source: time
      format: RFC3339
  - labels:
      level:
      level_derived:
  - structured_metadata:
      prio:
      err_code: err_code
      err_symbol: err_symbol
      SQL_state: SQL_state
      subsystem: subsystem
  - output:
      source: msg

I extract all the key-value pairs using the json stage
The timestamp stage is there to ensure Promtail uses the time from the log statement instead of the time the file was last scraped
I chose to turn the level attribute into a label because log levels are usually the most important label, and also because the cardinality is quite low — 4.

Example

Equivalent logs in Loki:

NGINX Logs

- job_name: nginx_access_logs
  static_configs:
  - targets:
      - localhost
    labels:
      service_name: nginx_access_logs
      __path__: "/var/log/nginx/access-json.log"
  pipeline_stages:
  - json:
      expressions:
        timestamp: timestamp
        remote_addr: remote_addr
        message: message
        status: status
        request_method: request_method
        hostname: hostname
        trace_id: trace_id
        span_id: span_id
        server_name: server_name
  - timestamp:
      source: timestamp
      format: RFC3339
  - labels:
      status:
      request_method:
  - structured_metadata:
      remote_addr:
      hostname:
      trace_id:
      span_id:
      server_name:
  - output:
      source: message

Here, once again, I chose just 2 labels with a combined maximum cardinality of 4,500: 9 for HTTP methods times 500 for the range of status codes. Although practically it will be much smaller since not all of them are used.

And here is the custom NGINX log format:

log_format json escape=json '{"remote_addr":"$remote_addr","timestamp":"$time_iso8601","message":"$request","status":"$status","request_method":"$request_method","hostname":"$hostname","trace_id": "$otel_trace_id","span_id":"$otel_span_id", "user_agent": "$http_user_agent","server_name":"$server_name"}';

access_log  /var/log/nginx/access-json.log  json;

Java Agent

Backend

For the backend, the Java Agent automatically exports the Logback logs using OTLP and sends them to Grafana Alloy, where some resource attributes get indexed to labels by default. The relevant ones for me include deployment.environment.name(link) to differentiate dev, staging, and production; and service.name(link).

My configuration using environment variables for OpenTelemetry and Java:

export APP_VERSION=$(cat app-version)
export OTEL_SERVICE_NAME=expense_tracker_backend
export OTEL_JAVA_AGENT_LOCATION=/opt/opentelemetry
export JAVA_TOOL_OPTIONS="-javaagent:$OTEL_JAVA_AGENT_LOCATION/opentelemetry-javaagent.jar"
export SPRING_PROFILES_ACTIVE=dev
export OTEL_EXPORTER_OTLP_ENDPOINT=http://telemetry.davidgrath.com:4318
export OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name=dev,deployment.environment=dev,service.version=$APP_VERSION
java -jar "/opt/expense_tracker/expense-tracker-backend.jar"

The domain name is my local DNS entry that points to my Grafana Alloy instance:

Which is configured like this:

otelcol.exporter.otlphttp "loki_exporter" {
    client {
        endpoint = "http://localhost:3100/otlp"
    }

}
otelcol.receiver.otlp "otlp_receiver" {
    grpc {
    }

    http {
    }
    output {
        logs = [otelcol.exporter.otlphttp.loki_exporter.input]
    }
}

Example

Resulting Loki entry

Android

For the Android version, the JVM is fundamentally different, and so auto-instrumentation isn’t a thing. Manual instrumentation is needed and can be achieved with the Logback library

I’m using Dependency Injection with Dagger, so here’s how the OpenTelemetry object is configured in my Dagger Provides method:

@Provides
@Singleton
fun openTelemetry(buildConstants: BuildConstants): OpenTelemetry {
    val logsUrl = "${buildConstants.telemetryHttpUrl()}/v1/logs"

    val resource = Resource.builder()
        .put(ServiceAttributes.SERVICE_NAME, "expense-tracker-android")
        .put("android.os.api_level", Build.VERSION.SDK_INT.toString())  //Gotten from semconv docs
        .put("deployment.environment.name", buildConstants.environmentName())
        .build()
    val sdk = OpenTelemetrySdk.builder()
        .setPropagators(
            ContextPropagators.create(
                TextMapPropagator.composite(
                    W3CTraceContextPropagator.getInstance(),
                    W3CBaggagePropagator.getInstance()
                )
            )
        )
        .setLoggerProvider(
            SdkLoggerProvider.builder().addResource(resource)
                .addLogRecordProcessor(BatchLogRecordProcessor.builder(OtlpHttpLogRecordExporter.builder().setEndpoint(logsUrl).build()).build())
                .build())
        .buildAndRegisterGlobal()
    OpenTelemetryAppender.install(sdk)
    return sdk
}

Forgetting to append /v1/logs gave me a bit of trouble before I found out.
With the help of Mapped Diagnostics Context, I’m able to attach a randomly generated UUID to be able to filter my logs by deviceId:

Logging crashes

There are better tools for the job, including ACRA and Firebase Crashlytics. There’s even one within OpenTelemetry itself, but I wanted something basic that I could see directly in Loki.

By making use of the OpenTelemetry SDK and UncaughtExceptionHandler, I’m able to upload the log before the app shuts down:

val defaultUncaughtExceptionHandler = Thread.getDefaultUncaughtExceptionHandler()

val handler = object: UncaughtExceptionHandler {
    override fun uncaughtException(t: Thread, e: Throwable) {
        val processor = appComponent.logRecordProcessor()
        LOGGER.error("Uncaught exception", e)
        processor.forceFlush().join(200, TimeUnit.MILLISECONDS)
        processor.logRecordExporter.flush().join(200, TimeUnit.MILLISECONDS)
        //VERY IMPORTANT! Return the flow back to the system
        defaultUncaughtExceptionHandler?.uncaughtException(t, e)
    }

And I can see what happened in Loki:

This method feels a little hacky since I’m accessing the SDK directly, but I’ll work with it since I got my logs

Configuration summary

Conclusion

Now that my configuration is complete, I have a central place where I can search my logs across all my servers and services, while still letting SSH be a viable alternative.

In the next article, I’ll make use of distributed tracing to enable me to track the flow of execution from the app to the backend to the database using Tempo.

If you need any feedback, feel free to discuss in the comments

Thank you for your time.

Learning Fullstack Observability: Metrics

David — Mon, 06 Apr 2026 07:00:00 +0000

(Originally published on Medium)

Introduction

According to RedHat:

Observability refers to the ability to monitor, measure, and understand the state of a system or application by examining its outputs, logs, and performance metrics.

OpenTelemetry is a vendor-neutral framework for observability, and is made of 3* main pillars:

Metrics
Traces
Logs

(* Profiles are an upcoming 4th pillar, but still in development)

In the “Expense Tracker” series, this article will be the first of 3 where I apply OpenTelemetry to my app, primarily with the Grafana stack:

In order to find out which REST API requests are the slowest, what services are hogging the most memory, and how saturated my web server's thread pool is, I need to track the numbers behind them, and the tool fit for that is the first in this observability series: Prometheus.

For Expense Tracker, the exporters I’m interested in include:

Node exporter: for my Linux VMs
Windows exporter: for my host machine
MySQL exporter: for my database
NGinX exporter
OpenTelemetry Java Agent: for the backend. I could have also used Spring Actuator or the JMX exporter, but the agent is simpler for this project since it emits all 3 types of telemetry signals.

I’ll be going over the types of metrics used in Prometheus and how I apply them to create my Grafana dashboards.

Prometheus Metric Types

Metric types | Prometheus

Prometheus project documentation for Metric types

prometheus.io

Gauges

How much disk space is available on the system?

The answer is a number that can go up or down as the server runs. This is one of the main metric types in Prometheus, and for the Windows exporter, that metric looks like this:

windows_logical_disk_free_bytes{volume="C:"} 2.6363297792e+10

The first part is the actual metric. The second part in the curly braces is a label, and the number after is the actual value of the metric. The Windows exporter exposes this, and a lot of other metrics at http://localhost:9182/metrics, and Prometheus is made to scrape them at a defined interval, and with enough scrapes, we get a pretty graph:

Counters

How much load is my storage drive experiencing?

The answer lies with a counter. For Windows, the counter looks like this:

Counters only ever go up, unlike gauges, but the key to using them is in finding out how quickly they go up. That’s achieved with the rate function:

rate(windows_logical_disk_write_bytes_total{}[15s])

The time window in square braces needs to be at least bigger than the scrape interval. In my case, I chose a scrape interval of 2.5 seconds, so a rate function with a window of 1 second would return no data.

Histograms

Histograms are more complex, but in general, they’re for aggregation, and the question they helped me answer while I was running my load tests is:

What’s the minimum latency that 95% of requests to GET /transactions are getting?

And simply put, the answer is gotten with a histogram quantile.

histogram_quantile(0.95, 
  sum by (le,http_route) 
    (rate(http_server_request_duration_seconds_bucket{http_route="/transactions",http_request_method="GET"}[15m]))
)

The 0.95 is the actual percentile, sum is to aggregate by various labels like request method and request route, and rate is mainly used to adjust the time window.

The graph caps off at 10 because of the default bucket boundaries of le specified by the OpenTelemetry semantic conventions for HTTP metrics:

[ 0.005, 0.01, 0.025, 0.05, 0.075, 0.1, 0.25, 0.5, 0.75, 1, 2.5, 5, 7.5, 10 ]

le is a special label for “less or equals”, which is used to form a cumulative sum with the relevant metric.

This means that if most of the requests were significantly above 10 seconds, I’d have no way of knowing that from Prometheus.

My options include:

Relying on Tempo with a query to filter out the minimum duration traces

{traceDuration > 10s && resource.service.name="expense_tracker_backend"}

Use Tempo Metrics Generator, although by default the bucket only goes up to 16.384 seconds, which was much less than what I was experiencing at the extreme in my load tests
Use Tempo in combination with exemplars to find samples of the worst offenders
Customizing my bucket boundaries to include larger values.

I talk about how my performance test reached such high latencies in a later article.

Grafana Dashboards

Using what I’ve learned in combination with the various collectors lets me build some fun dashboards:

MySQL

Variables reference linked here.
Some of the panels include:

InnoDB memory usage to know if the server is running low
Bytes sent and received per second
Count of slow queries to know if any potentially inefficient queries are running

Node Exporter

Here I monitor the hardware similar to the way it's shown in Linux top or Windows Task Manager

RED Dashboard

Where RED stands for

Rate: number of requests per second, here I show the overall and also break it down by endpoint
Errors: the error rate. I chose to keep one panel for server errors and another for client errors
Duration: here I use p95 as discussed earlier, with a time window of 1 hour

Setup

I’ll briefly cover the actual setup and configuration I used

Metric scraping

The most basic configuration to get started with Prometheus is the scrape_configs block with the static_configs parameter:

scrape_configs:
 - job_name: windows_exporter
   scrape_interval: 2500ms
   static_configs:
    - targets: ['localhost:9182']
      labels:
        host_name: "LAPTOP-7H9JJDHB"

The config shown is for Windows, but it’s similar to my configuration for the other services

OpenTelemetry

Enabled at the command line with:

--web.enable-otlp-receiver

More information here. It uses a push-based model, which differs from Prometheus main pull-based model. This is mainly for use with Alloy, which is Grafana’s distribution of the OpenTelemetry collector. My configuration looks like this:

otelcol.exporter.otlphttp "prometheus_exporter_otlp" {
    client {
        endpoint = "http://localhost:9090/api/v1/otlp"
    }
}

otelcol.receiver.otlp "otlp_receiver" {
    grpc {
    }

    http {
    }
    output {
        metrics = [otelcol.exporter.otlphttp.prometheus_exporter_otlp.input]
    }
}

And my resulting Alloy graph is this:

Image by juicy_fish on Freepik

Alerting

CPU, Disk usage, and RAM usage are all very volatile metrics compared to storage usage, so I grouped them separately and set different time intervals for alerts to be sent, and then set the notifier to be the email one.

The basic setup looks like this:

And a sample alert looks like this:

Conclusion

So that’s pretty much it. This is the first of the 3 Observability articles, and the next one will be about how I gather all the logs from my various services into a central place using Loki.

Please feel free to discuss your thoughts and suggestions in the comment section

Thank you for your time

Expense Tracker: Learning CI/CD

David — Mon, 09 Mar 2026 07:00:00 +0000

(Originally published on Medium)

Introduction

At some point, I had no idea what CI and CD were about, so I decided to learn about them. I got to learn enough to be able to do a simple Build-Test-Deploy cycle, so now I know enough to define them in my own words:

Continuous Integration: JUnit tests are run on every commit to the dev branch
Continuous Delivery: the JAR is also built and sent straight to the test server when the tests pass, with potential smoke tests such as a “User can log in” test being run.
Continuous Deployment: the build is also sent to production after the smoke tests pass
Sten from Atlassian does a great job of explaining the differences between the 3 terms

My tool of choice is Jenkins, mainly because I prefer a self-hosted approach, and I’ll talk about the Jenkinsfile I wrote for Expense Tracker.

Architecture overview

I have 2 main VM’s, both of which are running Ubuntu

Controller This is where Jenkins itself runs.
Test Server This is where the “dev” profile of the app will run. It’s also where I’ll be running my tests with the “ci” profile.

Jenkinsfile Walk-through

So at a basic level, the tasks mainly involved:

Resetting the database
Generating the relevant code
Testing
Building
Deploying

MySQL Credentials

Using credentials

Jenkins – an open source automation server which enables developers around the world to reliably build, test, and deploy their software

jenkins.io

environment {
  MYSQL_CREDS = credentials('mysql_credentials')
}

The credentials used here consist of a username and password combination stored within Jenkins. Other credential types I use in the file include Secrets Text for Ansible and a Secrets file for the Spring configuration. I did this to make sure I never have to commit my credentials directly to source control. HashiVault is also an option, but I decided it wasn't needed for my use case.

Use agent “Test Server”

Using Jenkins agents

Jenkins – an open source automation server which enables developers around the world to reliably build, test, and deploy their software

jenkins.io

Managing Nodes

Jenkins – an open source automation server which enables developers around the world to reliably build, test, and deploy their software

jenkins.io

agent {
  node {
    label 'Test Server'
  }
}

Here, I used ssh-keygen to generate a private key and copy it over as a secret into Jenkins.

Flyway

stage('Flyway') {
  steps {
    sh 'chmod +x ./gradlew'
    sh './gradlew setUpCIDb'
    dir ('database/flyway') {
      sh 'flyway migrate -environment=ci  -user=$MYSQL_CREDS_USR -password=$MYSQL_CREDS_PSW'
    }
  }
}

Here, I drop and recreate the database before running migrations on it. There’s an exclusive database for this task, and it is suffixed with “_ci”. Flyway was downloaded and symlinked on this machine into /usr/bin for this to work, and I created an environment just for this

jOOQ

stage('JooQ') {
  steps {
    sh './gradlew jooqCodeGenCi'
  }

jOOQ enables me to work with a schema-first approach for the database by generating classes based on the tables and views I have, so this block is necessary for all the needed classes to be present.

System properties governing code generation

jooq.org

The main issue I had here was that even though jOOQ supports using command-line properties, it didn’t work until I used the driver, url, username, and password all on the command line. Mixing command-line authentication with configuration file properties didn’t work.

tasks.register('jooqCodeGenCi', Exec) {
    dependsOn jooqCodeGenCopy

    workingDir "${rootDir}/database/jooq"
    commandLine "java", "-Djooq.codegen.propertyOverride=true", 
      "-Djooq.codegen.jdbc.driver=com.mysql.jdbc.Driver", 
      "-Djooq.codegen.jdbc.url=jdbc:mysql://172.26.48.26:3306/expense_tracker_database_ci", 
      "-Djooq.codegen.jdbc.username=${System.getenv("MYSQL_CREDS_USR")}", 
      "-Djooq.codegen.jdbc.password=${System.getenv("MYSQL_CREDS_PSW")}", 
      "-Djooq.codegen.propertyOverride=true", 
      "-cp", "*",
      "org.jooq.codegen.GenerationTool", 
      "configuration-ci.xml"

}

OpenAPI

stage('Open API') {
  steps {
    sh './gradlew openApiGenerate'
  }
}

Similar to jOOQ, I use an API-first or spec-first approach in my code, meaning that all of the entities and paths that my controllers rely on need to be generated first.

Documentation for the spring Generator | OpenAPI Generator

METADATA

openapi-generator.tech

https://github.com/OpenAPITools/openapi-generator/blob/master/modules/openapi-generator-gradle-plugin/README.adoc

Build

stage('Build') {
  steps {
    sh './gradlew assemble'
  }
}

Compiles the classes

Test

Using environment variables

Jenkins – an open source automation server which enables developers around the world to reliably build, test, and deploy their software

jenkins.io

Pipeline Syntax

Jenkins – an open source automation server which enables developers around the world to reliably build, test, and deploy their software

jenkins.io

environment {
  SPRING_PROFILES_ACTIVE='ci'
}

I start by setting the active Spring Profile to “ci”, one that I’ve made exclusively for this pipeline.

sh 'rm -f app/src/main/resources/secrets-ci.properties'
sh 'chmod o+w -R app'
withCredentials([file(credentialsId: 'expense-tracker-backend-secrets-ci', variable: 'expenseTrackerApplicationSecrets')]) {
  sh 'cp $expenseTrackerApplicationSecrets app/src/main/resources/secrets-ci.properties'
}
sh './gradlew test'

I then remove any potentially existing secrets file before copying over a new one from Jenkins credentials. What I store there includes:

The JDBC password
The JWT private key

The -f flag is to ensure an exit code of 0 if the file does not exist, because with a non-zero exit code, the pipeline will fail.

And then I test. How exactly I test is discussed in the previous article

Flyway Dev

stage('Flyway Dev') {
  steps {
    dir ('database/flyway') {
      sh 'flyway migrate -environment=dev  -user=$MYSQL_CREDS_USR -password=$MYSQL_CREDS_PSW'
    }
  }
}

Exactly like my first Flyway migration, but this time for my “dev” environment. There’s no DROP-ing this time, though

Since there were no issues with the CI migration, I’m confident there will be no issues with this migration.

Ansible

stage('Ansible setup') {
  agent {
    node {
      label 'built-in'
    }
  }
  environment {
    ANSIBLE_BECOME_PASS = credentials('ansible-become-password')
  }
  steps {
    dir ('ansible') {
      sh 'ansible-playbook -i inventory playbooks/setup_service.yaml  --private-key ~/.ssh/jenkins_agent_key -e "ansible_become_password=\'$ANSIBLE_BECOME_PASS\'"'
    }
  }
}

I won’t go into the full Ansible playbook here, but it essentially configures my app to run with the OpenTelemetry Java Agent and Pyroscope using SystemD.

I needed to run this off the Jenkins host machine, which is Controller in this case — the built-in node, and my main issue was finding the name to use from the docs. I found it in a forum instead.

Deploy

stage('Deploy') {
  steps {
    sh './gradlew bootJar'
    sh 'cp "./app/build/libs/expense-tracker-backend.jar" /opt/expense_tracker/expense-tracker-backend.jar'
    sh 'cp app-version /opt/expense_tracker/app-version'
    sh 'chmod 777 /opt/expense_tracker/expense-tracker-backend.jar'
    withCredentials([file(credentialsId: 'expense-tracker-backend-secrets-dev', variable: 'expenseTrackerApplicationSecrets')]) {
      sh 'sudo rm -f /opt/expense_tracker/secrets-dev.properties'
      sh 'cp $expenseTrackerApplicationSecrets /opt/expense_tracker/secrets-dev.properties'
      sh 'chmod 700 /opt/expense_tracker/secrets-dev.properties'
      sh 'sudo chown root:root /opt/expense_tracker/secrets-dev.properties'
    }
    sh 'sudo systemctl enable expense_tracker_backend'
    sh 'sudo systemctl daemon-reload'
    sh 'sudo systemctl restart expense_tracker_backend'
    sh 'sudo systemctl status expense_tracker_backend'
  }
}

The only really interesting parts in this block are here:

sh 'sudo systemctl enable expense_tracker_backend'
sh 'sudo systemctl daemon-reload'
sh 'sudo systemctl restart expense_tracker_backend'
sh 'sudo systemctl status expense_tracker_backend'

Usually, sudo can’t be run without a password prompt, and this is a problem for an automated script. However, I can get it to work for specific commands using the sudoers file. The directive needs to be the last in the file.

jenkins ALL=(ALL) NOPASSWD: /usr/bin/systemctl, /usr/bin/chown, /usr/bin/rm

I made sure to only grant access to the commands I needed and nothing more.

I had cases where the service would restart but still crash due to a mistake I made, like some improper configuration. The status command is there to fail and make sure I know something went wrong.

Post-Script Execution

post {
  always {
    junit 'app/build/test-results/**/*.xml'
  }
}

Recording tests and artifacts

Jenkins – an open source automation server which enables developers around the world to reliably build, test, and deploy their software

jenkins.io

Here I export the results of the test run from the agent so they can be viewed from within Jenkins.

And here’s the full Jenkinsfile:

pipeline {
    environment {
        MYSQL_CREDS = credentials('mysql_credentials')
    }
    agent {
        node {
            label 'Test Server'
        }
    }
    // Branch dev
    stages {

        stage('Flyway') {

            steps {
                sh 'chmod +x ./gradlew'
                sh './gradlew setUpCIDb'
                dir ('database/flyway') {
                    sh 'flyway migrate -environment=ci  -user=$MYSQL_CREDS_USR -password=$MYSQL_CREDS_PSW'
                }
            }
        }
        stage('JooQ') {
            steps {
                sh './gradlew jooqCodeGenCi'
            }
        }
        stage('Open API') {
            steps {
                sh './gradlew openApiGenerate'
            }
        }
        stage('Build') {
            steps {
                sh './gradlew assemble'
            }
        }
        stage('Test') {
            environment {
                SPRING_PROFILES_ACTIVE='ci'
            }
            steps {
                sh 'rm -f app/src/main/resources/secrets-ci.properties'
                sh 'chmod o+w -R app'
            withCredentials([file(credentialsId: 'expense-tracker-backend-secrets-ci', variable: 'expenseTrackerApplicationSecrets')]) {
                sh 'cp $expenseTrackerApplicationSecrets app/src/main/resources/secrets-ci.properties'
            }
                sh './gradlew test'
            }
        }
        stage('Flyway Dev') {
            steps {
                dir ('database/flyway') {
                    sh 'flyway migrate -environment=dev  -user=$MYSQL_CREDS_USR -password=$MYSQL_CREDS_PSW'
                }
            }
        }
        stage('Ansible setup') {
            agent {
                node {
                    label 'built-in'
                }
            }
            environment {
                ANSIBLE_BECOME_PASS = credentials('ansible-become-password')
            }
            steps {
                dir ('ansible') {
                    sh 'ansible-playbook -i inventory playbooks/setup_service.yaml  --private-key ~/.ssh/jenkins_agent_key -e "ansible_become_password=\'$ANSIBLE_BECOME_PASS\'"'
                }
            }
        }
        stage('Deploy') {
            steps {
                sh './gradlew bootJar'
                sh 'cp "./app/build/libs/expense-tracker-backend.jar" /opt/expense_tracker/expense-tracker-backend.jar'
                sh 'cp app-version /opt/expense_tracker/app-version'
                sh 'chmod 777 /opt/expense_tracker/expense-tracker-backend.jar'
                withCredentials([file(credentialsId: 'expense-tracker-backend-secrets-dev', variable: 'expenseTrackerApplicationSecrets')]) {
                    sh 'sudo rm -f /opt/expense_tracker/secrets-dev.properties'
                    sh 'cp $expenseTrackerApplicationSecrets /opt/expense_tracker/secrets-dev.properties'
                    sh 'chmod 700 /opt/expense_tracker/secrets-dev.properties'
                    sh 'sudo chown root:root /opt/expense_tracker/secrets-dev.properties'
                }
                sh 'sudo systemctl enable expense_tracker_backend'
                sh 'sudo systemctl daemon-reload'
                sh 'sudo systemctl restart expense_tracker_backend'
                sh 'sudo systemctl status expense_tracker_backend'
            }
        }
    }
    post {
            always {
                junit 'app/build/test-results/**/*.xml'
            }
    }
}

And a sample success run

Conclusion

There are several areas for improvement:

My app version is manually updated right now, and my own forgetfulness means that I’d rather automate incrementing it.
All of this was done using the Git daemon. I’d rather set up Gitea so I can use webhooks instead of manually clicking build whenever I commit to dev
I could also use an artifact management tool like Artifactory or Sona Nexus to keep the past few JAR files in case I need them

But I’m also enjoying some worthwhile benefits:

The build verification phase happens automatically
I don’t have to ssh and scp the JAR file manually

Now that the application backend is up and running, next I’ll be talking about how I use OpenTelemetry to ensure I have full visibility into both ends of the app at runtime, starting with Prometheus.

Thank you for your time

How I finally wrapped my head around unit testing

David — Sat, 28 Feb 2026 19:37:50 +0000

(Originally posted on Medium)
This is the second article in my Expense Tracker series. In my introductory article, I covered the overall architecture and the topics this series will cover. Here, I discuss how I automated my tests.

It all Started with Postman

Expense Tracker isn’t actually my first project where I started learning about testing. I have another learning project written in Spring Boot, aptly named “Shopping App”. The pattern I started with was something like this:

Write code for the controller
Run the project with gradlew bootRun
Open Postman
Confirm the endpoint is working

Early on, this cycle already felt unnecessarily repetitive, and I knew there had to be a better way. I started by looking into Postman scripting, but then I thought, “Isn’t there a way I could do this natively in Java?”, and that’s when I realized what Unit Testing was about.

Writing a Basic Authentication Test

In this instance, I'm checking to see if sign up works at the basic level. What happens in the case of a username being taken or the password being insecure isn't my concern just yet, but here's the same flow translated to testing code using the Spring testing framework:

@Test
fun successfulSignUpFlow() {
    val signUpRequest = SignUpRequest("username", "password", Locale.US.toLanguageTag());
    val objectWriter = ObjectMapper().writer();
    val signUpJson = objectWriter.writeValueAsString(signUpRequest);
    val signUpResult = mockMvc
        .perform(
            MockMvcRequestBuilders.post("/auth/signUp")
                .contentType(MediaType.APPLICATION_JSON)
                .content(signUpJson)
        )
        .andExpect(MockMvcResultMatchers.status().isOk())
        .andReturn();
    assertTrue(signUpResult.getResponse().getContentAsString().startsWith("eyJ")) //pretty much constant for every JSON Base64
}

And the test result looks like this in IntelliJ:

Testing my code like this meant a few key things for me:

I no longer had to use Postman to verify the correctness of my new code
I now have an automated way to verify any previous tests I would have done manually
I can specify all the relevant outcomes, such as HTTP 400 for an empty text field, or HTTP 404 for entities that don’t belong to the user, and be sure that they occur

I’ve read from a variety of sources to help improve my understanding of what testing should be about, and a key one has been Martin Fowler, especially his post “Given When Then”, which is about Behaviour Driven Development, or BDD, a term coined by Dan North. I’ll use an arithmetic example as I go along.

Before, when I thought about testing, all I could think up was trivial things like “Verify 1+1 = 2”, and it felt so trivial and pointless, and I wondered why people would be able to create tests like these. I did look at other examples, but I couldn’t understand what was going on back then. Now, with my current understanding, I see that it’s mainly about capturing expected behaviours as well as edge cases.

Let’s assume that this simple addition function is in Java, and we’re dealing with integers. Using int already eliminates the need for a few edge cases, but some other questions pop up:

Are a and b both positive?
What if a+b exceeds the 32-bit or 64-bit limit?

A few other questions might pop up if varied input is allowed:

What about scientific notation?
What about different bases like hexadecimal or binary?

This example is rather trivial, and the actual constraints for an app like this may be very different, but it does help me to illustrate some key things I’ve learned.

Tests serve as a form of documentation of the multiple possible ways to use a system. Based on the example above, some behaviors would look like this:

Given a or b negative when add then return invalid input
Given sum overflows when add return maximum positive number
Given sum overflows when add return invalid input
Given a and b in scientific when add then output is scientific
Given a and b in when add hex then return hex
Given a and b in different formats when add then return invalid input
Given a and b in different formats when add then output in format of a
Given a and b in different bases when add then output in higher base

Some of these are contradictory alternative responses to the same scenario, but it illustrates the point that there’s now no ambiguity in how I want things to work out.
I'm aware that there's another very popular format for test cases called "Arrange, Act, Assert", but I've chosen to stick with "Given When Then" since it also works.
Next, I’ll cover the various layers of my 3 code bases and how I applied testing to each of them.

The Database Layer

Known as the “Data Access Object” or “DAO” layer in the Android app and the “Repository” layer in the Spring project.
The tests I wrote here were mainly just to verify if my queries ran correctly whenever I went beyond simple column SELECTs such as with COUNT DISTINCT(), subqueries, or my personal favorite: WINDOW functions

Cumulative Sum

fun getTransactionCumSumByDate(
    profileId: Long, debitOrCredit: Boolean
): List<DateAmountSummary> {

    val window = DSL.orderBy(TRANSACTION.DATED_AT).rangeBetweenUnboundedPreceding().andCurrentRow()
    var selectStep = dslContext.select(
        TRANSACTION.DATED_AT.`as`("aggregateDate"),
        DSL.sum(TRANSACTION.AMOUNT).over(window)
            .`as`("sum"))
        .from(TRANSACTION)
        // Rest of selectStep

    val alias = selectStep.asTable("x")
    val selectFrom = dslContext.select(
        alias.field("aggregateDate"),
        DSL.max(alias.field("sum"))).from(alias)
    val seekStep =  selectFrom
        .groupBy(alias.field("aggregateDate"))
        .orderBy(alias.field("aggregateDate"))


    val result = seekStep.fetch().into(DateAmountSummary::class.java)

    return result
}

I was combining a window function with an aggregation via a subquery, and this was not very straightforward to me, so I wrote a test to help confirm that the number of rows and the final output was as expected:

@Test
fun cumSumByDateTest() {

    val folder = //Temp folder for images

    val firstTransactionDate = LocalDate.parse("2025-01-01")
    val secondTransactionDate = LocalDate.parse("2025-01-02")
    val thirdTransactionDate = LocalDate.parse("2025-01-04")
    val fourthTransactionDate = LocalDate.parse("2025-01-04")
    val fifthTransactionDate = LocalDate.parse("2025-01-05")

    val accountId = getDefaultAccountId(userRepository, profileRepository, accountRepository)
    val user = userRepository.findByUuid(TestConstants.DEFAULT_USER_ID)!!
    val profile = profileRepository.findByStringId(user.id!!, Constants.DEFAULT_PROFILE_ID)

    val dataBuilder = DataBuilder(folder, timeHandler, null, Constants.DEFAULT_PROFILE_ID, testUtilComponent)
    dataBuilder.createTransaction()
        .withItem("Description", "miscellaneous", BigDecimal(110.00))
        .atDate(firstTransactionDate)
        .commit()
    dataBuilder.createTransaction()
        .withItem("Description", "miscellaneous", BigDecimal(300.00))
        .atDate(secondTransactionDate)
        .commit()
    dataBuilder.createTransaction()
        .withItem("Description", "miscellaneous", BigDecimal(450.00))
        .atDate(thirdTransactionDate)
        .commit()
    dataBuilder.createTransaction()
        .withItem("Description", "miscellaneous", BigDecimal(750.00))
        .atDate(fourthTransactionDate)
        .commit()
    dataBuilder.createTransaction()
        .withItem("Description", "miscellaneous", BigDecimal(1_000.00))
        .atDate(fifthTransactionDate)
        .commit()

    val sums = transactionRepository.getTransactionCumSumByDate(profile!!.id!!, true, firstTransactionDate, null, emptyList(), emptyList(), emptyList() , emptyList(),  emptyList())
    val firstSum = sums.find { it.aggregateDate.compareTo(firstTransactionDate) == 0 }!!
    val secondSum = sums.find { it.aggregateDate.compareTo(secondTransactionDate) == 0 }!!
    val thirdSum = sums.find { it.aggregateDate.compareTo(thirdTransactionDate) == 0 }!!
    assertEquals(4, sums.size)
    assertEqualsBD(BigDecimal(110.00), firstSum.sum)
    assertEqualsBD(BigDecimal(410.00), secondSum.sum)
    assertEqualsBD(BigDecimal(1610.00), thirdSum.sum)
}

The Business Logic Layer

I chose to name them as the “Repository” layer in the Android app and the “Service” layer in the Spring project (confusing, I know)

Tests here were mainly to confirm if any transformations I applied to the raw data were valid, for example, when I get the transaction cumulative sum by date in this chart:

Since not every day within the date interval is guaranteed to have an entry, I'd like to interpolate those sub-intervals with the most recent amount so that the chart has a more natural-looking spread:

@Test
fun intervalCumSumTest() {
    val folder = //Temp folder for images
    val dataBuilder = DataBuilder(folder, timeHandler, null, Constants.DEFAULT_PROFILE_ID, testUtilComponent)

    val user = userRepository.findByUuid(TestConstants.DEFAULT_USER_ID)!!
    val profile = profileRepository.findByStringId(user.id!!, Constants.DEFAULT_PROFILE_ID)!!
    val profileId = profile.id!!
    val firstTransactionDate = LocalDate.parse("2025-01-02")
    val secondTransactionDate = LocalDate.parse("2025-01-02")
    val thirdTransactionDate = LocalDate.parse("2025-01-07")
    val fourthTransactionDate = LocalDate.parse("2025-01-09")
    val fifthTransactionDate = LocalDate.parse("2025-07-11")



    dataBuilder.createTransaction()
        .withItem("Description", "miscellaneous", BigDecimal(110.00))
        .atDate(firstTransactionDate)
        .commit()
    dataBuilder.createTransaction()
        .withItem("Description", "miscellaneous", BigDecimal(300.00))
        .atDate(secondTransactionDate)
        .commit()


    dataBuilder.createTransaction()
        .withItem("Description", "miscellaneous", BigDecimal(450.00))
        .atDate(thirdTransactionDate)
        .commit()
    dataBuilder.createTransaction()
        .withItem("Description", "miscellaneous", BigDecimal(750.00))
        .atDate(fourthTransactionDate)
        .commit()
    dataBuilder.createTransaction()
        .withItem("Description", "miscellaneous", BigDecimal(1000.00))
        .atDate(fifthTransactionDate)
        .commit()

    // Here, transactionService is responsible for the interpolation
    val transactionSumByDates = transactionService.getCumSumByDate(profileId, true, LocalDate.parse("2025-01-02"), null, emptyList(), emptyList(), emptyList(), emptyList(), emptyList())
    val sum1 = transactionSumByDates.find { it.aggregateDate == firstTransactionDate }!!
    var missingSum1 = transactionSumByDates.find { it.aggregateDate == LocalDate.parse("2025-01-03") }!!
    val missingSum2 = transactionSumByDates.find { it.aggregateDate == LocalDate.parse("2025-01-08") }!!
    val missingSum3 = transactionSumByDates.find { it.aggregateDate == LocalDate.parse("2025-01-10") }!!
    assertEqualsBD(BigDecimal(410), sum1.sum)
    assertEqualsBD(BigDecimal(410), missingSum1.sum)
    assertEqualsBD(BigDecimal(860), missingSum2.sum)
    assertEqualsBD(BigDecimal(1610), missingSum3.sum)

}

Accompanying charts:

The UI Layer

(Unique to Android)
My tests here were a mix of input validation, such as password constraints, and proper conditional behavior, such as the correct dialog popping up for invalid PDFs, but the test I want to highlight is about character count.

I went on a bit of a tangent (including but not limited to the Tonsky.me blog, X’s developer docs, StackOverflow), and decided that "Code Point" will be how I define what a "Character" is, and use it to determine my character limit, so my test here is to confirm that I’m counting and truncating graphemes correctly using ICU4J:

@Test
fun whenUserPastesTextAndTotalLengthOverCharacterLimitThenTextTruncatedByGrapheme() {
    val context = ApplicationProvider.getApplicationContext<ExpenseTracker>()
    //Assume max length is 500
    // `text` is one code point short of that limit
    val text = String(CharArray(Constants.MAX_NOTE_CODEPOINT_LENGTH - 1) {
        'a'
    })
    val manRunning = "\uD83C\uDFC3\u200D\u2642\uFE0F" //4 codepoints - 1 surrogate pair + 3 regular
    val manRunningCount = 4
    // Copy to clipboard
    addDetailedTransactionActivityScenario.onActivity {
        it.runOnUiThread {
            val clipBoardManager = context.getSystemService(Context.CLIPBOARD_SERVICE) as ClipboardManager
            var clipData = ClipData.newPlainText("simple text", text + manRunning)
            clipBoardManager.setPrimaryClip(clipData)
        }
    }
    // Ctrl+V
    onView(withId(R.id.edit_text_add_detailed_transaction_note)).perform(
        click(), ViewActions.pressKey(
        EspressoKey.Builder().withCtrlPressed(true).withKeyCode(KeyEvent.KEYCODE_V).build()))
    onView(withId(R.id.edit_text_add_detailed_transaction_note)).check { view, noViewFoundException ->
        val v = view as EditText
        val text = v.text.toString()
        val count = text.codePointCount(0, text.length)
        // A simpler text limiter would have kept the first part of the surrogate pair, D83C, which has no sensible meaning
        assertEquals(Constants.MAX_NOTE_CODEPOINT_LENGTH - 1, count)
    }
    onView(withId(R.id.text_view_add_detailed_transaction_note_length_indicator)).check(matches(withText("${Constants.MAX_NOTE_CODEPOINT_LENGTH - 1}/${Constants.MAX_NOTE_CODEPOINT_LENGTH}")))

    //Similar test, but confirming with just emojis
    val menRunning = manRunning.repeat(Math.floorDiv(Constants.MAX_NOTE_CODEPOINT_LENGTH, manRunningCount)+1) //4 x 126 = 504
    addDetailedTransactionActivityScenario.onActivity {
        it.runOnUiThread {
            val clipBoardManager =
                context.getSystemService(Context.CLIPBOARD_SERVICE) as ClipboardManager
            var clipData = ClipData.newPlainText("simple text", menRunning)
            clipBoardManager.setPrimaryClip(clipData)
        }
    }
    onView(withId(R.id.edit_text_add_detailed_transaction_note)).perform(
        clearText(), ViewActions.pressKey(
            EspressoKey.Builder().withCtrlPressed(true).withKeyCode(KeyEvent.KEYCODE_V).build()))
    onView(withId(R.id.edit_text_add_detailed_transaction_note)).check { view, noViewFoundException ->
        val v = view as EditText
        val text = v.text.toString()
        val count = text.codePointCount(0, text.length)
        assertEquals(Constants.MAX_NOTE_CODEPOINT_LENGTH, count)
    }
    onView(withId(R.id.text_view_add_detailed_transaction_note_length_indicator)).check(matches(withText("${Constants.MAX_NOTE_CODEPOINT_LENGTH}/${Constants.MAX_NOTE_CODEPOINT_LENGTH}")))
    onView(withId(R.id.edit_text_add_detailed_transaction_note)).perform(clearText())
}

And here’s the test being run by Espresso at 0.5 speed:

The Controller Layer

(Unique to Spring)

Very similar to the UI layer, it’s also mostly input validation, but instead of dialogs and error indicators, I check to see if I’m returning the correct status and custom error code:

@Test
fun givenCategoryIdExistsAndCategoryIdBelongsToAnotherUserWhenCreateTransactionThenNotFound() {

    val user = userRepository.findByUuid(TestConstants.DEFAULT_USER_ID)!!
    val profile = profileRepository.findByStringId(user.id!!, Constants.DEFAULT_PROFILE_ID)!!

    val userId = userService.createUser("user2", "password", Locale.US)
    val profile2 = profileRepository.findByStringId(userId, Constants.DEFAULT_PROFILE_ID)!!
    val accounts = accountRepository.getAllByProfileId(profile.id!!)
    val misc = categoryRepository.findByProfileIdAndStringId(profile2.id!!, "miscellaneous")
    var basicTransactionRequest = BasicTransactionCreateRequest(accounts[0].id!!, true, BigDecimal.TEN, "Description", misc!!.id, "Etc/UTC")

    val jws = createBearerToken("username", jwtSigningKey!!, JWT_ALG, timeHandler)
    val objectWriter: ObjectWriter = ObjectMapper().writer()
    val json: String = objectWriter.writeValueAsString(basicTransactionRequest)
    mockMvc
        .perform(
            MockMvcRequestBuilders.post("/transactions/basic")
                .header("Authorization", "Bearer ${jws}")
                .header("Profile-Id", profile.stringId)
                .contentType(MediaType.APPLICATION_JSON)
                .content(json)
        )
        .andExpectAll(
            //Error code should be 404
            MockMvcResultMatchers.status().isNotFound,
            MockMvcResultMatchers.jsonPath(
                "$.$CUSTOM_PROBLEM_DETAILS_ATTRIBUTE_ERROR_CODE", 
                //Make sure correct descriptive error code is returned
                Matchers.equalTo(ErrorCode.ENTITY_NOT_FOUND.code)
            )
        )
}

Now that I’ve covered the various layers, I’ll talk about 3 more issues that came up while I was learning:

Decision Table

I usually come up with my tests in my head, basically. I start with one possibility, like:

Given Number field is empty…

Then I’d expand to another one like:

Given Number field is non-positive…

And it’s worked fine for most of my tests, but for the edit feature of Expense Tracker, this unfortunately wasn’t enough.

For a brief overview, in the edit feature:

The dbId property determines if an item is “new” or “existing”
Similarly, images also have a dbId property as well as dbIsLinked to indicate whether there was already an entry in the database that linked the image to the item

data class AddTransactionItem(
    val id: Int,
    val dbId: Long?,
    val category: CategoryUi,
    val amount: BigDecimal? = null,
    val description: String? = null,,
    val images: List<AddEditTransactionFile> = emptyList(),
    val deletedDbImages: List<AddEditTransactionFile> = emptyList()
)

data class AddEditTransactionFile(
    val dbId: Long?,
    val uri: Uri,
    val mimeType: String,
    val sha256: String,
    val sizeBytes: Long,
    val dbIsLinked: Boolean = false
)

In this example screen, the first item is from the database, and the orange has a non-null dbId and dbIsLinked is set to true, while the second item is new and so has a null dbId .

My main issue was knowing what to do when saving the changes of that edit. I needed to know how to deal with images that satisfied my main constraints:

If the same image is added to more than one item, then simply link them instead of adding duplicates
If the image has been removed from an item, and that was the last item it was linked to, remove it from the user’s device

Coming up with test cases for this felt complicated, simply put. I looked at Karnaugh maps and Finite State machines because I felt they would help out, but in the end, I settled for something that loosely resembles a truth table, which I later found to be called a Decision table.
I itemized my variables like this:

Is the item new or in the database?
Has the item been marked as deleted?
Is the image new or in the database?
Has the image been marked deleted?
Does the image have a database link to the item?

With a clear idea of what they were, I could now go through all the possibilities. Since there are 5 variables in total, there are 2⁵=32 possibilities*
(* It’s possible for the variable to have more than 2 states, but I haven’t encountered a case like that yet)
For the sake of brevity, I’ll show the first 8 rows.

Rows 3 and 5 are marked “Impossible” because of the condition: not( ImageInDb ) and ImageLinked
Similarly, 7 and 9 are marked that way because of not( ItemInDb ) and ImageLinked
Rows 4 and 8 don’t need any action because I don’t care about images that won’t be linked to an item
Row 1 means that I have to persist the image to the database first before I create a link to the item, and row 6 means I skip the first part

The edit feature so far has been the only time where I’ve really needed the table, but I’m glad I got to use it.

Overall, this method gave me some notable benefits:

A way to mark impossible states and relevant states. In my case, I only needed to use 8 of the 32 rows, but I got to know which ones were which
Those 8 rows can each become a unit test

Seeding Data

As the number of my tests grew, I found myself repeatedly having to put custom entries in the database through the repository methods. This ended up taking up a good portion of my test code in some cases.

I wanted something that would simplify this process so that I could focus on testing outcomes instead of creating data. The idea mainly came from the OpenAPI generator code base, where I did a bit of work. They use various utilities such as CodegenConfigurator to make it so that configuration doesn’t take as long. After some searching, one of my main helps was this article from Lukas Eder of jOOQ:

The Java Fluent API Designer Crash Course – Java, SQL and jOOQ.

Ever since Martin Fowler’s talks about fluent interfaces, people have started chaining methods all over the place, creating fluent API’s (or DSLs) for every possible use case. In princi…

blog.jooq.org

And I ended up coming up with my own basic DSL to help me out:

interface Commit {
    /**
     * Returns the list of transaction IDs
     */
    fun commit(): List<Long>
}
interface TransactionBuilder: Commit {
    fun withItem(description: String): TransactionBuilder
    fun withItem(description: String, category: String, amount: BigDecimal): TransactionBuilder
    fun withDefaultItemPrice(defaultPrice: BigDecimal): TransactionBuilder
    fun withNewAccount(currencyCode: String, accountName: String): TransactionBuilder
    fun debitOrCredit(debitOrCredit: Boolean): TransactionBuilder
    fun withDetailedItem(): TransactionItemBuilder
    fun repeatIntoDates(dates: List<LocalDate>): TransactionBuilder

    /**
     * Inclusive, Inclusive
     */
    fun repeatIntoDateRange(startDate: LocalDate, endDate: LocalDate): TransactionBuilder
    fun atDate(date: LocalDate): TransactionBuilder
}

interface TransactionItemBuilder: Commit {
    fun mainDetails(description: String, category: String, amount: BigDecimal): TransactionItemBuilder
    fun otherDetails(brand: String?, quantity: Int, variation: String, referenceNumber: String?): TransactionItemBuilder
    fun withImages(vararg resource: TestData.Resource): TransactionItemBuilder
    fun build(): TransactionBuilder
}

Relevant commit link

And in a test to verify that I’m fetching the statistics for accounts correctly, here’s an example where I make data seeding slightly more readable:

Setup and Teardown

For setup, I have utility methods to insert basic data for a default profile.

For teardown, I chose to use an approach where I essentially delete the whole database when a test is finished. In Android, it’s simple with Room’s clearAllTables , but in the backend, there’s no direct utility method I found, so instead I did this:

Generated documentation with SchemaSpy
Looked at the deletionOrder.txt file
Wrote DELETE methods that corresponded to each table and ran them in the same order

Testcontainers are probably a better way to handle this, and Spring’s Transactional annotation as well, but I’m sticking with the current method since it’s not too much trouble

Conclusion

Overall, I’ve gained some key things:

A framework for identifying edge cases — even the rare ones — ensuring they are handled by design rather than by accident.
Increased confidence that when I make any changes to a complicated feature like transaction editing, I likely haven’t broken existing behavior
A shorter feedback loop

There are also other kinds of tests, like end-to-end, acceptance, and integration tests, but I think those topics are different enough to not have to discuss here, especially since the meaning of some can change significantly per environment. I do talk about performance testing, though, but that’s for later.

Now that I’ve learned how to automate my tests, next I’ll talk about how I’ve learned to automate my deployments using Continuous Integration with Jenkins.

Please share your experiences with how testing and test culture is (or is not) implemented in your own code bases in the comments.

Thank you for your time.

Building a Personal Expense Tracker with OpenTelemetry and CI/CD

David — Wed, 25 Feb 2026 23:11:52 +0000

(Originally posted on Medium)
I decided to make a personal app to track my expenses. Spreadsheets served me well, but there are some core features that I couldn’t get out of that
method:

The ability to attach receipts to my records
The ability to add images of the items I purchase
Viewing and filtering the statistics across any time period without needing new PivotTables

There are excellent apps that already exist for this, including:

Though none of them fit exactly what I wanted. I could have kept searching, but beyond the need for a custom solution, I also saw the opportunity to challenge myself to learn certain topics, including:

Unit Testing: with the Android and Spring frameworks
Continuous Integration and Continuous Delivery: with Jenkins
Observability: with OpenTelemetry and the Grafana stack
Performance Testing: with JMeter
Infrastructure: including Ansible, NGINX, and Linux concepts

And so I decided to go ahead and build the app for myself

App Structure

There are 2 versions of the app:

Expense Tracker: offline, the first version of the app, and essentially my template for the next version. The source is available here.
ExTrack: the full client-server version, split into the Java backend and the native Android version

Entities

Transaction: made up of one or more items, bound to an account, and has a currency code

Transaction items have a description, an amount, a category for classification, and images

Other entities include Category, Account, Document, and Image.
The basic use cases include adding transactions and viewing statistics:

Statistics screen:

This screen is loosely based on Simple Time Tracker's stats screen.

Architecture Overview

Most of this was done on my local network.

Tech Stack

Backend: Java 17, Spring Boot, jOOQ, OpenAPI
Observability: Grafana Loki 3.2, Grafana 11, Grafana Tempo 2.6, Prometheus 3.0, Grafana Pyroscope 1.18
Infrastructure: Jenkins, NGINX 1.28, MySQL 8.0, various Prometheus exporters
Operating Systems: Ubuntu 22, Windows 11, Windows 10

How it fits together

Prometheus reads metrics from a wide variety of exporters that I use. Promtail scrapes log files and sends them to Loki. Tempo is for distributed tracing. Using OpenTelemetry, all 3 signals are sent through NGINX to Alloy, and I visualize everything in Grafana. Pyroscope is a special case that I'll discuss in a separate article.

Conclusion

The next few articles will go into detail on how I learned each topic with the app, with the first 2 covering how I automated my testing and deployment processes.

Let me know your thoughts about this new series in the comments. I'll be cross-posting the remaining articles from Medium gradually.

Thank you for your time