DEV Community: Kevin Kimani

Next-Level Observability with OpenTelemetry

Kevin Kimani — Fri, 10 Apr 2026 15:35:40 +0000

When something goes wrong in my applications, logging is almost always the first tool I reach for. I'll throw a few log statements at the start and end of a function, sprinkle some into the exception handlers, and I'll have a basic picture of what's happening. For simple services running on a single instance, that's usually enough; I can scroll through the log file, spot the error, and trace it back to its cause.

But as my systems have grown, that same approach doesn't work. Logs pile up from multiple sources, executions start interleaving, and the error I'm staring at no longer tells me enough. I can see that something failed, but I can't trace why.

In this tutorial, I'll walk you through how I moved beyond basic logging by instrumenting a Kotlin and Spring Boot backend service with OpenTelemetry. You'll learn how OpenTelemetry's tracing model gives you the execution context that logs alone can't provide. By the end of this guide, you'll have a working instrumented service and a clear mental model for building more observable backend systems.

Why I Needed Next-Level Observability

Modern backend systems are rarely linear. One operation might fan out to several downstream services, retry on failure, or execute concurrently across multiple instances and threads. All of these patterns create failure modes that are genuinely hard to explain after the fact.

I ran into this personally when a background job processing hundreds of records across overlapping executions started throwing errors. My logs showed the errors, but they couldn't tell me which execution each error belonged to, or whether the other concurrent executions succeeded or failed in the same way.

The gap between seeing an error and understanding what led to it is the problem. I'd have the error message and a timestamp, but no way to connect either to the broader execution context. In a system running multiple concurrent job executions, log lines from different runs freely interleave, and thread names get recycled by the thread pool. Without a way to uniquely identify each execution, every line in my log file was an isolated fact with no reliable way to group it with others from the same run.

Observability gave me a structured view of what my system did and in what order. It does this through traces, records of complete operations that carry unique identifiers. I can filter my logs by those trace identifiers and see the entire history of a specific execution, clearly. Metrics add another dimension by revealing patterns over time that no single log entry can show. Together, they transformed my debugging from guesswork into a structured investigation.

What OpenTelemetry Provides

OpenTelemetry is an open-source observability framework that defines a unified model for collecting three types of signals from your applications:

Traces: represent the full lifecycle of a request or an operation as it moves through your system. A trace is made up of spans, where each span represents a unit of work within the operation, such as an HTTP call or a background task. Each span contains a trace ID and a span ID as part of its context, where the trace ID ties back the span to its parent operation (trace) and the span ID uniquely identifies the single specific step within the operation.
Metrics: capture aggregated measurements over time, such as how long an operation takes and the error rates. This helps to give you a statistical view of the overall system health.
Logs: represent discrete events that happened at a specific point in time. When correlated with trace context, logs stop being isolated entries in a file and become anchored events within a specific execution, which makes it easy to understand exactly what happened and why.

The OpenTelemetry ecosystem has three main components: instrumentation, the Collector, and exporters. Instrumentation is how you integrate OpenTelemetry into your application, using language-specific SDKs to create spans, record metrics, and propagate context. The Collector is an optional middleware component that receives, processes, and exports telemetry data to one or more backends. Exporters are the plugins that actually send your data to a specific destination, like Prometheus or Jaeger.

What made me commit to OpenTelemetry as a long-term strategy is its vendor neutrality. Before it existed, instrumentation was tightly coupled to specific vendors, and switching meant rewriting instrumentation code throughout your entire codebase. OpenTelemetry fixed that by separating instrumentation from destination; I can instrument my service once using the standard API, and then swap exporters as my infrastructure evolves.

Setting Up Next-Level Observability with OpenTelemetry

If you want to follow along, you'll need the following:

An IDE, such as IntelliJ IDEA.
JDK 17 or later and Git CLI installed on your local machine.

Here is a rough architecture diagram of what we'll build:

The application consists of a Spring Boot Service running in a single Java Virtual Machine (JVM). The Task Scheduler triggers the OrderSummaryJob at regular intervals. The job reads orders from an embedded H2 database via Spring Data JPA, processes them, and writes summaries back to the database. The OpenTelemetry Java Agent sits within the JVM, automatically instrumenting the job and injecting trace context into the Mapped Diagnostic Context (MDC). This context flows through the log output, allowing you to correlate all logs from a single execution when multiple executions run concurrently.

Setting Up the Starter Template

To keep things focused on observability, I've prepared a Kotlin and Spring Boot application with a scheduled job already in place:

git clone --single-branch -b starter-template https://github.com/kimanikevin254/jetbrains-otel-order-summary.git

Open the project in your code editor.

The most important file in this project is the src/main/kotlin/com/example/order_summary/service/OrderSummaryJob.kt, which defines the scheduled job. The job reads orders created in the last 24 hours from an H2 database via Spring Data repositories, processes them one by one, and writes summaries back to the DB. The job runs every five minutes using Spring's @Scheduled annotation. The summaries generated by this job can later be consumed by other parts of a larger system, such as dashboards, analytic pipelines, or downstream services that need a periodic snapshot of order volume and revenue.

The logging approach is straightforward and very common. You log when the job starts, log each order being processed, log if an exception occurs, and log when the job finishes:

@Service
class OrderSummaryJob(
   private val orderRepository: OrderRepository,
   private val orderSummaryRepository: OrderSummaryRepository
) {
   private val logger = LoggerFactory.getLogger(OrderSummaryJob::class.java)

   @Scheduled(fixedDelay = 300000) // 5mins in ms
   fun generateSummary() {
       logger.info("Starting order summary job...")

       val periodEnd = LocalDateTime.now()
       val periodStart = periodEnd.minusHours(24)

       val orders = orderRepository.findByCreatedAtAfter(periodStart)
       if (orders.isEmpty()) {
           logger.info("No orders found in the last 24 hours. Skipping summary generation.")
           return
       }
       logger.info("Found ${orders.size} orders to process")

       var processedCount = 0
       var totalAmount = BigDecimal.ZERO

       for (order in orders) {
           try {
               logger.info("Processing order ${order.id} for customer ${order.customerId}...")

               // Simulate processing work
               Thread.sleep(2000)

               // Simulate occasional failures
               if (order.amount > BigDecimal("400")) {
                   throw RuntimeException("Order amount exceeds threshold: ${order.amount}")
               }

               totalAmount = totalAmount.add(order.amount)
               processedCount++
           } catch (e: Exception) {
               logger.error("Failed to process order ${order.id}: ${e.message}")
               // Continue processing other orders
           }
       }

       val summary = OrderSummary(
           totalOrders = orders.size,
           totalAmount = totalAmount,
           periodStart = periodStart,
           periodEnd = periodEnd
       )

       orderSummaryRepository.save(summary)
       logger.info("Order summary job completed. Total: ${orders.size} orders, Amount $totalAmount")
   }
}

To see it in action, run the following command in your terminal:

./gradlew bootRun

Once the application starts, you can see the logs. Execute the command tail -f logs/order-summary.log in a separate terminal to stream the logs from the configured log file:

2026-02-24T15:49:47.304+03:00  INFO 605417 --- [order-summary] [scheduling-1] c.e.o.service.OrderSummaryJob            : Found 12 orders to process
2026-02-24T15:49:47.305+03:00  INFO 605417 --- [order-summary] [scheduling-1] c.e.o.service.OrderSummaryJob            : Processing order 1 for customer CUST-10001...
2026-02-24T15:49:49.306+03:00  INFO 605417 --- [order-summary] [scheduling-1] c.e.o.service.OrderSummaryJob            : Processing order 3 for customer CUST-10003...
2026-02-24T15:49:51.307+03:00  INFO 605417 --- [order-summary] [scheduling-1] c.e.o.service.OrderSummaryJob            : Processing order 7 for customer CUST-10007...
2026-02-24T15:49:53.308+03:00  INFO 605417 --- [order-summary] [scheduling-1] c.e.o.service.OrderSummaryJob            : Processing order 8 for customer CUST-10008...
2026-02-24T15:49:55.308+03:00  INFO 605417 --- [order-summary] [scheduling-1] c.e.o.service.OrderSummaryJob            : Processing order 9 for customer CUST-10009...
2026-02-24T15:49:57.310+03:00 ERROR 605417 --- [order-summary] [scheduling-1] c.e.o.service.OrderSummaryJob            : Failed to process order 9: Order amount exceeds threshold: 458.23
...

2026-02-24T15:50:11.322+03:00 ERROR 605417 --- [order-summary] [scheduling-1] c.e.o.service.OrderSummaryJob            : Failed to process order 19: Order amount exceeds threshold: 427.98
2026-02-24T15:50:11.340+03:00  INFO 605417 --- [order-summary] [scheduling-1] c.e.o.service.OrderSummaryJob            : Order summary job completed. Total: 12 orders, Amount 1680.31

The logs show clean, linear execution. Each step follows the previous, and errors are easy to associate with what was happening. This works well for infrequent, single-instance execution.

Introducing Complexity

As a project evolves, two changes might occur:

The business might demand near real-time visibility into the order metrics, which means that the job needs to run more frequently. Say, every five seconds instead of every five minutes.
The application may be deployed across multiple instances for high availability.

Let's start by running the job more frequently to see how this affects our current logging approach. To do this, open the src/main/kotlin/com/example/order_summary/OrderSummaryApplication.kt file and add the following line of code to the main application class:

@EnableAsync

This enables async execution. Remember to add the following import statement to the same file:

import org.springframework.scheduling.annotation.EnableAsync

Open the src/main/kotlin/com/example/order_summary/service/OrderSummaryJob.kt file and replace @Scheduled(fixedDelay = 300000) // 5mins with the following:

@Async
@Scheduled(fixedRate = 5000) // 5secs in ms

Changing from fixedDelay to fixedRate means the job starts every five seconds regardless of whether the previous execution has finished. Adding @Async ensures that each execution runs on its own thread from Spring's task executor pool, preventing slow jobs from blocking the scheduler. This is a common pattern when scaling background jobs to handle higher throughput.

Remember to add the following import statement to the same file:

import org.springframework.scheduling.annotation.Async

Restart the application and observe the logs. You should see something like this:

2026-02-24T16:17:40.469+03:00  INFO 610799 --- [order-summary] [task-1] c.e.o.service.OrderSummaryJob            : Starting order summary job...
2026-02-24T16:17:40.596+03:00  INFO 610799 --- [order-summary] [task-1] c.e.o.service.OrderSummaryJob            : Found 12 orders to process
2026-02-24T16:17:40.597+03:00  INFO 610799 --- [order-summary] [task-1] c.e.o.service.OrderSummaryJob            : Processing order 1 for customer CUST-10001...
...
2026-02-24T16:17:47.468+03:00  INFO 610799 --- [order-summary] [task-2] c.e.o.service.OrderSummaryJob            : Processing order 3 for customer CUST-10003...
2026-02-24T16:17:48.602+03:00  INFO 610799 --- [order-summary] [task-1] c.e.o.service.OrderSummaryJob            : Processing order 9 for customer CUST-10009...
2026-02-24T16:17:49.469+03:00  INFO 610799 --- [order-summary] [task-2] c.e.o.service.OrderSummaryJob            : Processing order 7 for customer CUST-10007...
2026-02-24T16:17:50.460+03:00  INFO 610799 --- [order-summary] [task-3] c.e.o.service.OrderSummaryJob            : Starting order summary job...
2026-02-24T16:17:50.472+03:00  INFO 610799 --- [order-summary] [task-3] c.e.o.service.OrderSummaryJob            : Found 12 orders to process
2026-02-24T16:17:50.473+03:00  INFO 610799 --- [order-summary] [task-3] c.e.o.service.OrderSummaryJob            : Processing order 1 for customer CUST-10001...
...
2026-02-24T16:17:54.476+03:00  INFO 610799 --- [order-summary] [task-3] c.e.o.service.OrderSummaryJob            : Processing order 7 for customer CUST-10007...
2026-02-24T16:17:54.608+03:00  INFO 610799 --- [order-summary] [task-1] c.e.o.service.OrderSummaryJob            : Processing order 14 for customer CUST-10014...
2026-02-24T16:17:55.460+03:00  INFO 610799 --- [order-summary] [task-4] c.e.o.service.OrderSummaryJob            : Starting order summary job...
2026-02-24T16:17:55.473+03:00  INFO 610799 --- [order-summary] [task-4] c.e.o.service.OrderSummaryJob            : Found 12 orders to process
2026-02-24T16:17:55.474+03:00  INFO 610799 --- [order-summary] [task-4] c.e.o.service.OrderSummaryJob            : Processing order 1 for customer CUST-10001...
2026-02-24T16:17:55.475+03:00 ERROR 610799 --- [order-summary] [task-2] c.e.o.service.OrderSummaryJob            : Failed to process order 9: Order amount exceeds threshold: 458.23

The logs are now completely interleaved. Executions from task-1, task-2, and task-3 are all running simultaneously, processing the orders, and logging to the same output. When an error occurs, like the failure on order 9 at 16:17:55, it's not easy to figure out which job execution the log belongs to and which orders were successfully processed before the error occurred in that specific execution.

You might think searching by thread name, such as task-1, would solve this, but Spring's thread pool reuses threads. After task-1 finishes its first execution, it picks up execution 9, then execution 17, and so on. Searching by thread name now gives you mixed logs from multiple unrelated executions. In production, where multiple application instances run behind a load balancer, thread names are no longer unique across your system.

This is where plain logging breaks down. I can see that something failed. I can't explain what happened leading up to it.

A Better Solution: Adding OpenTelemetry

To fix the missing execution context, I wanted each job execution to be treated as a logical unit of work, with a unique trace ID attached to it. All logs emitted during that job should include that ID. Even when multiple executions run concurrently and logs interleave, I can filter by trace ID and see exactly what happened in a single run.

OpenTelemetry provides several ways to instrument applications. I'll use the Java Agent here, which automatically instruments your application without requiring any changes to the source code. It's genuinely the path of least resistance.

Let's start by downloading the agent JAR file. Execute the following commands in the project root folder to create a directory for the agent and download it:

mkdir -p agents
curl -L https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar \
-o agents/opentelemetry-javaagent.jar

Verify the download using the following command:

ls -lh agents/opentelemetry-javaagent.jar

You should see a file around 24MB in size.

Add the agents/ directory to your .gitignore file so that the JAR file is not committed to version control. You can use the command echo "agents/" >> .gitignore or add it manually.

Once you've confirmed that the agent was downloaded successfully, it's time to configure it. You need to attach it to the JVM when running the application by passing it as an argument. Open the build.gradle.kts file and add the following configuration:

tasks.bootRun {
   jvmArgs = listOf(
       "-javaagent:${projectDir}/agents/opentelemetry-javaagent.jar",
       "-Dotel.service.name=order-summary-service",
       "-Dotel.traces.exporter=logging",
       "-Dotel.metrics.exporter=none",
       "-Dotel.logs.exporter=none"
   )
}

Here's what each argument does:

-javaagent tells the JVM to load the OpenTelemetry agent before your application starts
-Dotel.service.name sets the name of your service in the telemetry data
-Dotel.traces.exporter=logging prints trace data to the console. No external backend is needed for this guide
-Dotel.metrics.exporter=none and -Dotel.logs.exporter=none disable metrics and log exporting since that is outside the scope of this guide

Lastly, you need to update the log patterns to include trace context. The OpenTelemetry agent automatically injects trace_id and span_id into the logging context (Mapped Diagnostic Context). To display these values in your application logs, open the src/main/resources/application.properties file and add the following:

logging.pattern.console=%d{HH:mm:ss.SSS} [%thread] [trace_id=%mdc{trace_id} span_id=%mdc{span_id}] %-5level %logger{36} - %msg%n
logging.pattern.file=%d{HH:mm:ss.SSS} [%thread] [trace_id=%mdc{trace_id} span_id=%mdc{span_id}] %-5level %logger{36} - %msg%n

The %mdc{trace_id} and %mdc{span_id} directives extract values from the MDC that the agent populates automatically.

Now, let's restart the application and observe the logs:

17:19:43.715 [task-1] [trace_id=da673f1ec49eba77264c5912584e7183 span_id=74c708e335a974e3] INFO  c.e.o.service.OrderSummaryJob - Starting order summary job...
17:19:43.856 [task-1] [trace_id=da673f1ec49eba77264c5912584e7183 span_id=74c708e335a974e3] INFO  c.e.o.service.OrderSummaryJob - Found 12 orders to process
17:19:43.857 [task-1] [trace_id=da673f1ec49eba77264c5912584e7183 span_id=74c708e335a974e3] INFO  c.e.o.service.OrderSummaryJob - Processing order 1 for customer CUST-10001...
17:19:45.860 [task-1] [trace_id=da673f1ec49eba77264c5912584e7183 span_id=74c708e335a974e3] INFO  c.e.o.service.OrderSummaryJob - Processing order 3 for customer CUST-10003...

...

17:19:53.704 [task-3] [trace_id=4a969bbb00634e0ee36b2fbda1399d8a span_id=0a602f1a58df2f71] INFO  c.e.o.service.OrderSummaryJob - Starting order summary job...
17:19:53.715 [task-3] [trace_id=4a969bbb00634e0ee36b2fbda1399d8a span_id=0a602f1a58df2f71] INFO  c.e.o.service.OrderSummaryJob - Found 12 orders to process
17:19:53.715 [task-3] [trace_id=4a969bbb00634e0ee36b2fbda1399d8a span_id=0a602f1a58df2f71] INFO  c.e.o.service.OrderSummaryJob - Processing order 1 for customer CUST-10001...
17:19:53.868 [task-1] [trace_id=da673f1ec49eba77264c5912584e7183 span_id=74c708e335a974e3] ERROR c.e.o.service.OrderSummaryJob - Failed to process order 9: Order amount exceeds threshold: 458.23

All logs from the first execution share the same trace_id (da673f1ec49eba77264c5912584e7183), while logs from the third execution have a different trace_id (4a969bbb00634e0ee36b2fbda1399d8a). Even though both executions are running concurrently and their logs are interleaved, you can now filter by trace ID to isolate a single execution.

For example, to see logs only from the first execution, you could search for trace_id=da673f1ec49eba77264c5912584e7183 in a log aggregation tool such as Amazon CloudWatch.

All the code used in this tutorial is available on GitHub.

Where to Go From Here

Once I had trace IDs in my logs, a few next steps were obvious.

For even more powerful filtering, I could add structured fields to logs for record IDs or job phases, for example, logging order_id as a dedicated field alongside the trace ID, letting me query all executions that touched a specific order.

I could also export logs alongside traces to an observability backend like Jaeger or Grafana, which would let me visualize the full trace as a timeline showing how long each step took and where errors occurred. The OpenTelemetry agent supports multiple backends by simply changing the exporter configuration, so I can start with logging (as we did here) and migrate to a full observability platform later without touching my instrumentation code.

The same pattern works for API handlers, other background jobs, or any async work in the system. Once OpenTelemetry is in place, every part of the application automatically benefits from trace context propagation.

Conclusion

Good observability isn't just about adding more logs. It's about adding structure and context to the signals your system already emits. With OpenTelemetry, I was able to turn interleaved, confusing output into isolated, queryable execution traces.

Building with observability in mind changes how I think about system design. Traces reveal where boundaries should exist. Metrics show trends that make capacity planning and SLO definition more grounded in actual production behavior. And well-instrumented code is simply more readable. When every operation is traced, you think more carefully about what constitutes a meaningful unit of work, which benefits both the tools and the people reading the code later.

Defensive Programming: A Guide to Building Resilient API Clients

Kevin Kimani — Thu, 20 Nov 2025 16:23:34 +0000

Imagine this: it's payday, and my system calls a payroll application programming interface (API) to process salaries. The request times out, so my client retries it. However, the initial request actually went through, but I just didn't receive the confirmation. Suddenly, every employee is double-paid, or worse, no one gets paid at all. One small glitch on the network level just turned into a very expensive mistake that can cause serious financial and compliance issues.

Thankfully, defensive programming can help. Instead of assuming that everything works as expected, defensive programming involves writing API clients that anticipate failures and protect against them. Networks are unreliable, services have downtimes, and bugs happen, but a resilient client treats these as the norm, not an exception. By designing with failure in mind, you can avoid critical errors, like duplicate charges, missed payments, or cascading retries that overwhelm a service.

In this guide, I'll share techniques for building resilient API clients. I show how to implement retries safely using exponential backoff and why idempotent requests are important. I illustrate these points with examples of Gusto Embedded's Payroll APIs.

Understand Why Defensive Programming Matters for API Clients

We all know APIs don't run in perfect environments. Failures are not only possible but expected. You can see everything from transient network drops and HTTP 500 errors under peak load to rate-limit responses like HTTP 429 Too Many Requests. Even DNS resolution failures or partial timeouts can cause requests to hang indefinitely. Even with robust infrastructure, no API can guarantee 100 percent uptime or consistency. If my API client design assumes success by default, one minor issue can break my integration and ripple through the rest of my system.

In domains with high stakes like payroll, defensive programming is even more important. If retries aren't handled correctly, duplicate adjustments or repeated tax withholdings can occur, leading to inaccurate payroll results. These issues can harm trust, create compliance risks, and be costly to fix.

A defensive mindset is critical to building resilient API clients. Instead of assuming every request succeeds, assume the requests may be lost, delayed, or duplicated. Design each interaction with the expectation of failure. When you plan for failure at every step, your client can keep functioning gracefully even when the API or network is unpredictable.

Build Resilient API Clients Using Defensive Programming

Resilient API clients aren't built with a single technique but through a set of practices designed to predict and contain failure.

Handle Failures Correctly with Retries

Retries are the primary tool developers reach for when dealing with unreliable APIs. The idea is this: if a request fails, send it again. However, not all failures are created equal, and retrying blindly can do more harm than good.

Manage Transient and Permanent Failures

Before retrying after an error, you need to decide whether you should retry or stop and fail fast. The distinction between transient and permanent failures helps guide that logic.

Transient errors are faults that may resolve on their own if you try again. Examples include network timeouts, service overloads, or temporary outages in downstream systems. In HTTP, you often see status codes like 500 Internal Server Error, 502 Bad Gateway, 503 Service Unavailable, or even 504 Gateway Timeout. These are signals that the server isn't able to fulfill the request right now, but may be able to later. In these cases, retrying makes sense, provided you back off appropriately.

A special case worth mentioning is 429 Too Many Requests. This indicates that your client has hit a rate limit. While it's technically a transient error, you shouldn't retry immediately. Instead, check the Retry-After header in the response (if present) and wait for the specified duration before sending another request. This ensures your client respects server limits and avoids being throttled further.

Permanent errors reflect problems that can't be resolved with a retry. These include 400 Bad Request, 401 Unauthorized, 403 Forbidden, 404 Not Found, and 422 Unprocessable Entity due to invalid parameters. Once one of these errors is returned, retrying is futile. Instead, your client should fail fast and surface the error for intervention.

Prevent the Risk of Naive Retry Strategies

Naive retry strategies can introduce new problems. If every client retries immediately after a timeout, the server can become overwhelmed by a surge of duplicate requests. Even worse, a poorly designed client can execute the same operation twice, leading to unintended side effects, like submitting the same payroll run multiple times. Error handling must go beyond simply trying again.

Combine Exponential Backoff with Jitter

To avoid these pitfalls, the industry standard is to combine exponential backoff with jitter. Exponential backoff spaces out each retry by increasing the delay each time: wait one second, then two, then four, then eight, and so on. Jitter adds a random variation to these wait times, ensuring multiple clients don't retry at the same time and overwhelm the server.

Here's a simple Python example showing exponential backoff with jitter when calling an API endpoint. This example uses a GET request for simplicity. The same retry logic applies to POST, PUT, PATCH, and DELETE operations as well. Just make sure those operations are idempotent before retrying so that multiple attempts do not cause unintended side effects:

import requests
import time
import random

def make_request_with_retries(url, max_retries=5):
   last_exception = None

   for attempt in range(max_retries):
       try:
           response = requests.get(
               url,
               headers={
                   "accept": "application/json",
                   "X-Gusto-API-Version": "2025-06-15",
                   "authorization": "Bearer <YOUR-COMPANY-API-TOKEN>"
               },
               timeout=5
           )
           response.raise_for_status()
           return response.json()

       except requests.exceptions.RequestException as e:
           last_exception = e

           # Don't retry on client errors like 400 or 401
           if response is not None and 400 <= e.response.status_code < 500:
               raise e

           # Don't sleep after the last attempt
           if attempt < max_retries - 1:
               wait_time = (2 ** attempt) + random.uniform(0, 1)
               print(f"Retry {attempt + 1}/{max_retries} failed: {e}. Retrying in {wait_time:.2f}s...")
               time.sleep(wait_time)
           else:
               print(f"Retry {attempt + 1}/{max_retries} failed: {e}. Max retries reached.")

   raise last_exception or Exception("Max retries reached. Request failed.")

# Example usage:
data = make_request_with_retries("https://api.gusto-demo.com/v1/payrolls/<PAYROLL_UUID>/receipt")

This code attempts to get a single payroll receipt with increasing wait times between retries. Each failure triggers an exponential delay (one second, two seconds, four seconds, and so on) with a bit of randomness added (the jitter) to avoid synchronized retry attempts. If all attempts fail, it raises the last exception so that the calling code can log or handle it appropriately.

Handle Idempotency: Design Safe Retries

Retries are only truly safe when they're paired with idempotency. In the context of APIs, idempotency means that a request can be executed multiple times but only produces the same outcome as the initial successful execution. It's the safeguard that prevents retries from accidentally causing duplicate side effects.

In real-world systems, idempotency protects against the unpredictable behavior of networks and clients. Imagine a user submitting a payment form or signing up for an account, and their browser freezes midway. If they refresh and the client resends the same request, idempotency ensures the user isn't charged twice or created as a duplicate record (in cases where there are no unique identifiers, such as username or email).

The most common way to achieve this is through a unique identifier that links all the attempts of a request to a single logical operation. The Gusto Embedded Payroll APIs support this pattern with the Idempotency-Key header. When a client submits a request with an idempotency key, the server stores both the request and its result. If the client retries with the same key against the same endpoint, the server doesn't run the operation again; it simply returns the previously stored result.

Version-based object management takes this a step further. For update operations, the API requires a version field that helps prevent race conditions. If the resource has changed since the client's last request, the API returns a 409 Conflict instead of silently overwriting data. This guarantees that retries don't create conflicting state changes even under concurrent operations.

This approach is clearly important in payroll systems where correctness is important. Imagine creating a payroll run for your workforce. Without an idempotency key, a retry after a timeout may be interpreted as a second payroll run, leading to employees being double-paid and the company facing compliance issues. With idempotency in place, all retries are tied back to the same logical run, so employees are paid exactly once, no matter how many times the request is retried.

Here's an example of how you can send a payroll creation request with an idempotency key header to guarantee safe retries:

import uuid
import requests

def create_payroll(company_id, idempotency_key, payroll_data):
   headers = {
       "Authorization": "Bearer <YOUR-COMPANY-ACCESS-TOKEN>",
       "accept": "application/json",
       "X-Gusto-API-Version": "2025-06-15",
       "Content-Type": "application/json",
       "Idempotency-Key": idempotency_key  # unique key for this operation,
   }

   url = f"https://api.gusto-demo.com/v1/companies/{company_id}/payrolls"

   response = requests.post(
       url,
       json=payroll_data,
       headers=headers
   )
   response.raise_for_status()
   return response.json()

# Example usage
idempotency_key = str(uuid.uuid4())
payload = {
  "off_cycle": True,
  "off_cycle_reason": "Bonus",
  "start_date": "2025-11-11",
  "end_date": "2025-11-18"
}

result = create_payroll(company_id='123', idempotency_key=idempotency_key, payroll_data=payload)
print(result)

This code sends a create payroll request to Gusto while using an idempotency key to ensure safe retries. It generates a unique key with uuid.uuid4() and passes it in the Idempotency-Key header so the API can recognize repeat attempts as the same operation.

Implement Resilient API Client Patterns

Once you understand retries and idempotency, the next step is to apply them consistently. Resilient API clients aren't just a collection of one-off fixes; they're built around a set of design patterns that treat reliability as a core concern. These patterns assume that every request can fail, and they put safeguards in place so those failures never compromise correctness.

Building resilience also means thinking beyond a single API call. In distributed systems, dependencies can go down, networks can degrade, and partial failures can ripple across services. Robust clients use techniques, such as fallback services (eg using cached data or a secondary endpoint when the primary is unavailable), graceful degradation (serving limited functionality instead of breaking completely), and circuit breakers to prevent cascading failures. Together with retries, timeouts, and idempotency, these techniques form the foundation of reliability in APIs.

Here are the key patterns that guide the design of reliable API clients:

Pattern	Why It Matters	Example in Practice
Treat every request as if it can fail.	Networks and servers are inherently unreliable. Coding as if success is guaranteed leads to brittle integrations.	Always implement timeouts, retries, and error handling logic.
Ensure critical operations are idempotent.	Without idempotency, retries can trigger duplicate side effects, like double payments.	Use an `Idempotency-Key` (or equivalent identifier) to safely retry operations. For example, Gusto supports this through the `Idempotency-Key` header.
Monitor and log retry behavior.	Silent retries hide systemic issues. Visibility is needed for debugging and capacity planning.	Record retry attempts and their outcomes in your client logs.
Back off intelligently.	Immediate retries amplify the load and risk further failure. Exponential backoff with jitter smooths traffic spikes.	Retry requests with increasing wait times and randomized delays.
Fail fast on permanent errors.	Retrying invalid requests wastes resources and delays recovery.	Stop retries when the API returns `4xx` client errors like `401 Unauthorized`.

When applied together, these patterns transform an integration from fragile to dependable. Instead of breaking down under real-world conditions, the client absorbs failures, retries safely, and communicates clearly when something is wrong.

Optimistic Version Control

Another method of ensuring data integrity is utilizing optimistic version control. Object-based versions—essentially snapshots of a given resource—are used to process concurrent updates correctly, ensuring data integrity in a multi-user environment. It is a form of optimistic concurrency control, where the system 'assumes' that conflicts are rare and only checks for them at the time of an update.

Conclusion

Building resilient API clients means embracing the reality that things fail: networks drop, servers timeout, and dependencies falter. You need to anticipate these failures—handle retries with exponential backoff and make requests idempotent so developers can ensure their integrations behave predictably, even under stress.

JS Solutions for CodeWars Katas: Part 2

Kevin Kimani — Tue, 24 May 2022 06:05:33 +0000

1.Write an algorithm that takes an array and moves all of the zeros to the end, preserving the order of the other elements.

moveZeros([false,1,0,1,2,0,1,3,"a"]) // returns[false,1,1,2,1,3,"a",0,0]

My solution:

const moveZeros = arr => arr.filter(num => num !== 0).concat(arr.filter(num => num === 0))

Long version:

function moveZeros(arr) {
  let zerosArr = arr.filter(num => num === 0)
  let nonZerosArr = arr.filter(num => num !== 0)
  return nonZerosArr.concat(zerosArr)
}

2.Write a function that accepts an array of 10 integers (between 0 and 9), that returns a string of those numbers in the form of a phone number.

Example
createPhoneNumber([1, 2, 3, 4, 5, 6, 7, 8, 9, 0]) // => returns "(123) 456-7890"
The returned format must be correct in order to complete this challenge.
Don't forget the space after the closing parentheses!

const createPhoneNumber = numbers =>
   `(${numbers.slice(0,3).join('')}) ${numbers.slice(3,6).join('')}-${numbers.slice(6, 10).join('')}`

JS Solutions for CodeWars Katas

Kevin Kimani — Sun, 22 May 2022 14:42:02 +0000

In this article I will be documenting the solutions for codewars challenge questions that I am able to solve:

1.Your task is to sum the differences between consecutive pairs in the array in descending order.

For example:

[2, 1, 10] -> 9
In descending order: [10, 2, 1]

Sum: (10 - 2) + (2 - 1) = 8 + 1 = 9

If the array is empty or the array has only one element the result should be 0 ( Nothing in Haskell ).

My solution:

function sumOfDifferences(arr) {
  arr.sort(function(a, b){return b - a});
  let diff = 0
  for(let i=0;i<=arr.length - 2;i++){
    diff = diff + (arr[i] - arr[i + 1])
  }
  return diff
}

2.A square of squares
You like building blocks. You especially like building blocks that are squares. And what you even like more, is to arrange them into a square of square building blocks!

However, sometimes, you can't arrange them into a square. Instead, you end up with an ordinary rectangle! Those blasted things! If you just had a way to know, whether you're currently working in vain… Wait! That's it! You just have to check if your number of building blocks is a perfect square.

Task
Given an integral number, determine if it's a square number:

In mathematics, a square number or perfect square is an integer that is the square of an integer; in other words, it is the product of some integer with itself.

The tests will always use some integral number, so don't worry about that in dynamic typed languages.

Examples
-1 => false
0 => true
3 => false
4 => true
25 => true
26 => false

My solution:

var isSquare = function(n){
  if(n>-1){
    if(JSON.stringify(Math.sqrt(n)).includes('.')){
      return false
    }
    else{
      return true
    }
  }
  else{
    return false
  }
}

Refactored solution:

var isSquare = n => Math.sqrt(n) % 1 === 0;

3.Implement a function that adds two numbers together and returns their sum in binary. The conversion can be done before, or after the addition.

The binary number returned should be a string.

Examples:(Input1, Input2 --> Output (explanation)))

1, 1 --> "10" (1 + 1 = 2 in decimal or 10 in binary)
5, 9 --> "1110" (5 + 9 = 14 in decimal or 1110 in binary)

My solution:

function addBinary(a,b){
  return (a+b).toString(2)
}

4.Given an array of integers, find the one that appears an odd number of times.

There will always be only one integer that appears an odd number of times.

Examples
[7] should return 7, because it occurs 1 time (which is odd).
[0] should return 0, because it occurs 1 time (which is odd).
[1,1,2] should return 2, because it occurs 1 time (which is odd).
[0,1,0,1,0] should return 0, because it occurs 3 times (which is odd).
[1,2,2,3,3,3,4,3,3,3,2,2,1] should return 4, because it appears 1 time (which is odd).

My solution:

function findOdd(A) {
  const count = {};

  for (const element of A) {
    if (count[element]) {
      count[element] += 1;
    } else {
      count[element] = 1;
    }
  }

  for (const property in count) {
    if(count[property] % 2 !== 0){
      return parseInt(property)
    }
  }
}

5.Make a program that filters a list of strings and returns a list with only your friends name in it.

If a name has exactly 4 letters in it, you can be sure that it has to be a friend of yours! Otherwise, you can be sure he's not...

Ex: Input = ["Ryan", "Kieran", "Jason", "Yous"], Output = ["Ryan", "Yous"]

i.e.

friend ["Ryan", "Kieran", "Mark"] shouldBe ["Ryan", "Mark"]
Note: keep the original order of the names in the output.

My solution:

const friend = friends => friends.filter(friend => friend.length === 4)

How to Rotate a String in JavaScript

Kevin Kimani — Sun, 22 May 2022 09:10:59 +0000

Recently, I applied a job in a certain company that was hiring a React developer. I sent my CV, did a coding assessment and passed. After this I was invited for an in-person interview where I was required to write code on paper.

The coding problem was: write code in the programming language of your choice that rotates a given string a specified number of times.

At first I freaked out as I had not tackled such a problem before, keep in mind that I am not a guru. I took the pen and paper and started writing code.

After struggling for a few minutes, I was able to write code for positive number rotation but I didn't know that they also required the code to also handle negative number rotation and my time was up.

Anyway, I was able to solve the negative rotation part in my mind on my way home and I want to share it here in case you ever encounter such a problem.

Here's is the JS code to rotate a string(handles both left and right rotation):

const rotate = (text, n) => {
  txtArr = text.split("") 

  if(n>0){
    for(let i = 1; i<=n; i++){
      let firstel = txtArr[0]
      txtArr.shift()
      txtArr.push(firstel)
    }
  }
  else if(n<0){
    for(let i = -1; i>=n; i--){
      let lastel = txtArr[txtArr.length-1]
      txtArr.pop()
      txtArr.unshift(lastel)
    }
  }

  console.log(txtArr.join(""))
}

rotate("hello", -2)

Code explanation:

I will explain the code with the steps involved:

Convert the text into an array using the text.split() method.
Use an if statement to check whether n is positive, n>0.
If n is positive:

use a for loop to loop through the textArr n times.
get the first element in the array using txtArr[0]
Remove the first array element using txtArr.shift()
Add the first array element to the end of the aaray using txtArr.push(firstel)

4.If n is negative:

use a for loop to loop through the textArr n times.
get the last element in the array using txtArr[txtArr.length - 1]
Remove the last array element using txtArr.pop()
Add the last array element to the beginning of the aaray using txtArr.unshift(lastel)

5.If n is equal to 0, return it as it is

Conclusion

That's all for this article. Hope you learn a thing or two. See you in the next one :)

HTML For Beginners: Cool Stuff You Can Do With HTML Only

Kevin Kimani — Wed, 02 Mar 2022 07:34:47 +0000

Hey there web developer. Hope you doing good. In this short article we will take a look at a few cool things that you can actually achieve with HTML only. I'll try to keep this article short and brief. Lets jump right into it!

1. Creating an Accordion with HTML only.

Yap! This is possible. An accordion is mostly used in FAQ section of a website. Let's take a look at simple code snippet we can use to create this:

<details>
  <summary>I am an accordion</summary>
  <p>I am the accordion body. Lorem ipsum dolor sit amet consectetur adipisicing elit. Reiciendis fugiat quia provident cum laudantium nobis minus magni a ex officia! Ab ex nemo tenetur nihil?</p>
</details>

2. Progress Bar with HTML only.

This is mostly used by developers in their portfolios to show their their level of expertise in a certain field as a percentage. Below is a code snippet for achieving this:

<label for="progressBar">Web Dev: </label>
<progress id="progressBar" value="95" max="100"></progress><span> 95% </span>

3. Autocomplete with HTML only.

Autocomplete is a feature in which an application predicts the rest of a word a user is typing. Let's see how we can achieve this. As you type in the provided input field, you will see a list of the options defined in the datalist.

<input list="programmingLanguages">
<datalist id="programmingLanguages">
  <option value="JavaScript">
  <option value="Python">
  <option value="PHP">
</datalist>

4. Color Picker with HTML only.

This is self explanatory. It is used to pick a color 😂. After picking a color, it gives you the option of picking a colors rgb, hex or hsl value. To set a default color that is not black you just need to specify the value attribute. Let's see how we can achieve this:

<label for="colorPicker">Select a color: </label>
<input type="color" id="colorPicker" name="colorPicker" value="#ff0000">

5. Simple Calculator with HTML only.

This has some JavaScript but its pretty basic and beginner friendly. And of course I wanted to include it in this list 😁. All the JavaScript in this code snippet does is just convert the value entered by the user from a string to an integer Let's take a look at how we can do it:

<form oninput="result.value=parseInt(a.value)+parseInt(b.value)">
      <input type="number" id="b" name="b" value="10" /> +
      <input type="number" id="a" name="a" value="10" /> =
      <output name="result" for="a b">20</output>
 </form>

Conclusion

And that is all for this article my friends. Hope you enjoyed it and learn a few things. See you in the next one. Peace out!✌️