DEV Community: Ricardo Medeiros

You Probably Understood the Client Server Model Wrong

Ricardo Medeiros — Thu, 11 Dec 2025 15:19:31 +0000

Most developers intuitively grasp the client–server model. They know the backend must serve many users while the frontend runs a single isolated instance per person. The problem is not one of misunderstanding but of misapplied mental models.

Too often, frontend interaction patterns quietly seep into backend architecture. This happens almost invisibly: because a user clicks a button and waits, the backend endpoint is implemented to perform the entire workflow before returning.

Because the UI shows near-real-time progress, developers reach for WebSockets, SSE, or long-polling, assuming the backend must act with the same immediacy. As the interface suggests an atomic action, the backend is built to execute a full, complex sequence in a single synchronous request.

These decisions feel natural when thinking from the user outward, but they ignore the fact that frontend and backend systems live under radically different constraints. The frontend exists per user, per tab, per device. If it blocks or performs work synchronously, only one person is affected.

In the other hand, the backend is a shared computational surface, serving thousands of simultaneous users from a limited pool of machines. A blocking call that seems harmless in the UI becomes costly when multiplied by a large user base.

This distinction is critical. For lightweight operations—like fetching a user profile or toggling a setting—mimicking the frontend’s synchronous expectations is perfectly acceptable; the resource cost is negligible. The danger arises when we apply that same 'request-response' immediacy to complex business logic or third-party integrations.

When a heavy, valuable workflow is forced to fit inside the fragile lifespan of a simple HTTP request, the backend stops being a scalable coordinator and becomes a brittle bottleneck.

Consequently, a heavy workflow executed synchronously does not scale when many people trigger it at once. Frontend mental models make perfect sense for the UI layer but become harmful when projected onto the backend.

The first step toward correcting this misalignment is to visualize the structural asymmetry. Each user runs their own frontend instance, but a comparatively small set of backend servers must collectively serve all of them.

When design decisions assume parity between these layers, the backend becomes overloaded—not because of traffic volume alone, but because of how tightly the backend’s execution model is coupled to human interactions.

Frontend Per User, Backend Shared by Everyone

This diagram illustrates the heart of the issue: every user has a dedicated frontend instance, but all users share the backend. If backend routes mimic user-level workflows synchronously, they consume server resources as if each request were serving only that user.

This leads to patterns where backend operations are tied to the lifetime of an HTTP request or WebSocket connection, causing servers to hold open sockets, allocate memory, keep database transactions alive, or block worker threads while waiting for slow I/O. These patterns work fine when thinking like a single UI instance, but they collapse under real-world concurrency.

Even when developers use asynchronous programming constructs—such as async/await—the architecture remains synchronous if long-running work is still executed within the request lifecycle.

Architectural asynchrony is not achieved by non-blocking code alone; it comes from offloading work, decoupling the response from the completion of the operation, and treating the backend as a shared coordinator of workflows rather than a synchronous executor of user-driven commands.

The Shift Toward Asynchronous Endpoints

A backend that scales must adopt a different perspective: the purpose of an endpoint is not to “complete the job” but to “accept the job.” True asynchronous architecture emerges not from code-level semantics but from systemic decoupling.

The backend should perform only the minimal steps required to validate input, authenticate the caller, record the intent, and enqueue the real work elsewhere. It should return immediately—often with a 202 Accepted status—handing back a job identifier that the frontend can use to check progress.

This “accept, then process” pattern changes the nature of the system entirely. Instead of backend servers waiting idly during long-running operations, the work is delegated to background processors or workers that run independently of the request lifecycle. This frees backend capacity for new incoming requests, allowing the system to serve more users without proportional increases in server count.

Asynchronous Workflow Sequence

In this design, the backend no longer mirrors the interaction model of the UI. The frontend and backend operate on independent clocks: the UI can poll for updates at its own pace, and the backend can schedule work according to resource availability.

Polling, often dismissed as simplistic, becomes a powerful pattern for background workflows because it keeps servers stateless and scalable, avoids long-lived connections, and shifts the complexity of immediacy away from the backend.

Status Endpoints and Cache-Assisted Polling

When the frontend receives a job identifier, it begins periodically querying a /jobs/:id endpoint for status updates. This endpoint is simple by design: it retrieves the current state of a workflow from the database or, more efficiently, from a cache.

This pattern allows backend servers to remain fully stateless. As no request needs to wait for job completion, the backend can scale horizontally without sticky sessions or shared in-memory state. Caching enhances this model. A cache-aside pattern—where the API checks the cache first and falls back to the database only on a miss—is often sufficient for moderate traffic.

For larger systems, a write-through approach pays off: workers update both the database and the cache when job statuses change. This ensures that nearly all frontend polling requests hit the cache rather than the database, dramatically reducing load.

Status Endpoint Architecture

This architecture exemplifies how decoupled systems naturally scale. Backend servers focus solely on orchestrating requests and responding quickly. Workers and queues handle the heavy lifting. Polling reads mostly from cache. No part of the system depends on a long-lived user connection.

How Serverless Completes the Picture

Serverless platforms elevate this model by providing isolation and elasticity at the execution level. While backend servers remain stable and responsive, serverless functions scale independently based on event volume.

Each job can run in its own execution context without interfering with others, effectively giving the backend a burst capacity proportional to incoming workload rather than to the number of servers provisioned.

This division of responsibilities is transformative. Backend servers become thin control planes responsible only for coordination, while serverless workers perform compute-heavy or long-running tasks. Instead of scaling servers, the system scales events. When a thousand users submit jobs simultaneously, a thousand serverless functions can execute in parallel without any user blocking the backend.

Serverless architecture embodies the principle that the backend should not behave like the UI. Instead of executing workflows synchronously, backend servers delegate. Instead of waiting, they acknowledge. Instead of owning long-lived operations, they outsource them to stateless, ephemeral functions optimized for scaling.

A New Mental Model for Cloud-Native Backends

Reframing the backend begins with recognizing that frontend patterns should not dictate backend structures. The UI can wait; the backend must not. The UI can block; the backend must stay available. The UI serves a single human; the backend serves everyone simultaneously.

When dealing with simple CRUD-like operations that lack aggregated business logic, a synchronous model remains the simplest and most effective choice. However, when involving complex business logic, third-party integrations, or intensive traffic peaks, that same model creates a bottleneck. In those high-stakes scenarios, synchronous coupling leads to critical failures that can bring the entire system down.

Once this mental shift clicks, it becomes clear why asynchronous endpoints, polling, queues, workers, caches, and serverless execution represent not advanced architectural strategies but necessary correctives to an intuitive but incorrect assumption: that backend systems should behave like the interfaces that sit on top of them.

Backend architecture must reflect backend realities. Designing it with a frontend mindset is what leads to systems that stall, choke, or become expensive to operate. Designing it with asynchronous principles produces architectures that scale effortlessly, fail gracefully, and serve users far beyond what synchronous workflows could ever sustain.

Marked UUIDs Pattern

Ricardo Medeiros — Fri, 28 Jun 2024 20:24:40 +0000

Introduction

So, here I'm again. Writing after some time without doing anything like it, but I've decided to share this, because I think this solution saved a lot of work and problems that system could had. The context was this: my team was responsible for an application that was been integrated into a SAP system by another company, and, we had several PSP integrations to provide our services.

Some of this PSP test enviroments didn't work at all, they were flaky and inconsistant with responses and even the types of the response properties between the test and production environment. We needed to provide a way for the SAP consultancy to be able to test the functionalities of our plataform without the instability of the PSP test enviroment, and then, we decided to add a flag to mock the PSP response to be successfull (since it was the outcome that we were more interested in).

It was all great, untill we realized that we needed to test the refund functionality. Now what? As our application was built upon microsservices with domain segregation. We had an Anti-Corruption-Layer (ACL) between it and the PSP, so we had no idea of what was happening there. Both payment and refund were in the ACL, but it hadn't any persistance, as was built to be a facade between the third-party service provider and our domain. How could we determine if a payment was done in the actual service provider test environment or was mocked by our ACL?

Aproaches

Add a parameter in the response of the ACL indicating if this response is mocked or not;
Add a database in the ACL to save all the payments created, and if they were mocked or not.

Aditional Parameter

If we had chosen this path, it would require that the application consuming the ACL was changed, to consider this new information about mocking, then, this consuming application would need to save this in order to provide this information back to the ACL when refund were requested. It required to much development just to enable this, and it didn't add anything to our product, just it's testability experience.

Database in the ACL

Imagine this, a service that operated without databases in production would require one to work in test enviroment. I promptly refused, even not enjoying the first option, this was way worst. Not only would require aditional infrastructure, but also add futher complexity and latency to the system.

Avoinding these pains

So, I've started to think. How could I know this information without adding any furter infrastructure or adding parameters? Somehow, I needed to be able to split this register with the information that they already had. With that in mind, I've started to come up with a solution.

UUID - The Trojan Horse

Every transation would return an UUID that should be generated by the PSP (or not). Somehow mocking this information in a way that it would be differentiable from the PSP generated ID's wouldn't affect the transaction information, all the further information required would be maintained. With that in mind, I wen't on several experiments in order to make this possible, and that's how I've Achieved it:

Marked UUID

UUIDs are pseudo-random 128-bit identifiers that are often represented as 32 hexadecimal digits, displayed in five groups separated by hyphens, in the form 8-4-4-4-12 (e.g., 123e4567-e89b-12d3-a456-426614174000).

UUIDv4

UUIDv4 is one of the most commonly used versions of UUIDs. It is designed to be a universally unique identifier generated using random or pseudo-random numbers. Here's a detailed overview of UUIDv4:

Characteristics of UUIDv4

Randomness: UUIDv4 relies on random or pseudo-random numbers to generate the unique identifier.
No External Information: Unlike other versions of UUIDs, such as UUIDv1 (which incorporates timestamps and MAC addresses), UUIDv4 does not include any external information about the generating system.
Simplicity: UUIDv4 is straightforward to implement because it only requires a good source of randomness.

Structure of UUIDv4

A UUIDv4 is a 128-bit value, typically represented as a 36-character string in the format 8-4-4-4-12, separated by hyphens.

Example UUIDv4

f47ac10b-58cc-4372-a567-0e02b2c3d479

Breakdown of UUIDv4

8 characters: Randomly generated 32 bits.
4 characters: Randomly generated 16 bits.
4 characters: Randomly generated 12 bits, and 4 bits for the version (0100 for version 4).
4 characters: Two or three bits for the variant (10x for RFC 4122), and - 13 or 14 randomly generated bits.
12 characters: Randomly generated 48 bits.

Marking UUIDv4

The process to mark this ID should be something that couldn't be done by an valid UUID generation, but It should be compatible with UUID implementations in order to be stored sucesfully, avoiding any changes outter the Anti-Corruption Layer code.

I've selected the last 12 characters section to mark the ID, but I needed for it to be unmistaken with real IDs, so I've devised this algorithm:

Take the ID without the last 12 characters and the hypens
Use a hash algorithm like SHA-256 to creat a mark
Grab the first 12 characters (48 bits) of the hash and replace the original generated 12 characters

That's it. Stupidly simple, works with the language implementation (in my case was golang) and was unmistaken with a real gerenated UUID. But how can we determine if an ID is marked? It is as simple as to create one:

Take the ID without the last 12 characters and the hypens
Use a hash algorithm like SHA-256 to creat a mark
Grab the first 12 characters (48 bits) of the hash and compare it to the IDs last 12, if they're equal, this ID was marked.

Solution

So, how did I've employed this to solve the problem? Simple, when I had the flag to mock the PSP responses enabled, I've returned at the payment endpoint the mocked response with an marked ID. In the refund endpoint, if the flag was enabled, I checked if the ID was marked, if so, I would return a mocked refund response. If not, I would make the PSP call, as this payment was created in the PSP enviroment.

Conclusion

If something you gotta do seems to require too much corromption to your application, look for alternative approaches to this problem and don't be afraid to try new things out!

Any sugestions, comments or corrections, fell free to reach me out at LinkedIn.

Ricardo MedeirosFollow

Senior Cloud Software Engineer @Caylent

KAFKA + KSQLDB + .NET #1

Ricardo Medeiros — Tue, 30 Nov 2021 17:19:05 +0000

Hi, I'm Ricardo Medeiros, .NET back end developer @vaivoa, and today I'm going to walk you through using ksqlDB to query messages produced in kafka by a .NET/C# producer. For this example, I will be deploying my enviroment as containers, described in a docker compose file, to ensure easy reproducibility of my results.

The source code used in this example is avaliable here.

Services

First, let's talk about the docker compose environment services. the file is avaliable here.

.NET API Producer

Automaticaly generated .NET api with docker compose service

ksqldbdemo:
    container_name: ksqldbdemo
    image: ${DOCKER_REGISTRY-}ksqldbdemo
    build:
      context: .
      dockerfile: Dockerfile

This producer service needs the .NET generated dockerfile shown below:

FROM mcr.microsoft.com/dotnet/aspnet:5.0 AS base
WORKDIR /app
EXPOSE 80
EXPOSE 443

FROM mcr.microsoft.com/dotnet/sdk:5.0 AS build
WORKDIR /src
COPY ["ksqlDBDemo.csproj", "."]
RUN dotnet restore "ksqlDBDemo.csproj"
COPY . .
WORKDIR "/src/"
RUN dotnet build "ksqlDBDemo.csproj" -c Release -o /app/build

FROM build AS publish
RUN dotnet publish "ksqlDBDemo.csproj" -c Release -o /app/publish

FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "ksqlDBDemo.dll"]

ZooKeeper

Despite not been necessary since Kafka 2.8, ZooKeeper coordinates kafka tasks, defining controllers, cluster membership, topic configuration and more. In this tutorial, it's used the confluent inc. ZooKeeper image, due to it's use in the reference material. It makes Kafka more reliable, but adds complexity into the system.

zookeeper:
    image: confluentinc/cp-zookeeper:7.0.0
    hostname: zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

Kafka

Kafka is an event streaming plataform capable of handling trillions of events a day. Kafka is based on the abstraction of an distributed commit log. Initialiy developed at LinkedIn in 2011 to work as a message queue, but it has evolved into a full-fledge event streanming platfmorm. Listed as broker in the services, is the core of this tutorial. It's configuration is tricky, but using it as follows worked well in this scenario.

 broker:
    image: confluentinc/cp-kafka:7.0.0
    hostname: broker
    container_name: broker
    depends_on:
      - zookeeper
    ports:
      - "29092:29092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1

ksqlDB

ksqlDB is a database built to allow distributed stream process applications. Made to work seamsly with kafka, it has a server that runs outside of kafka, with a REST API and a CLI application that can be run separatly and it's used in this tutorial.

ksqlDB Server

In this example, it's used the confluent inc image of the ksqlDB server, once more, due to it's widespread usage.

ksqldb-server:
    image: confluentinc/ksqldb-server:0.22.0
    hostname: ksqldb-server
    container_name: ksqldb-server
    depends_on:
      - broker
    ports:
      - "8088:8088"
    environment:
      KSQL_LISTENERS: http://0.0.0.0:8088
      KSQL_BOOTSTRAP_SERVERS: broker:29092
      KSQL_KSQL_LOGGING_PROCESSING_STREAM_AUTO_CREATE: "true"
      KSQL_KSQL_LOGGING_PROCESSING_TOPIC_AUTO_CREATE: "true"

ksqlDB CLI

The same goes for the ksqlDB CLI service, that also use the confluent inc image.

ksqldb-cli:
    image: confluentinc/ksqldb-cli:0.22.0
    container_name: ksqldb-cli
    depends_on:
      - broker
      - ksqldb-server
    entrypoint: /bin/sh
    tty: true

Kafdrop

Kafdrop is a Web UI for viewing kafka topics and browsing consumer groups. It makes kafka more accessible.

kafdrop:
    container_name: kafdrop
    image: obsidiandynamics/kafdrop:latest
    depends_on:
      - broker
    ports:
      - 19000:9000
    environment:
      KAFKA_BROKERCONNECT: broker:29092

Tutorial

Now it's the time that you have been waiting, let's make it work!

Enviroment

For this tutorial, you'll need a docker desktop installation, either it's on a Linux distribution or on Windows with WSL and git.

Cloning the project

A Visual Studio project is avaliable here, it has docker support and already deploys all the services needed for this demo in the IDE. However, you will be fine if you don't want or can't use Visual Studio. Just clone it, running the following comand on the terminal and directory of your preference:

 $ git clone https://github.com/jjackbauer/ksqlDBDemo.git

Use the following command to move to the project folder:

 $ cd /ksqlDBDemo

And, in the project folder, that contains the docker-compose.yml run the following command to deploy the services:

$ docker compose up -d

after this command, make sure that all services are running. Sometimes services fall, but it is okay. In order to see if everything is running ok, it's possible to see the services running in docker desktop, as shown bellow:

Or you can execute the following command:

$ docker ps

Which should output something like this:

CONTAINER ID   IMAGE                               COMMAND                  CREATED       STATUS       PORTS
  NAMES
b42ce9954fd9   ksqldbdemo_ksqldbdemo               "dotnet ksqlDBDemo.d…"   2 hours ago   Up 2 hours   0.0.0.0:9009->80/tcp, 0.0.0.0:52351->443/tcp   ksqldbdemo
0a0186712553   confluentinc/ksqldb-cli:0.22.0      "/bin/sh"                2 hours ago   Up 2 hours
  ksqldb-cli
76519de6946e   obsidiandynamics/kafdrop:latest     "/kafdrop.sh"            2 hours ago   Up 2 hours   0.0.0.0:19000->9000/tcp
  kafdrop
11c3a306ee01   confluentinc/ksqldb-server:0.22.0   "/usr/bin/docker/run"    2 hours ago   Up 2 hours   0.0.0.0:8088->8088/tcp
  ksqldb-server
07cef9d69267   confluentinc/cp-kafka:7.0.0         "/etc/confluent/dock…"   2 hours ago   Up 2 hours   9092/tcp, 0.0.0.0:29092->29092/tcp
  broker
3fa1b9a60954   confluentinc/cp-zookeeper:7.0.0     "/etc/confluent/dock…"   2 hours ago   Up 2 hours   2888/tcp, 0.0.0.0:2181->2181/tcp, 3888/tcp     zookeeper

WEB API

Now, with all services up and running, we can access the WEB API Swagger to populate our Kafka topics. The code is very simple and it's avaliable in the repository.

The WEB API swagger is deployed at http://localhost:9009/swagger/index.html. As shown in the image bellow, it has two endpoints and they create events that could be created by indepent microservices. One for creating an event that creates a userName in the system and another that takes an Id and generates a three digit code.

Then you can create an User with the user name of your choise, as shown:

And it will have an assigned unique Id, as demonstrated:

Now, you can get a three digit code for your user Id as displayed:

And a random code is generated for the selectd, as we can observe in the image that follows:

Kafdrop

We can use the kafdrop UI the check if everything is okay. Kafdrop is deployed at http://localhost:19000/.
There, you will find all the brokers and topics avaliable. It should look like this:

KSQL CLI

After all that, you'll be able to create your streams of data and query it using ksqlDB. On your preferential terminal, use the command:

docker exec -it ksqldb-cli ksql http://ksqldb-server:8088

Creating streams

And then you are in the ksql CLI and are free to create your streams and queries. First, let's create a stream for each one of our topics:

CREATE STREAM stream_user (Name VARCHAR, Id VARCHAR)
  WITH (kafka_topic='demo-user', value_format='json', partitions=1);

CREATE STREAM stream_code (Id VARCHAR, code INT)
  WITH (kafka_topic='demo-code', value_format='json', partitions=1);

Create a materialized view

You can join the client data with the most recent randomized code. to achieve this, you must create a materialized view table, that joins both streams as seen in the ksqldb script that follows:

CREATE TABLE currentCodeView AS
>   SELECT user.Name,
>   LATEST_BY_OFFSET(code.code) AS CurrentCode
>   FROM stream_code code INNER JOIN stream_user user
>   WITHIN 7 DAYS ON code.Id = user.Id
>   GROUP BY user.Name
>EMIT CHANGES;

Making a push query

After that, we can query this materialized view:

SELECT * FROM currentCodeView 
  EMIT CHANGES;

This push query keep on running until you hit cntrl+c to cancel it.

Conclusions

In this tutorial it's demonstrated that in a kafka + ksqlDB enviroment, you can make SQL queries and also join on data that comes from different events, which is one of most complexities envolved with microsservices systems. And it is what ksqlDB solves by enabling SQL operations over Kafka topics.
It's my goal to explore the possibilites allowed by this ecosystem and I hope to bring more knowledge on this topic in another articles here. Any sugestions, comments or corrections, fell free to reach me out at LinkedIn.

Ricardo MedeirosFollow

Senior Cloud Software Engineer @Caylent

References

ksqlDB Quickstart
ksqlDB Overview
Kafka .NET Client
ksqlDB Documentation - Data Types Overview
KSQL and ksqlDB
Welcome to Apache ZooKeeper
What is ZooKeeper & How Does it Support Kafka?
What is Apache Kafka®?
ksqlDB - The database purpose-built for stream processing applications
An overview of ksqlDB
CREATE TABLE AS SELECT
How to join a stream and a stream
Time and Windows in ksqlDB Queries
Time operations

Disclaimer

A VaiVoa incentiva seus Desenvolvedores em seu processo de crescimento e aceleração técnica. Os artigos publicados não traduzem a opinião da VaiVoa. A publicação obedece ao propósito de estimular o debate.

KAFKA + KSQLDB + .NET #1

Ricardo Medeiros — Mon, 29 Nov 2021 20:13:06 +0000

The source code used in this example is avaliable here.

Services

First, let's talk about the docker compose environment services. the file is avaliable here.

.NET API Producer

Automaticaly generated .NET api with docker compose service

ksqldbdemo:
    container_name: ksqldbdemo
    image: ${DOCKER_REGISTRY-}ksqldbdemo
    build:
      context: .
      dockerfile: Dockerfile

This producer service needs the .NET generated dockerfile shown below:

FROM mcr.microsoft.com/dotnet/aspnet:5.0 AS base
WORKDIR /app
EXPOSE 80
EXPOSE 443

FROM mcr.microsoft.com/dotnet/sdk:5.0 AS build
WORKDIR /src
COPY ["ksqlDBDemo.csproj", "."]
RUN dotnet restore "ksqlDBDemo.csproj"
COPY . .
WORKDIR "/src/"
RUN dotnet build "ksqlDBDemo.csproj" -c Release -o /app/build

FROM build AS publish
RUN dotnet publish "ksqlDBDemo.csproj" -c Release -o /app/publish

FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "ksqlDBDemo.dll"]

ZooKeeper

zookeeper:
    image: confluentinc/cp-zookeeper:7.0.0
    hostname: zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

Kafka

 broker:
    image: confluentinc/cp-kafka:7.0.0
    hostname: broker
    container_name: broker
    depends_on:
      - zookeeper
    ports:
      - "29092:29092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1

ksqlDB

ksqlDB Server

In this example, it's used the confluent inc image of the ksqlDB server, once more, due to it's widespread usage.

ksqldb-server:
    image: confluentinc/ksqldb-server:0.22.0
    hostname: ksqldb-server
    container_name: ksqldb-server
    depends_on:
      - broker
    ports:
      - "8088:8088"
    environment:
      KSQL_LISTENERS: http://0.0.0.0:8088
      KSQL_BOOTSTRAP_SERVERS: broker:29092
      KSQL_KSQL_LOGGING_PROCESSING_STREAM_AUTO_CREATE: "true"
      KSQL_KSQL_LOGGING_PROCESSING_TOPIC_AUTO_CREATE: "true"

ksqlDB CLI

The same goes for the ksqlDB CLI service, that also use the confluent inc image.

ksqldb-cli:
    image: confluentinc/ksqldb-cli:0.22.0
    container_name: ksqldb-cli
    depends_on:
      - broker
      - ksqldb-server
    entrypoint: /bin/sh
    tty: true

Kafdrop

Kafdrop is a Web UI for viewing kafka topics and browsing consumer groups. It makes kafka more accessible.

kafdrop:
    container_name: kafdrop
    image: obsidiandynamics/kafdrop:latest
    depends_on:
      - broker
    ports:
      - 19000:9000
    environment:
      KAFKA_BROKERCONNECT: broker:29092

Tutorial

Now it's the time that you have been waiting, let's make it work!

Enviroment

For this tutorial, you'll need a docker desktop installation, either it's on a Linux distribution or on Windows with WSL and git.

Cloning the project

 $ git clone https://github.com/jjackbauer/ksqlDBDemo.git

Use the following command to move to the project folder:

 $ cd /ksqlDBDemo

And, in the project folder, that contains the docker-compose.yml run the following command to deploy the services:

$ docker compose up -d

Or you can execute the following command:

$ docker ps

Which should output something like this:

CONTAINER ID   IMAGE                               COMMAND                  CREATED       STATUS       PORTS
  NAMES
b42ce9954fd9   ksqldbdemo_ksqldbdemo               "dotnet ksqlDBDemo.d…"   2 hours ago   Up 2 hours   0.0.0.0:9009->80/tcp, 0.0.0.0:52351->443/tcp   ksqldbdemo
0a0186712553   confluentinc/ksqldb-cli:0.22.0      "/bin/sh"                2 hours ago   Up 2 hours
  ksqldb-cli
76519de6946e   obsidiandynamics/kafdrop:latest     "/kafdrop.sh"            2 hours ago   Up 2 hours   0.0.0.0:19000->9000/tcp
  kafdrop
11c3a306ee01   confluentinc/ksqldb-server:0.22.0   "/usr/bin/docker/run"    2 hours ago   Up 2 hours   0.0.0.0:8088->8088/tcp
  ksqldb-server
07cef9d69267   confluentinc/cp-kafka:7.0.0         "/etc/confluent/dock…"   2 hours ago   Up 2 hours   9092/tcp, 0.0.0.0:29092->29092/tcp
  broker
3fa1b9a60954   confluentinc/cp-zookeeper:7.0.0     "/etc/confluent/dock…"   2 hours ago   Up 2 hours   2888/tcp, 0.0.0.0:2181->2181/tcp, 3888/tcp     zookeeper

WEB API

Now, with all services up and running, we can access the WEB API Swagger to populate our Kafka topics. The code is very simple and it's avaliable in the repository.

Then you can create an User with the user name of your choise, as shown:

And it will have an assigned unique Id, as demonstrated:

Now, you can get a three digit code for your user Id as displayed:

And a random code is generated for the selectd, as we can observe in the image that follows:

Kafdrop

We can use the kafdrop UI the check if everything is okay. Kafdrop is deployed at http://localhost:19000/.
There, you will find all the brokers and topics avaliable. It should look like this:

KSQL CLI

After all that, you'll be able to create your streams of data and query it using ksqlDB. On your preferential terminal, use the command:

docker exec -it ksqldb-cli ksql http://ksqldb-server:8088

Creating streams

And then you are in the ksql CLI and are free to create your streams and queries. First, let's create a stream for each one of our topics:

CREATE STREAM stream_user (Name VARCHAR, Id VARCHAR)
  WITH (kafka_topic='demo-user', value_format='json', partitions=1);

CREATE STREAM stream_code (Id VARCHAR, code INT)
  WITH (kafka_topic='demo-code', value_format='json', partitions=1);

Create a materialized view

You can join the client data with the most recent randomized code. to achieve this, you must create a materialized view table, that joins both streams as seen in the ksqldb script that follows:

CREATE TABLE currentCodeView AS
>   SELECT user.Name,
>   LATEST_BY_OFFSET(code.code) AS CurrentCode
>   FROM stream_code code INNER JOIN stream_user user
>   WITHIN 7 DAYS ON code.Id = user.Id
>   GROUP BY user.Name
>EMIT CHANGES;

Making a push query

After that, we can query this materialized view:

SELECT * FROM currentCodeView 
  EMIT CHANGES;