DEV Community: Thiago Nascimento Figueiredo

A brief example about message communication with Kafka

Thiago Nascimento Figueiredo — Fri, 20 Jan 2023 01:24:24 +0000

This article came up as an idea when I was reviewing some content I already had studied and I was working with. Since I needed to study a few more issues related to Kafka usage into application solutions I got the idea to share a few information related to that. It is a very entry level article. I'm not a Kafka specialist, but I already worked with solutions that needed to use asynchronous communication model and message systems. Based on this first information I'll try to approach the subject covering the following items:

Bring some considerations when using synchronous and asynchronous scenarios in solutions;
Sample of Kafka usage in the context of an asynchronous communication scenario (producer and consumer basics implementation).

The idea is not to explore the trade-offs over Kafka as technology or platform. So, there will be no deep dive through this direction. On this AWS link about Amazon Managed Streaming for Apache Kafka there are a few comparisons when looking to Kafka and RabbitMQ. This can give an idea about the differences when Kafka is compared with tools for messaging solutions.

Synchronous and asynchronous communication

Over those years I have worked with IT it is possible to see that it is kind common see situations where developers have some difficulty distinguish when to use synchronous and asynchronous communication over solution scenarios. Most of the times it is possible to perceive some considerations missing in the evaluation of the requirements to design the solution to be implemented, or a mistaken evaluation over which could be a proper approach to be applied to some problem. It is very helpful the understanding of the main characteristics of both approaches when doing this evaluation.

Synchronous vs. Asynchronous integration

In a summary, the main point that distinguish the synchronous communication from an asynchronous communication will be the need of receiving a response or not to keep processing something. When the response is needed for keep processing something, the synchronous communication scenario will be most of the times a better approach. When in a communication scenario it is possible or necessary to detach the producer behavior from the consumer behavior in such a way that a response is not needed to keep processing something, almost most of the times the asynchronous approach will fits better.

It is good to enforce that there is no silver bullet when we discuss solution design. The choices come with trade-offs to be analysed where the chosen option has good points and problems to be handled, and also there might happen some requirements that can invalidate some solution options. For example, it is possible to work with APIs for an asynchronous communication design if your corporation works only with API driven approach. The service gives a response to the request with a HTTP status code 202 to mention that something will be processed later. And them some callback endpoint can be called later when the processing of this request finishes. But this approach has its trade-offs.

When considering Kafka as a component in your solution it is possible to say that most of the cases its usage will fit into an Enterprise Integration Pattern related to Messaging Patterns. With Kafka it combines two messaging models (queuing and publish-subscribe) to provide the key benefits of each to consumers. More details about this can be found on this AWS link about Amazon Managed Streaming for Apache Kafka.

Producer and Consumer example using Kafka

Apache Kafka is a real-time data streaming technology capable of handling trillions of events per day. It is commonly used to build real-time streaming data pipelines and event-driven applications. It provides distributed, high-throughput, low-latency, fault-tolerant platform for handling real-time data feeds - known as events. More details can be found in the Apache Kafka official page. The main idea here is to show a sample of Kafka usage to send and consume messages for asynchronous communication.

To illustrate a producer and an consumer it was created 2 Kotlin applications using Kafka as samples that consumes Tweets based on some parameters and save them in a ElasticSearh repository to allow search of the Tweets content. In a real world application it can be used to track subjects or themes that are trend topics for any specific reason. More details about the implementation itself can be checked here.

In this sample it was not deeply explored the set of consumer configurations or producer configurations. The main idea is to highlight a simple implementation to produce messages/events to Kafka and consume them. The source of the information are the filtered Twitters. All the components were linked to a docker-compose structure to make it easier to handle the sample.

In the applications it was created configuration components for having all the Kafka configuration for consumer and producer.

Consumer configuration:

   @Bean
    fun consumerFactory(): ConsumerFactory<String?, Any?> {
        val props: MutableMap<String, Any> = HashMap()
        props[ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG] = servers
        props[ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG] = StringDeserializer::class.java
        props[ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG] = StringDeserializer::class.java
        props[ConsumerConfig.AUTO_OFFSET_RESET_CONFIG] = "earliest"
        return DefaultKafkaConsumerFactory(props)
    }

    @Bean
    fun kafkaListenerContainerFactory(): ConcurrentKafkaListenerContainerFactory<String, Any>? {
        val factory = ConcurrentKafkaListenerContainerFactory<String, Any>()
        factory.consumerFactory = consumerFactory()
        factory.containerProperties.ackMode = ContainerProperties.AckMode.MANUAL_IMMEDIATE
        factory.containerProperties.isSyncCommits = true;
        return factory
    }

Producer configuration:

    @Bean
    fun producerFactory(): ProducerFactory<String, Any> {
        val configProps: MutableMap<String, Any> = HashMap()
        configProps[ProducerConfig.BOOTSTRAP_SERVERS_CONFIG] = servers
        configProps[ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG] = true
        configProps[ProducerConfig.ACKS_CONFIG] = "all"
        configProps[ProducerConfig.COMPRESSION_TYPE_CONFIG] = "snappy"
        configProps[ProducerConfig.LINGER_MS_CONFIG] = 5
        configProps[ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG] = StringSerializer::class.java
        configProps[ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG] = StringSerializer::class.java
        return DefaultKafkaProducerFactory(configProps)
    }

    @Bean
    fun kafkaTemplate(): KafkaTemplate<String, Any> {
        return KafkaTemplate(producerFactory())
    }

Since the tweet message is a big JSon object I didn't try to map it to any object. It was serialized as a JSon String on both sides of the communication. The details about the tweets consumption is in the mentioned GitHub repository. After reading it, it is published to the Kafka topic:

@Component
class KafkaTweetsProducer(
    private val kafkaTemplate: KafkaTemplate<String, Any>
) {

    companion object {
        private val LOGGER = LoggerFactory.getLogger(KafkaTweetsProducer::class.java)
    }

    fun send(key: String, tweet: String) {
        LOGGER.info("Tweet message: {}", tweet)
        kafkaTemplate.send("tweets-message-topic", key, tweet)
    }

}

The consumer reads the tweet in the same way as it arrives from the Kafka topic to save it to the ElasticSearch repository:

@Component
class KafkaTweetsConsumer(
    private val elasticSearchClient: RestHighLevelClient,
    @Value("\${elasticsearch.index}")
    private val tweeterIndex: String
) {

    companion object {
        private val LOGGER = LoggerFactory.getLogger(KafkaTweetsConsumer::class.java)
    }

    @KafkaListener(topics = ["tweets-message-topic"], groupId = "simple-kotlin-tweets-message-consumer")
    fun consume(tweet: String) {
        LOGGER.info("got tweet: {}", tweet)
        val indexRequest = IndexRequest(tweeterIndex).source(tweet, XContentType.JSON)
        val indexResponse = elasticSearchClient.index(indexRequest, RequestOptions.DEFAULT)
        LOGGER.info("ElasticSearch id: {}", indexResponse.id)
    }
}

The README file from the repository has instructions to configure and run the sample. The result of the communication about the producer and the consumer can be seen in a Kibana interface:

Final considerations

Kafka as technology and tool has a lot of real powerful use-cases. It is a technology worthy to explore for microservices approach, event driven architecture, asynchronous communication, real time processing, streaming data, and several other scenarios. I hope this article can be useful for an entry level reading about it.

References:

Asynchronous Communication; The What, The Why, And The How - Sandeep Kashyap
Apache Kafka - Fundamentals, Use cases and Trade-Offs - Dhruvesh Patel
Apache Kafka Documentation - Apache Kafka
What is Kafka? - Confluent
What is Kafka? - AWS
Managing Asynchronous Workflows with a REST API - AWS

Web APIs - Different approaches and how to choose

Thiago Nascimento Figueiredo — Fri, 21 May 2021 14:17:58 +0000

Web APIs are one of the most common resources used for design several solutions related to integration scenarios, microservice applications, and other very common solutions that are built on top of the HTTP protocol.

If we look to WEB APIs as a solution alternative, it is possible to see that it became an interesting option because it works over an interoperable protocol widely used. This option mitigates several problems that we might face when building the system's integration and features. Since we are talking about APIs, it is important to understand some principles that we need to use to guide a solution approach: APIs – Application program interface – are most commonly expressed as a set of operations, associated data definitions, and the semantics of the operations on some underlying system (David Emery - Association for Computing Machinery).

There are other similar definitions, but taking this one as a basis we can see that semantic is something very important when designing an API. The API semantics depends on the WEB API Design approach, its usage of HTTP verbs, definitions about application operations, and other elements. And in several moments development teams will question themselves which approach they need to use. Those choices come to the trade-off analysis. The best tool to guide this analysis is to understand better the WEB APIs Design approaches and check which one fits better to your problem requirements.

Web API Design approaches

When evaluating possible Web API design approaches it will fit into a few possible options: RPC, REST, or a "query language" API style (GraphQL). Basically, those are concepts of how to design your API that follow specifications that are often drafted up by various working groups. For example: SOAP is a W3C recommendation, gRPC has as authors Google Inc., REST is a paradigm with no organization or group responsible for it.

By checking on the characteristics of each design style it will be possible to start evaluating which approach matches your problem. When looking at the RPC style, it will depend on the specific API implementation. REST and GraphQL have more parameters related to the standards and paradigms.

RPC

RPC APIs are about executing a block of code on another server, that when implemented in HTTP or AMQP can become a Web API. It is possible to say that:

They are action-based;
The API specification depends on the implementation;
Can be stateless or stateful;
No specific semantic definition:
- No HTTP verbs best practice recommendation;
- No HTTP status code usage recommendation;
- Best practices recommendations depend on the chosen RPC API.

The development of an RPC API is similar to create programming libraries. The name of the actions are in the URI and they can be compared to a function invocation. Definitions of parameters and returns according to the operation needs in the query string or body, and modern implementations can make simple interactions with high performance. There is no discoverability (How to start? What to call?). In general, can become hard to understand and maintain due to the tendency to create new endpoints when new functions are needed. This creates a tight coupling solution.

REST

REST is a paradigm described by Roy Fielding in a dissertation in 2000. It describes a client-server relationship where server-side data are made available through representations of data in simple formats. To be considered a RESTful API it is necessary to fit into the restrictions described in this dissertation. It uses as most common formats JSON or XML but could be anything. When describing a RESTful API we can say that:

It is resource-based;
Nouns of the resources are used to define URIs;
It is stateless;
There is a strict semantic definition based on HTTP:
- It has HTTP verbs and status code semantic usage;
- Resources and their relationships represented in the API’s URIs;
- Content negotiation for different message formats;
Service URIs are easily readable due to their strict semantic;
Hypermedia allows actions and relationships to be made discoverable.

The RESTful APIs are considered good options when there are issues involved like common workflows to be followed, caching of resource information, not a very reduced payload restriction, content negotiation necessary to provide a response to the clients, etc.

GraphQL and "Query language" APIs

GraphQL was built by Facebook and it is a data-driven API design approach that provides a query language for APIs and a runtime for fulfilling those queries with your existing data. GraphQL can be essentially considered an RPC API style with a default procedure, with a lot of good ideas from the REST/HTTP community. It is possible to mention that its main characteristics are:

Stateless;
It is based on entity graphs and entities are not identified by URIs;
Custom data recovery:
- Ask for specific resources and specific fields;
- Operation parameters and returns parametrized;
Reduced number of HTTP requests necessary to retrieve data for multiple resources;
Low network overhead.

How to choose?

It is important to look into the requirements that the application needs to meet so it becomes possible to look at the different API design approaches to find which one fits best into the problem to be solved. Based on the highlighted issues of each API style that were mentioned it is possible to illustrate some scenarios.

Among the different approaches that were presented, when evaluating the options through RPC APIs it will depend on the specific implementation option. For example, gRPC can be used in scenarios where a system requires a set amount of data or processing routinely, and in which the requester is either low power or resource-jealous (IoT is a great example of this). Or scenarios like SOAP interaction where is needed asynchronous processing and invocation, formal contracts, and stateful operations.

One good scenario to explain where REST fits is when we bring the reality of microservice applications, where different bounded contexts must be decoupled. Things within a context can treat their own APIs in private and it is possible to do changes whenever needed without affect other domains. It is even possible to use an RPC approach when pieces inside a bound context need to communicate internally. But when there is communication among different contexts it is important to be loosely coupled. Another scenario is when this interaction between client and server requires a workflow among a set of well-defined resources (for example, a payment workflow).

Proposing a scenario where GraphQL would be a good fit, we can consider an interaction among the client of an API where to recover information needed will demand several requests or iterations to achieve the final purpose. For example, lest suppose there is the need to call “/users/” endpoint to fetch the initial user data, then call a “/users//posts” endpoint to return all the posts for a user, and then call a “/users//followers” to return a list of followers per user.

When evaluating this interaction through the perspective of a data-driven API this information could be recovered into a single interaction. It is possible to be considered also in a scenario where the relationship between the resources is hard to be handled through a set of interactions into RESTful resources. Or also when there is needed to recover a customized set of information involving fields of different resources to compose a single response.

One thing that is important to have in mind is that there’s no silver bullet. Tackle the problem with the tool that handles it according to its characteristics. Sometimes there will be more than one possible option, but with different tradeoffs that the choice will bring.

References

Standards, APIs, Interfaces and Bindings

David Emery - Association for Computing Machinery: http://oldwww.acm.org/tsc/apis.html

Understanding RPC, REST and GraphQL

Phil Sturgeon: https://apisyouwonthate.com/blog/understanding-rpc-rest-and-graphql

Picking the right API Paradigm

Phil Sturgeon: https://apisyouwonthate.com/blog/picking-the-right-api-paradigm

GraphQL — The Query language for APIs

Arun Rajeevan: https://arunrajeevan.medium.com/graphql-the-query-language-for-apis-4e79ae303100

When to Use What: REST, GraphQL, Webhooks, & gRPC

Kristopher Sandoval: https://nordicapis.com/when-to-use-what-rest-graphql-webhooks-grpc/

SOAP vs REST 101: Understand The Differences

SoapUI: https://www.soapui.org/learn/api/soap-vs-rest-api/

Logs - Why, good practices, and recommendations

Thiago Nascimento Figueiredo — Wed, 14 Apr 2021 11:05:19 +0000

Why logging is important?

Logging information is important to understand the behavior of the application and to understand problems in several different scenarios. Into the application lifetime, it will eventually crash, a server will go down, users may complain about a bug that “randomly” appears, or the client could realize that some data is missing, and there is no clue when or why this data got lost. Those are some examples of scenarios where there is a need to understand the application behavior through the development application lifecycle, track problems during QA validations and incident scenarios, and other possible situations. That is why the application solution logs must be designed, implemented, and tested.

Considering those issues related to the importance of logs it is necessary to define for the application what, how, and when to log and even how to extract meaningful data from logs. This brings the need to define a structure for log messages. The need to define a log structure is to extract the information needed (most of the time with the support of tools).

Another issue that is worthy to mention is that log messages need to be sent at least in every layer of the architecture with meaningful and enough contextual information. Having at least one log entry per request/result in every layer of the application allows providing accurate context about what the user was doing when a specific error or situation happened.

Once it is defined the structure of the log messages, the next step is to have clear what to log and how to log. The log messages for the application need to inform about errors (to use them for troubleshooting purposes), but also useful information about successful requests to have a clear idea of how users work with the application. In this subject, we get through the usage of log level messages related to the purpose of the information we want to inform.

Logging levels

The usage of a good logging strategy needs to take into consideration the proper usage of log levels for its messages. This strategy related to log level messages is essential to avoid “noise” when looking for information and track possible evidences to evaluate a problem or analyze the application behavior. The main cons about this are that when a problem comes up, you are likely to proper contextual information to be analyzed.

There will be frameworks to help handle logging messages for the different programming languages and the definition among the log levels can slightly vary among the framework implementations. But they follow the same semantic purpose. The following example shows this for a few Java frameworks possible to be used.

The log level information has well-defined usage recommendations for each situation:

Good practice recommendations

According to the presented reasons over the application logging practices, there are a few good practices and recommendations to be followed to provide log messages that are meaningful and useful.

General good practices

Set the current log level via external configuration: the modification of the current log level messages should always be an operational task done in runtime. This allows fast action to problem troubleshooting and situation analysis without stopping the application.
Default log level information configuration by environments: For the production environment it is interesting to have the default log level as “Information”. This will allow that Fatal, Error, Warning, and Information log entries will be written. while Debug and Trace log entries will be suppressed. Environments that need more detailed information can have default log level set to debug or trace.
The proper log level is important to avoid noise: log application information with the proper log level avoids mistaken interpretation between data that represent problems and data that represents application behavior:
- Use the Warning and Information Log Levels properly: Warning should be for logs that are not errors, but are not normal behavior. Information should be for application normal execution when something important is to be communicated.
- Log catastrophic failures using Critical/Fatal: when there is an unrecoverable error during the application start-up or execution log it using the Critical log level. Tools can use this log level for emitting alerts.
- Step through code using the Debug log level: When your application is misbehaving it is necessary to get enhanced visibility into what is going on. The good use of the Debug log level can get you more detailed about what is occurring.
- Inspect variables using the Trace log level: The use of trace log level to write the values of variables, parameters, and settings to the logs for inspection can help you when the debug log level isn’t enough. This mimics the behavior of inspecting the contents of variables with the debugger attached.
Never log personal identifying Information or secrets: It is important because of GDPR, CCPA, and other privacy laws and for security issues.

Events log good practices

Use clear key-value pairs: tolls extract fields from events when you search, creating structure out of unstructured data.
Create events that humans can read: complex encoding that would require lookups to make event information intelligible.
Use timestamps for every event: The correct time is critical to understanding the proper sequence of events, for debugging, analytics, and deriving transactions.
Use correlation IDs for tracking operations: Correlation ID (also known as a Transit ID) is a unique identifier value that is attached to requests and messages that allow reference to a particular transaction or event chain. They are helpful when debugging and tracking transactions through the system and follow them across machines, networks, and services.
Avoid logging binary information: It is not good for reading what is happening through the log events and also does not work well when tools are indexing the information.
Use data structured formats: They are readable by humans and machines and can be easily parsed by most programming languages right in your browser. This helps tools and log search when evaluating log events.
Identify the source (class, function, or filename): Useful to understand where the problem is.

Operational good practices

Log locally to files: If you log to a local file, it provides a local buffer and you aren't blocked if the network goes down.
Use tools for streaming and monitoring logs: Tools can collect logging data and then send this information to the indexers. They will be able to work well among the large amount of information generated by the logging activity.
Use rotation and retention policies: Logs can take up a lot of space. Good rotation strategies can help to decide when to destroy or back up your logs (if needed). It is especially essential considering scenarios for cloud solutions where storage price is important.
Collect events from all possible sources: The more data you capture, the more visibility you have. Some example of possible sources:
- Application logs;
- Database logs;
- Network logs;
- Configuration files;
- Performance data.

Tools for dealing with log information

Without tools for dealing with log information, the amount of data created by logging activity can be meaningless. Besides that, in a scenario where there are several instances of distributed applications and several services that those applications integrate, those tools are helpful to track information that comes from different sources. Each of those sources formats and stores logs in their own way, making it really difficult to find useful data.

The logging tools that we have available do streaming over log messages and centralize its access, allowing log search and aggregation to turn all this generated data into useful information. There are several examples of logging tools nowadays, but here follow the example of the structure of two of them. The first image is from an example using splunk, and the following image is an example using the Elastic stack:

Those tools get the log messages through collectors and store the information into repositories where the information is indexed. When the collectors get the information, they analyze the log structure to process its data before storage. Once the log messages are stored and indexed it is possible to search information, create charts and create alerts through the interface of those tools. Here follow a few examples of the mentioned log searching UI. The presented UIs are from the tools Splunk, Graylog, and Kibana.

Splunk:

Graylog:

Kibana:

As it is possible to see, those tools provide very interesting features related to logging activity. That brings a good reason to work with the messages following a well-defined structure, good practices, and recommendations. It makes the log information possible to be processed, tracked, and handled by the referred tools so they become useful.

References

DEV Community: Thiago Nascimento Figueiredo

A brief example about message communication with Kafka

Synchronous and asynchronous communication

Synchronous vs. Asynchronous integration

Producer and Consumer example using Kafka

Final considerations

References:

Web APIs - Different approaches and how to choose

Web API Design approaches

RPC

REST

GraphQL and "Query language" APIs

How to choose?

References

Logs - Why, good practices, and recommendations

Why logging is important?

Logging levels

Good practice recommendations

General good practices

Events log good practices

Operational good practices

Tools for dealing with log information

References

Logging Best Practices:

Logging best practices in an app or add-on for Splunk Enterprise:

The Importance of logging: introducing Elastic Stack

9 Logging Best Practices Based on Hands-on Experience