DEV Community: mafflerbach

Maintaining and Supporting Over 50 Services with Java, Apache Camel, and ActiveMQ/Kafka.

mafflerbach — Sun, 07 May 2023 19:12:02 +0000

Maintaining and supporting over 50 services can be quite a challenging task, but with the right tools and processes in place, it can become a manageable task. In this blog post, we will share some of our general experiences on how we maintain and support these services.

We are running our services in java using the apache camel framework and using activeMq or kafka as a message bus system. All of our services implement the Dead Letter Queue (DLQ) pattern, which is implemented on the framework level. The modules of our services are strictly divided into filter, enrich, transform or send.

One of the daily tasks of our support team is to check if messages are failing and then contacting the stakeholder to determine whether we should dump and forget the message or replay it. It can get tedious to communicate with the creator or receiver of the message and wait for their response, as we are just a "man in the middle" and do not have any ownership over the content of the message.

However, our centralized logging and monitoring system, consisting of log4j and fluentd, has proven to be a useful tool. The logs are accessible via openSearch and allow us to quickly identify any issues. Monitoring is handled by Grafana and Prometheus as a data source, while Opsgeni serves as our alerting system. With all these tools in place, we are quickly alerted whenever something goes wrong, allowing us to address it promptly.

Automated testing is also crucial in maintaining our services. We rely on integration tests based on jbehave, which helps us to detect issues early in the development cycle. We have also implemented a standard failure handling mechanism that is implemented at the framework level. This mechanism handles error messages and puts them on the right dead letter queue while also implementing an automated retry mechanism.

In conclusion, maintaining and supporting over 50 services can be quite challenging, but with the right tools, processes, and mindset in place, it becomes a manageable task. Good communication, automated testing, a robust monitoring system, and a profound failure handling mechanism are all essential aspects of maintaining these services. By leveraging these tools and processes, we ensure that our services remain reliable, performant, and meet our stakeholder's needs.

How to Improve Service Resilience and Reliability with Error Monitoring and Logging

mafflerbach — Fri, 28 Apr 2023 16:56:35 +0000

How to Improve Service Resilience and Reliability with Error Monitoring and Logging

When building complex applications on Kubernetes, it's important to ensure that your services are reliable and resilient. One way to do this is by implementing error monitoring and logging, which can help you quickly detect and diagnose issues, as well as identify potential areas for improvement. In this blog post, we'll explore some best practices for error monitoring and logging in a Kubernetes environment with Java-based services.

Why Error Monitoring and Logging is Important

Error monitoring and logging provides several benefits, including:

Improved Service Reliability: By monitoring your services for errors and logging relevant information, you can quickly identify and resolve issues before they become major problems.
Faster Debugging and Troubleshooting: When an issue does occur, having detailed logs and error messages can help you quickly identify the root cause of the problem and develop a solution.
Better Service Performance: By analyzing your logs and monitoring metrics, you can identify areas for improvement and optimize your services for better performance.

Best Practices for Error Monitoring and Logging in Kubernetes

To implement error monitoring and logging in a Kubernetes environment with Java-based services, here are some best practices to follow:

1. Use a Centralized Logging Solution

With multiple services running on a Kubernetes cluster, it can be challenging to monitor logs and error messages across all services. That's why it's important to use a centralized logging solution, such as Elasticsearch or OpenSearch, which allows you to collect, analyze, and search logs from all of your services in one place.

In this setup, you can use Elastic File System (EFS) to store logs as files, which can then be collected by a log aggregation agent like Fluentd. Fluentd can then forward these logs to your Elasticsearch or OpenSearch instance for indexing and searching.

2. Monitor Kubernetes Metrics

In addition to monitoring logs and error messages, it's also important to monitor Kubernetes metrics, such as CPU and memory usage, network traffic, and application performance. This can help you identify potential issues and optimize your services for better performance.

To monitor Kubernetes metrics, you can use Prometheus and Grafana. Prometheus collects metrics from your Kubernetes cluster and services, while Grafana provides a user-friendly dashboard for visualizing and analyzing these metrics.

3. Set Up Alerts

Monitoring logs and metrics is important, but it's even more important to set up alerts for critical errors and issues. This allows you to be notified immediately when an issue occurs, so you can quickly respond and prevent any potential downtime or performance degradation.

For this, you can use a tool like Opsgenie, which provides intelligent alerting based on the severity and type of issue. You can configure Opsgenie to send notifications to specific individuals or groups via email, SMS, or phone call.

4. Use Breadcrumb Logging

Finally, it's important to implement breadcrumb logging, which involves adding small pieces of information to your logs at key points in your application. This can help you identify the flow of your application and the context in which errors occur, which can be especially useful when trying to troubleshoot complex issues.

To implement breadcrumb logging in your Java-based services, you can use a logging library like Logback or Log4j2, which provide support for adding context information to your logs.

Conclusion

By following these best practices for error monitoring and logging in a Kubernetes environment with Java-based services, you can improve the reliability and resilience of your services, as well as optimize their performance. With centralized logging, Kubernetes metrics monitoring, alerts, and breadcrumb logging, you can quickly detect and diagnose issues, and ensure that your services are performing at their best.

PlantUML and Jira: Combining Forces to Simplify Gantt Chart Creation

mafflerbach — Fri, 21 Apr 2023 15:56:35 +0000

Are you tired of manually creating Gantt charts to track your project's progress? Look no further than PlantUML and Jira! These two powerful tools can combine forces to simplify the process and save you time.

To get started, you'll need to extract the Jira tickets related to an Epic. Don't worry, it's easier than it sounds. Simply navigate to your Epic in Jira and select the "Export" option. From there, choose "Excel (all fields)" and save the file to your computer.

Now that you have your Jira tickets, it's time to create your PlantUML file for the Gantt chart. If you're not familiar with PlantUML, it's a tool that allows you to create diagrams using a simple text syntax. To create a Gantt chart, you'll need to use the "gantt" keyword followed by the tasks and their start and end dates.

To incorporate your Jira tickets, you can add them as tasks in your PlantUML file using the Jira ticket number as the task name. You can also include additional information such as the task assignee and due date.

Here is an example of what your PlantUML code might look like:

@startgantt
Project starts the 2023-04-21
Project starts the 21th of april 2023


[Jira Ticket-1] starts 2023-04-21 and  lasts  3 days 
[Jira Ticket-2] starts 2023-04-24 and  lasts  2 days 
[Jira Ticket-3] starts 2023-04-26 and  lasts  1 days 
[Jira Ticket-4] starts 2023-04-27 and  lasts  4 days 
[Jira Ticket-5] starts 2023-05-03 and  lasts  2 days 
[Jira Ticket-6] starts 2023-05-05 and  lasts  1 days 
[Jira Ticket-7] starts 2023-05-06 and  lasts  3 days 
[Jira Ticket-8] starts 2023-05-10 and  lasts  2 days 
[Jira Ticket-9] starts 2023-05-12 and  lasts  3 days 
[Jira Ticket-10] starts 2023-05-15 and lasts 1 days 


@endgantt

For generating a PNG you just call

java -jar plantuml.jar path/to/your/file -tpng -o path/for/outpu/

Once you have your PlantUML file set up, it's time to generate the Gantt chart. There are several ways to do this, but one popular option is to use the PlantUML extension for Visual Studio Code. Simply open your PlantUML file in Visual Studio Code and use the extension to generate the Gantt chart.

Of course, there are other ways to connect the dependencies between the tickets and add more information. You can find here more documentation, and perhaps other ideas for creating a quick timeline for your project.

And there you have it, a Gantt chart created from your Jira tickets using PlantUML! Not only does this simplify the process, but it also ensures that your Gantt chart is up-to-date with your project's progress.

So next time you need to create a Gantt chart, consider using PlantUML and Jira. It may not be as exciting as a superhero team-up, but it will certainly make your project management tasks more efficient.

In conclusion, by combining PlantUML and Jira, you can create Gantt charts quickly and easily. With just a few simple steps, you'll have a comprehensive overview of your project's progress. So why not give it a try and see how much time you can save?

DLQ Integration Patterns: A high-level Comparison of Apache Camel and Spring Boot

mafflerbach — Fri, 14 Apr 2023 08:12:35 +0000

This article provides only a high-level overview of the DLQ integration patterns in Apache Camel and Spring Boot. There is much more to explore and learn about these integration frameworks and their capabilities. The DLQ integration pattern is just the tip of the iceberg when it comes to the features and tools provided by Apache Camel and Spring Boot. Nevertheless, this article aims to provide an introduction to the DLQ integration pattern and to highlight the strengths and weaknesses of these frameworks in this context.

The use of Dead Letter Queues (DLQs) in messaging systems is, in my opinion, one of the best patterns for implementing resilient and decoupled systems, providing a way to handle messages that could not be successfully processed. DLQs are used to store messages that have failed processing, allowing the message to be retried later or manually inspected by a developer.

Integration frameworks such as Apache Camel and Spring Boot provide support for implementing DLQ integration patterns. In this article, we will compare the DLQ integration patterns provided by Apache Camel and Spring Boot and discuss their advantages and disadvantages.

About Apache Camel

DLQ integration patterns in Apache Camel

Apache Camel is a powerful open source integration framework that provides support for various integration patterns, including DLQ integration. Camel allows developers to configure a separate channel for messages that could not be processed. The failed message is sent to the DLC channel where it can be retried or manually inspected. Camel provides a wide range of components that allow developers to integrate with different systems and protocols, making it a solid choice for enterprise-level integration.

Benefits of DLQ Integration in Apache Camel

Apache Camel provides a wide range of components that allow developers to integrate with various systems and protocols.
Apache Camel has a large and active community that provides support and resources for developers.

Disadvantages of DLQ integration in Apache Camel

Configuring the DLC pattern in Camel can be complex and requires a good understanding of the framework.
Developers may need to write custom code to implement the DLQ integration pattern in Camel.
The documentation and usage of Camel can be limited compared to other frameworks, which can make it more difficult for developers to learn and use effectively.

About Spring Boot

Spring Boot is a popular open source framework for building enterprise-level applications. Spring Boot provides a set of features and tools that enable developers to quickly build and deploy applications. Spring Boot also provides support for integration with various messaging systems, making it a popular choice for implementing DLQ integration patterns.

DLQ Integration Patterns in Spring Boot

Spring Boot provides support for the DLQ integration pattern through the use of the Error Channel pattern. The Error Channel pattern allows developers to configure a separate channel for messages that could not be processed. The failed message is sent to the error channel where it can be retried or manually inspected.

Benefits of DLQ integration in Spring Boot

Spring Boot provides a set of features and tools that enable developers to quickly build and deploy applications.
The error channel pattern in Spring Boot provides a simple and easy-to-use way to handle failed messages.
Spring Boot has a large and active community that provides support and resources for developers.

Disadvantages of DLQ integration in Spring Boot

Spring Boot provides limited support for integration with certain messaging systems, which can be a limitation in some cases.
The error channel pattern in Spring Boot may not be as flexible as the DLC pattern in Apache Camel, which may limit some use cases.

Conclusion

Both Apache Camel and Spring Boot provide support for implementing DLQ integration patterns. Apache Camel provides a wide range of components that allow developers to integrate with different systems and protocols, making it a solid choice for enterprise-level integration. Spring Boot provides a set of features and tools that enable developers to quickly build and deploy applications, making it a popular choice for implementation.

Keeping Your Services Running Smoothly: A Guide to Java and Kubernetes Maintenance

mafflerbach — Fri, 07 Apr 2023 13:32:48 +0000

Welcome to the world of Java and Kubernetes maintenance, where the joy of coding meets the agony of production! Keeping your services running smoothly is no easy feat, especially when dealing with complex integration patterns like the Dead Letter Queue. But fear not, my fellow developer, for I am here to guide you through the treacherous waters of maintenance and help you avoid the pitfalls that come with it.

Let’s start with Apache Camel, the integration framework that has been making waves in the developer community for quite some time now. With its extensive library of components, Camel is the go-to choice for many developers when it comes to building complex integration solutions. However, with great power comes great responsibility, and Camel is no exception to this rule.

When working with Camel, one of the most common patterns that developers come across is the Dead Letter Queue (DLQ). For those of you who are unfamiliar with this pattern, it is used to handle messages that could not be delivered to their destination for various reasons. These reasons could range from network issues to incorrect message formats, and they can wreak havoc on your system if not handled properly.

To implement the DLQ pattern, you first need to create a separate queue where all the undeliverable messages will be sent. This queue is called the Dead Letter Queue, and it should be monitored regularly to ensure that no messages are stuck there. To achieve this, you can set up an alert system that notifies you whenever a message is added to the DLQ.

Now, let’s move on to the old stack, which consists of Spring and ActiveMQ. While this stack has been a reliable choice for many years, it’s time to move on and embrace the new stack based on Spring Boot and Kafka. Why, you ask? Well, for starters, Spring Boot provides a more streamlined development experience and allows you to focus on writing code rather than configuring the environment. And Kafka, on the other hand, is a highly scalable and fault-tolerant messaging system that can handle millions of messages per second.

Migrating from the old stack to the new stack may seem daunting at first, but fear not, for it is not as difficult as it seems. You can start by creating a new project based on Spring Boot and Kafka and gradually migrate your code from the old project to the new one. This way, you can keep your existing codebase intact while gradually moving towards the new stack.

But before you start migrating, make sure to thoroughly test your code to ensure that it works as expected. This is especially important when dealing with integration patterns like the DLQ, as any mistakes can have severe consequences. You can use tools like JUnit and Mockito to test your code and catch any errors before they make their way to production.

In conclusion, maintaining Java and Kubernetes services can be a daunting task, but with the right approach, it can also be a rewarding one. When working with integration frameworks like Apache Camel, make sure to follow best practices and pay special attention to patterns like the Dead Letter Queue. And when migrating from the old stack to the new stack, take it slow and test your code thoroughly to avoid any unwanted surprises. And remember, a touch of humor can make even the most challenging tasks more enjoyable. So, keep calm and code on!

Fugitive - vim and git

mafflerbach — Sat, 16 Jan 2021 17:30:50 +0000

vim-fugitive is the go-to git extension for VIM. I think there is no other git extension which is working so well.
Fugitive is not only a simple git wrapper, but it can act like, if its needed.

The most used mapping in my case are:


noremap <leader>ge :Gblame<CR>
noremap <leader>gs :Gstatus<CR>
noremap <leader>gc :Gcommit<CR>
noremap <leader>gl :Gpull<CR>
noremap <leader>gh :Gpush<CR>

With Gstatus you get a git status of your current repository.

If you want to stage a file. Use s or - for staging the file.
You can also stage all your file at once, if you are using s or - on the header Unstage.

If you want to unstage, us u on your item, or on the Stage header.

If you want to see a diff, you go on the line of your changed file and hit = or i.

If you want to open the file, hit Enter. The status buffer will stays, and the selected file will be open in a different buffer.

With Gcommit a additional split for your commit message will open. With save and close, the commit will be executed.

With Gpush your commit will be pushed to your remote.

In the case you have to revert you local changes, with Gread it will be performing a git checkout -- %

If you want to take a look on your git logs? No problem, fungitive is covering that too.
It will open your git log in a quickfix list, and it shows you what happened in previous commits.

How to build a Keyboard

mafflerbach — Sat, 09 Jan 2021 11:57:57 +0000

You believe it, or not - building a Keyboard is simple.
Building a split Keyboard is a little bit more complex, but if you understand the basics, it's still possible.
In this series, we are building a split mechanical keyboard with led lights. We will setup our firmware which is based on QMK, which is the most famous and customizable firmware for DIY keyboards. And we will create our keymaps and defining our layers on our will.

I build already a split keyboard and it was a fun experience, and it is my daily driver. But because it is fun to build one, I want to build my next keyboard, which will complete different in keys key positioning and look and feel.

If I put my both half together, you will see, it is a standard US 60% keyboard. I have two additional layer for my F keys, and audio and light control as well.

But to build a Keyboard is not cheap.
This build has a price tag of something around 130 Euros (~159 USD), and that is just summarize the part cost. The hours of work is not included.

Of course its easier, and less time consuming and cheaper, to just buy a keyboard. And in comparison to a massmarket mechanical keyboard ,it is a expensive Keyboard.
But I enjoy to tinker, I love the process of making things and the outcome will be a keyboard which is fitting my needs.

This time I decide to build a keyboard which is quite different to a normal staged layout. This layout called columnar staged it will be similar to an redox. I want to use this templates from thingivers.

The top and bottom plate will be modified, so that the Arduinos are placed visible and beside the key areas. With my current keyboard I got used to it, that both part have quite a distance to each others, to have the Arduinos right to the center will be not a problem for me. My next goal is, to have it as flat as possible. My current setup is a little bit to high in my opinion. Also the reason why the Arduinos is on the side instead underneath the top plate. The top and bottom plate is again clear acrylic and I want to use LEDs as back-lights. I know its possible to have LEDs for each keys with the custom firmware, but don't know if its maybe again the wish of having it as flat as possible for a hardwired keyboard.

Partlist

Switches Kailh Low Profile Red pro (Linear 35g) (70 PCS 50 EUR)
top and bottom plate (49 EUR Lasercut services)
arduino (Pro Micro) (3PCS 10 EUR)
3.5mm male and female jacks (5pcs 9 EUR)
keycaps (17 EUR)
diodes 1N4148 (100pcs 1.90 EUR)

Other stuff

solderiron
resistors
cables
usb cable with data line
spacer for top and bottom plate
perf board

You see, my next board will be also something around the 159 USD mark.

In my next part I will go deeper into Keyboard matrices, QMK firmware and how to create the keymaps.

Write github reviews comments from vim

mafflerbach — Sat, 02 Jan 2021 14:04:11 +0000

Critiq is a neovim plugin to review github pull requests.

To setup Critic you have to set environment variables in your shell.
Be aware - if you want to use it in a corporate field, you access token should authorized the sso from your organization.

Usually you are doing it in exporting this in you shell rc file.
In my case it is the ~/.zshrc in bash it is in the ~/.bashrc

export GH_PASS="Your github token generated [here](https://github.com/settings/tokens)"
export GH_USER="YourGithubUsername"

With :Critiq you get an separated buffer with all the open pull requests.

With enter, you will open the diff which belongs to the pr.
With :CritiqCommentLine you can right away commenting on the line. It will open a separated split buffer. With :CritiqSubmitComment
it will post your comment on github.

In the case that you are satisfied with the PR, you can also use CritiqApprove to approve the request.

Sadly this extension is not touched for a long time. It will still do the work, but occasionally you will facing some error, if you want to open a pr. But in general you are able to work with that extension.

This are the main feature in my opinion.

But there are a lot more possibilities if we take a look on the Commands:

:CritiqApprove
:CritiqBrowsePr
:CritiqCheckout
:CritiqComment
:CritiqCommentLine
:CritiqDeleteBranch
:CritiqEditLabels
:CritiqListComments
:CritiqListCommits
:CritiqLoadMorePrs
:CritiqMerge
:CritiqOpenFile
:CritiqOpenPr
:CritiqPull
:CritiqRequestChanges
:CritiqSubmitComment
:CritiqSubmitReview
:CritiqToggleLabel

I will try to adapt my workflow and figuring out if this will feel comfortable, but it looks promising.

qutebrowser loading browser session from command line

mafflerbach — Sun, 27 Dec 2020 22:23:33 +0000

This will be a shot one. In qutebrowser you have the possibility to save browser sessions. This will save all your current open tabs.
You can obviously loading this session files, and all the tabs will be restored.

In my case it comes handy, in support weeks when I have to monitor different pages.

And to have a more direct access, I wrote a two liner and combine rofi with that.

#!/bin/bash

sessions=$(ls -1 $HOME/.local/share/qutebrowser/sessions | rofi -dmenu -theme ~/dotfiles/i3/rofi.rasi)
if [ "$sessions" != "" ]; then
    qutebrowser ":session-load ${sessions/\.yml/}"
fi

Why I love programming

mafflerbach — Tue, 22 Dec 2020 21:44:53 +0000

I am now for 16 years in this industry. And still I become exited about new technologies and there possibilities.
I like to have the chance to program everything what comes in my mind.

I have a boring workflow? I can write a script for this.
I want to scan a qr code to prepare my groceries and prepping the week? I have a script for that.
The current look of my media center is not on point of my taste? I can change how I want it, or writing one from scratch.
I want to build my custom keyboard? Considered as done.

And with every small script, or every time I "just play" I am learning new stuff or even new languages.

I started with rust, because of curiosity, and start to work on a media center based on mpv.
On the way, I learn rust with real problems to solve. How to do Frontend, how to manage Backend. Writing to a database and on a socket to use mpv. Scanning the directories for Content, and ask third-party APIs to get movie informations and some other bits and bytes to solve my problems, or my feature list.

Today I played around with python and speech-recognition and hacking something together to have an alexa "clone".

I wrote a tool for smoke testing our solutions also in rust. This contains writing on a message queue, and getting the logs from the remote server, where the solutions is deployed. Of course in that manner, that I just can copy and paste the outcome to jira. Now when I am thinking of, my next task will be that the smoke tester will do the comment as well (yeah I am lazy af and hate jira).

I am using awesome wm. You can literally program every aspect and not only related to the look and feel. Its written in Lua. I write lots of smaller things in bash.

The power of coding in general is, to make my life easier when I am sitting at my pc. I really enjoy the process of fiddling stuff out, learn new languages or get a peak in side a new technology. To solve complex problems and provide a program for that in the best way possible, is pure excitement for me.

I hope that I will never lose that joy and my curiosity for my profession. I am really in the lucky position, that my hobby is my job.

Openfortivpn - automatet network interface switcher

mafflerbach — Thu, 17 Dec 2020 10:21:39 +0000

On our workplace we have to use Fortinet to connect to our VPN Network.
But if I'm connected to wifi and cable, some how I have issues to resolve addresses within the company network. So every time, I deactivate my wifi and establish then the VPN connection.
But - most of the time I forget this step. So I disconnect the VPN, deactivate wifi, and reactivate the connection. We have also two way Auth activated, so in a bad case I have to type three times Auth passwords (root pw, because its needed from client, network Auth from the company, and the two way Auth pin, which will be send to me)

So, I wrote a script...


#!/bin/bash
cableInterface="enp0s31f6"
wifiInterface="wlp3s0"

wireConnecton=$(ethtool "$cableInterface" | grep Link | cut -d' ' -f3)
wifiConnecton=$(ethtool "$wifiInterface" | grep Link | cut -d' ' -f3)

while :; do
    case "$1" in
        -c) 

            tmux kill-session -t VPN
            if [ "$wifiConnecton" == "yes" ] && [ "$wireConnecton" == "yes" ]; then
                echo "disable wifi"
                nmcli device disconnect "$wifiInterface"
            fi

            # activate wifi if i am not on cable
            if [ "$wifiConnecton" == "no" ] && [ "$wireConnecton" == "no" ]; then
                nmcli device connect "$wifiInterface"
            fi
            # get passwords and usernames
            sysPass=$(pass show path/to/rootPw | head -n1)
            pass=$(pass show path/for/vpnaccess | head -n1)
            user=$(pass show path/for/vpnaccess | tail -n1)

            # create new tmux session named VPN and establish vpn connection
            sleep 2
            tmux new-session -d -s "VPN"  sudo /usr/bin/openfortivpn -c /etc/openfortivpn/config -u "$user"  -p "$pass"
            sleep 1
            # send root password to system auth
            tmux send-keys -t "VPN:0" $sysPass Enter
            ;;
        -k) 
            # kill vpn conneciton 
            if [ "$wifiConnecton" == "no" ]; then
                 nmcli device connect "$wifiInterface"
            fi
            echo $(pass show path/to/rootPw | head -n1) | sudo -S killall openfortivpn 
            # kill session
            tmux kill-session -t VPN
            ;;
        *)
            break
    esac
    shift
done

But I want to add some additional breeze. I am using rofi not only as window switcher and application launcher. I trigger as well some custom script e.g. change my wallpaper and color theme, or trigger Jenkins builds. So I add this script to my custom launcher, and now I just have to type in the pin which will be send to my mobile phone.

Animated glassmorphism login in css and html

mafflerbach — Thu, 10 Dec 2020 09:23:43 +0000

I really like this blury glass effects, so I try to implement something in css and html.

I did't use any thirdparty js lib for the animation as well and it looks awesome (imo) and it was really simple to implement.

Here is jsfiddle: