We approach the project and a question arises - which load testing tool should we use? JMeter? Locust? K6? Or maybe Gatling? If we're familiar with the tool - it's okay. But what if we're dealing with a tool we haven't encountered before? Today I will tell you about my thoughts on k6 and JMeter. This comparison will be more general, which, I hope, will clear up many doubts. Instead of just relying on dry knowledge, I want to focus on my experience.
Technology Support
The first aspect we should consider when choosing a tool is the supported technologies. For example - if we want to performance-test Apache Kafka, we should check how strong the support for it is in the load testing tool. In the past, I often heard opinions that k6 is still not mature enough for real projects. Because of this, I decided to check how many technologies are supported by JMeter and how many by k6.
First and foremost, k6, when compared to JMeter, has very few built-in protocols. However, thanks to the extension capabilities with xk6, almost all protocols are available. There are single exceptions like FTP. All other protocols known from JMeter, such as TCP or SMTP, are available via extensions in k6. Interestingly, the handling of some protocols in k6 is more user-friendly than in JMeter. This includes, for example, WebSocket.
Do you like this content? Subscribe on YouTube 👍
In summary - the greater support for technologies in JMeter than in k6 can be considered a thing of the past. It is no longer an obstacle to using k6 in commercial projects, and the amount of protocol support is, in my opinion, comparable.
Infrastructure as a code vs UI
K6 and JMeter are two popular performance testing tools, but they offer different approaches in this field. K6 follows the "Infrastructure as Code" philosophy, where users create test scripts in JavaScript. This gives professionals greater flexibility and precision in defining tests, allowing for deep integration with various CI/CD tools and faster writing of test scenarios requiring advanced data operations.
On the other hand, we have JMeter, which focuses on the visual aspect of test creation through a graphical user interface (GUI). Although I am an advocate of writing most projects at the code level, it is not always an advantage. In the case of Microsoft 365 Dynamics applications, it turned out that the specifics of the project for which I was creating preliminary test scenarios were so complicated that writing test scenarios in k6 became a suffering. Simple test scenarios (due to data sent in JSON format) could even have over 2,000 lines of code. Writing scenarios in this case was simply unpleasant for me in k6. I emphasize that these are rather isolated cases that one should check oneself using a live application as an example.
Framework Construction
Another issue we should consider for long-term projects is creating and maintaining a framework. If we work in a team where there is more than one person, we should keep in mind that the code review of test scenarios written in JMeter will be one of the things you will hate. It's understandable since, in the code repository, we won't add comments due to the XML syntax, but to check the code itself, you have to run JMeter and click through each step one by one. It's simply exhausting.
In k6, however, thanks to writing scenarios at the code level, the review process is pleasant and straightforward. Moreover, nothing prevents several people from working simultaneously on multiple scripts. This not only speeds up the work process but also allows us to share knowledge within the team better.
Tool Performance
Even though we work daily as performance testers or performance engineers, we often forget why we do it. I mean that we overlook the performance of the tools we use, focusing only on the end application. We shouldn't trivialize this, because with larger loads, we can generate a significant cost of the environment that is unnecessary.
In articles about performance testing tools, you can often find information that k6 performs up to 10 times better than JMeter in terms of memory usage. Since this information is from 2021, I decided to check it two years later on the latest versions of JMeter and k6. At this point, I can say that although the results are not 10 times better, k6 currently performs much better. A separate material is planned for the performance check of both tools, so I encourage you to subscribe so you don't miss anything.
Reporting
Let's talk a bit about reporting in both tools. Although it's possible to generate a report in HTML format in both of them, the one available in k6 still lacks perfection. I pondered what this might be due to. The only thing that came to mind is that k6 is designed to build entire observability systems. What does this mean?
Traditional, and even somewhat primitive performance testing involves conducting them once in a while, say once a month or two. In such cases, it's understandable that preparing the architecture for reviewing performance test results may be a bit of overkill, and a generated report would suffice. However, when performance tests are integrated into CI/CD processes, test scenarios are often created and maintained, and our tests are automated, generating reports and sharing them among team members daily might be too time-consuming.
Here I also come to a conclusion - I have the impression that k6 is designed with automation in mind and integration into observability systems, while JMeter puts emphasis on short-term projects and rather manual analysis. Personally, I like this approach of k6 because it somehow pressures us to collaborate with other members of devops teams in creating system observability. Not only do we grow thanks to this, but we also automate a significant part of our work in the long run. If we want to be a performance engineer or architect, this knowledge is essential.
Tool Development Perspectives
In the world of IT, we shouldn't think short-term. Currently, hundreds of different technologies are emerging weekly, so we should consider the future prospects of the tool. JMeter is unquestionably the most commonly used in projects, making it a good choice for those who are starting their journey in IT. On the other hand, I wonder if, having been created over 20 years ago, its design assumptions are not outdated in today's world where everything is delivered at the code level.
I want to touch on k6. An important fact is that since June 2021, k6 has been developed under the auspices of Grafana Labs, experts in system monitoring and observability. Their most famous tool is Grafana, used for data visualization. Other interesting projects include Loki, Mimir, and Phlare, which are an integral part of the ecosystem of monitoring and data analysis tools, often used alongside Grafana.
I believe that such a large company's engagement in system monitoring in the development of a performance testing tool is a wise decision, with the potential to bring groundbreaking innovations in the field of performance. At this stage, it's evident that this is happening – as demonstrated by the ability to write functional tests and chaos tests from within a single tool.
Summary
Choosing between JMeter and k6 depends on the project specifics. JMeter, present in the market for two decades, offers visual test design and extensive technological support. On the other hand, k6, developed by Grafana Labs, promotes "Infrastructure as Code", making it more flexible for integrated CI/CD environments. In terms of technology support, both tools are comparable, but k6 is more resource-efficient. Regarding reporting, k6 leans towards automation, while JMeter leans towards traditional methods. In the long run, with Grafana Labs' support, k6 has the potential to bring innovations to performance tests. The final choice depends on individual needs and working comfort.
Top comments (0)