DEV Community: Anton Goncharov

Sample Surveys Application (Kotlin + Spring + RSocket + Vue.js Demo)

Anton Goncharov — Tue, 06 Jul 2021 11:30:49 +0000

Recently I started building a repo with boilerplate code to facilitate creation of my future full-stack pet projects. Despite it's work in progress and always will be, as I'll continue to implement new features and tweak existing ones, I wanted to share it with the community.

Check out the GitHub repo I'm covering in this post:

anton-goncharov / kotlin-vue-surveys-demo

Boilerplate Kotlin+Vue fullstack application made with Spring, JWT, WebFlux, H2, JPA, RSocket. Implements sample surveying functionality.

Summary

It's a sample "surveys" application with the actors of two roles: coordinators and respondents. Coordinators are allowed to create & edit polls, then review response statistics. Respondents have read-only view with the only option to submit survey responses.

One of the key features is that the statistics data is refreshed in real time on a survey stats page 📊. It works by using Rsocket and Reactive Streams.

Concepts

This demo application showcases the following list of concepts:

Kotlin and Spring backend API development

I've started using Kotlin on new backend projects 2 years ago, and it's my first repository to have boilerplate code for full-stack apps with Kotlin. Here I'm using Kotlin in pretty much the same Spring setup as I would use Java.

Rapid CRUD bootstrapping using Spring Data REST

It's arguable whether Spring Data REST is a good fit for a production-ready project. On the other hand, it allows a rapid start having full-blown API implementation with sorting, paging and filtering for given entities 💁.

Any given controller can be extended with additional handlers or reimplemented from scratch as a separate custom REST endpoint.

Database migrations with Liquibase

Hibernate JPA auto schema creation was used during the development. After the first version of the schema had been finalized, I generated the Liquibase changelog from JPA entities and disabled ddl-auto by Hibernate.

To populate schema with some initial data (predefined users), there's sql script on classpath.

Security

JWT-based authentication with role-based access is showcased with tokens issued by the backend. ❗️ Such a setup must not be used in production, this functionality is only to demonstrate the flow. In a real solution, use IAM solution like Keycloak or 3rd party SaaS.

Reactive flow for live data streaming

After reading the official guide on Spring Webflux + RSocket I got an idea to implement some real-time streaming functionality to try Reactive Streams in a demo application setup.

The survey statistics page is implemented using rsocket.js, it reads the survey response stream data from a Kotlin Flow that is populated with data from R2DBC H2 reactive-ready driver. The charts are updated on a survey stats page in real time.

Building with Gradle

As a sidenote, I've never used Gradle before, always preferring Maven as a classic tool that makes more sense to me 😃. This was an interesting challenge to create something new with Gradle, especially writing the build script with Kotlin DSL since there're not many ready-to-use examples.

Tech Stack

🔸 Backend

Kotlin 1.4
Spring Boot 2.4.3 (JPA Data REST / Hibernate + Security + WebFlux)
H2 Database with Liquibase database migrations
Gradle
JUnit 5

🔹 Frontend

Vue.js 2 (with Vuex, Vue-router, Vue-image-upload)
Bootstrap 4
RSocket
Apexcharts

Launching

🚀 To get it up and running follow the instructions from the GitHub repo.

Please reach me out if you have any questions or feedback. But keep in mind it's a boilerplate code withstanding permanent work in progress 🛠. Cheers!

Distributed Logs and Tracing with Spring, Apache Camel, OpenTelemetry and Grafana: Example

Anton Goncharov — Mon, 21 Jun 2021 12:02:46 +0000

I've built an Apache Camel & OpenTelemetry demo project. It showcases how to achieve distributed tracing across Camel routes and other components using OpenTelemetry, which is the OpenCensus and OpenTracing projects merged into a new single specification under CNCF. The project uses Spring Boot as a base service framework, Loki to store logs, Tempo to store traces, Grafana as a visualization software.

anton-goncharov / camel-opentelemetry-grafana-demo

An Apache Camel & OpenTelemetry demo project. Uses Spring Boot as a base service framework, Loki as a log storage, Tempo as a trace storage, Grafana as a visualization software.

Introduction

Apache Camel has added OpenTelemetry support in the 3.5 release. The official documentation doesn't focus on how to set everything up for logs and traces visibility, so the goal of this example is to fill this gap.

OpenTelemetry is an open-source project resulted from merging of OpenCensus and OpenTracing. Its purpose is to enable an application's observability by providing a set of standardized data collection tools for capturing and exporting metrics, logs and traces. It's one of the most active CNCF projects these days.

Using Camel in tandem with OpenTelemetry instrumentation allows us to have distributed tracing across different routes in one or more services, and to link application logs with these traces, which makes it a great alternative to ELK and similar solutions.

This logs & traces view is what we're aiming to achieve in the end:

Important note: this example uses Apache Camel 3.10 (the last available version as of June 2021), which depends on OpenTelemetry SDK for Java version 0.15.0. OpenTelemetry specification and its tools develop rapidly, now beyond 1.x release. In the Camel tracker I see the plans to upgrade dependency but today make sure 0.15.0 is used to run this example smoothly.

How it works

The project is a Spring Boot web application named 'hello-service' serving 2 endpoints. /hello responds with "Hello, ${name}" and /dispatch passes the request to a downstream service.

The docker-compose config spins up 5 container instances:

hello-service (1) (container "dispatch") accepts requests and dispatches them down to the hello-service (2)
hello-service (2) (container "hello") handles dispatched requests
loki is a logs storage
tempo is a traces storage
grafana is a web dashboard with loki/tempo data visualizations

Interacting hello-service (1) and hello-service (2) are here to demonstrate how OpenTelemetry traces requests across different containers.

The architecture overview:

Both 'hello' services send logs to Loki via loki-docker-driver and traces to Tempo via the OpenTelemetry exporter. Grafana uses Loki and Tempo as data sources. Collected logs are linked to the trace data by using traceId assigned by the OpenTelemetry instrumentation.

Service

OpenTelemetry instrumentation dynamically captures telemetry from a number of popular Java frameworks. To plug it in, add opentelemetry-javaagent-all.jar as a javaagent of the JAR application. This preparation is done in the Dockerfile.

For the Camel integration it's important that we use opentelemetry-javaagent-all.jar version 0.15.0 as camel-opentelemetry depends on this OTel Java SDK release.

ADD /agent/opentelemetry-javaagent-all.jar /etc/agent/opentelemetry-javaagent-all.jar

ENTRYPOINT ["java", "-javaagent:/etc/agent/opentelemetry-javaagent-all.jar" , "-jar", "observable-service.jar"]

For Camel 3.5+ it's required to add the camel-opentelemetry-starter dependency. Then, in a RouteBuilder class, set up the tracer as the following:

OpenTelemetryTracer ott = new OpenTelemetryTracer();
ott.init(this.getContext());

The docker-compose configuration has a few items to highlight.

OTEL_EXPORTER_OTLP_ENDPOINT is an endpoint where to send the opentelemetry data. In our case it's the Tempo's receiver.
We don't send metrics in this demo, so it's disabled to hide all related warnings.
The downstream endpoint point to the address of the second service to show how distributed tracing works.
The logging sections states that the container's stdout should be pipelined to the Loki storage by using the loki-docker-driver.

dispatch-service:
  image: hello-service:1.0.0
  container_name: dispatch
  networks:
    - "tempo-net"
  ports:
    - "8080:8080"
  environment:
    - OTEL_EXPORTER_OTLP_ENDPOINT=http://tempo:55680
    - OTEL_EXPORTER_OTLP_INSECURE=true
    - OTEL_METRICS_EXPORTER=none
    - OTEL_RESOURCE_ATTRIBUTES=service.name=dispatcher
    # sends requests to the 'hello-service' container
    - SRV_DOWNSTREAM_ENDPOINT=http://hello:8080/hello
  logging:
    driver: loki
    options:
      loki-url: "http://localhost:3100/loki/api/v1/push"

The second service instance has the similar setup except for the fact that it has no downstream service.

In application.properties, the logging.pattern formats log output in a way to include traceId and spanId. It's necessary for the system to be able to index logs and link them to traces.

Loki

Loki is a scalable log aggregation system to use with Grafana.

The Loki config is pretty much default. It exposes port 3100 so all the other docker containers can send logs there.

loki:
  hostname: "loki"
  image: grafana/loki:2.2.0
  ports:
    - "3100:3100"
  networks:
    - "tempo-net"
  command: "-config.file=/etc/loki/local-config.yaml"

Tempo

Tempo is a scalable trace storage. It has receivers for Jaeger, Zipkin and OpenTelemetry.

The tempo-config.yaml is mounted from the project. It disables auth and specifies that Tempo instance should expose an OpenTelemetry receiver. The default OTel port is 55680.

tempo:
    hostname: "tempo"
    image: grafana/tempo:latest
    networks:
      - "tempo-net"
    volumes:
      - ./tempo-config.yaml:/etc/tempo-config.yaml
    command: "-config.file=/etc/tempo-config.yaml"

Grafana

Grafana is a visualization and monitoring software that can connect to a wide variety of data sources and query them.

How to launch

I used Maven 3.8.1 + JDK 11 to create the demo, so it should work with all version from these onwards.

Steps to launch and try:

1. Prepare Docker

Install Loki driver to Docker (to stream logs from the standard output of a container)

docker plugin install grafana/loki-docker-driver:latest -- 
alias loki --grant-all-permissions

Create a network for the demo services:

docker network create tempo-net

2. Pack the service with Maven

mvn package

3. Build the container image

docker build . -t hello-service:1.0.0

4. Run the docker-compose script to bring up the demo

docker-compose run

5. Open a web browser and navigate to the Grafana dashboard: http://localhost:3000. Go to the Configuration->Data Sources menu item.

6. Add new Tempo data source (a storage for traces)

7. Add new Loki data source (a storage for logs)

Configure "Derived Field" in Loki Data Source to relate logs with traces

8. Send a sample request

curl http://localhost:8080/dispatch?name=Ada

9. After all set and executed as above, browse to the Grafana UI at http://localhost:3000. Open the "Explore" view and switch to the Loki view using to dropdown on the top.

Searching for {container_name="dispatch"} will output the recent logs for the hello-service (1) running in the dispatch container.

Note that each log entry now has traceId, a global identifier to track request across interacting services; and spanId that identifies local unit of work (eg. an individual route or a handler within a service).

Click on an entry that has a traceId. It would unfold a detailed info on the log entry. Next to the "traceId" field there's a button that would carry us to the corresponding tracing.

Voilà, we've got to the distributed tracing for this request. Each span has the corresponding log entries grouped under the "Logs" box on this panel.

The combined split view in Grafana allows browsing the logs and traces side by side.

Low-Code Tradeoffs

Anton Goncharov — Wed, 16 Jun 2021 22:01:27 +0000

I’ve been a low-code developer and architect for few years consulting clients worldwide. Even though “low-code” is a buzzword these days, the concept and its applications have been around for many years.

As it’s rather trendy today, I see that many companies great and small are tend to fit in low-code platforms for their needs. I have extensive experience working with such backend solutions in different environments and subjects areas. In addition, I participated in designing and building a new visual development platform. It all gave me an idea of sharing my best experience on this topic, and advise decision makers considering low-code could ultimately optimize software development processes in their organizations.

What is low-code?

Low-code, as the name states, is about building applications using abstractions over code to hide its complexity. This usually implies visual programming tools that make it possible to create something by combining ready blocks and configuring their interactions without traditional coding.

Such a tool rely on code generation, there’s always a powerful translator under the hood. It maps visual “low-code” representation into actual executable code, then wraps it into an deployable framework. Some of the low-code platforms have cloud integrations so they provide an option to publish the built application into a public domain.

Low-code embrace many use cases, in my post I’d like to focus on the most powerful tools that enable users to build APIs and microservices to automate complex processes from ground up.

I don’t want to highlight particular vendor platforms. Out of my experience, different solutions in this area have many features in common. All I’ll note below could be applied to a generic low-code visual tool allowing users to build pipelines and services by chaining pre-built blocks:

Where does it shine?

Firstly, I’d want to highlight the low-code usages where it really makes difference without significant tradeoffs.

It’s great for personal use, especially if one has a little or no experience in programming. Today we’ve got a great lot of services and smart devices. Even with a moderate interest in whole variety of applications, there’re scenarios that could make life easier: from IoT shortcuts to convenient data sync between web apps.

Example: write a pipeline that parses commands from a telegram bot to append new entries to a budget spreadsheet.

For solo developers to automate routine in their pet-projects.

Example: create a service handling orders from a single-page web shop, integrated with Mailgun API to send and track emails.

For startups to implement a throwaway MVP in order to test a hypothesis.

Example: a travel agency has decided to start an affiliate program and now they need to implement a dashboard for affiliates to use. While requirements haven’t yet been gathered and a development team hasn’t been packed up, business analysts may scrap a simple solution using a low-code platform.

Low-code in the world of enterprise

There’s always a lot of development going on in big companies. They’re constantly seeking for ways to optimize costs and speed up delivery of new features.

Low-code may seem to be a silver bullet as it enables more people to write code and accelerates development by reusing ready-to—use blocks. This usually translates to top managers being fascinated by potential benefits of putting development on low-code track.

My opinion that it’s a popular delusion, and I saw many companies chosen this way but eventually switched back to traditional coding. Below I’ll reason it by listing the frequent issues.

Common challenges

As I noted above, what low-code provides us is a convenient level of abstraction above the code. It’s great for rapid prototyping or in case we’re building something that simple so we don’t need to bother about its internals. As long as we start thinking about how underlying code works, and try to adjust the visual design accordingly, we actually no longer need the low-code layer. It’s a red flag when low-code platform users explain how an application works by using terms of the underlying code.
It generates code from the graphical representation. A platform version upgrade may change implementation details which affect how the visual language is translated to code. It may have implications on your application behavior, and such bugs are very difficult to trace down since the platform internals are hidden.
Typically a low-code IDE lacks tooling for editing and teamwork because for vendors it’s hard to implement and maintain such logic when it comes to visual programming.
- Git support is at primitive level. It goes smoothly only when there’s fast-forward merging. Other cases are tedious to resolve as even when visual graphs are merged correctly, the conflicts in components’ properties make it a mess.
- Most of low-code IDEs don’t have “find and replace” for strings across blocks inputs. A developer should click through all the components and manually rename target value.
- Some tools may partially provide it, but often there’s no functionality to navigate throughout the project, e.g. “Find Usages”, “Find Duplicates” or “Browse Hierarchy”. And as a consequence, it’s difficult to decouple logic as one can’t quickly jump between dependencies.
- As opposed to no-code platforms, a low-code IDE usually provides a very rudimentary code editor to include snippets of custom code where it’s needed. The editor’s functionality is limited so traditionally developers use another IDE to write code and copy it from there. It increases effort and leads to nasty bugs caused by minor discrepancies between contents of the editors.
- Lack of CI/CD integration in such tools is a common issue, as low-code platforms are by and large designed to be self-reliant having no perspective to participate in broader infrastructure.
- It’s possible to continue the list since there’re a lot of rough edges when it comes to low-code development tooling. I was occasionally upset not finding any automated testing features or hot reload of a running project.
I watched an interview with Anders Hejlsberg, a creator of Delphi, a visual development tool that was incredibly popular along with Visual Basic at the time when I was a student. He said that “when it comes to visual programming, a line of code is worth a thousand pictures because you just die a slow death in wires going from everywhere to everywhere”. https://youtu.be/nCVNpwXijVQ (21:20)

His main point was that visual programming doesn’t scale. I can’t agree more as what I’ve seen in long-living enterprise low-code projects was utter “blocks’n’wires” chaos. The use of off-the-shelf components is designed to greatly template development. But when we need a lot of different blocks with small adjustments, and we can't change their internals, we create new custom blocks, therefore generating more duplicated code.

Everything has its time and place

I love no-code and low-code platforms as they enable lots of creative people to put new ideas into practice. They start new ventures and automate everyday routine by manipulating data in a smart way.

The visual developments tools emerged a long time ago and will remain with us for a long time to come. New and more elegant products with better UX will appear. As many companies have tried to adapt low-code to their business, I consulted them along this way. After avoiding pitfalls on a few projects, I’ve gained enough experience and insight into the process.

To summarize, a low-code platform isn’t a good fit for a company where software development is a teamwork and when it has to scale. As of today, none of the tools on the market is suitable for building a full-blown service layer. Most prominent challenges include lack of professional tooling, scalability and infrastructure limitations. There’re great examples of handy low-code tools for enterprises but their main feature is that they effectively address one particular challenge (e.g., monitoring scenarios) not being platforms for building any kind of software.

It's Easy To Overengineer But Underwork

Anton Goncharov — Sun, 02 May 2021 10:00:52 +0000

Many developers have already learned how to write concise code, to build scalable software architecture; they turned infrastructure into a smart conveyor. They accustomed themselves to a high-tech toolset, which is suitable for dealing with foreseeable issues. A problem, that was a creative task just yesterday, today becomes a template. Engineering approach as it is. Script it, observe and enjoy.

In the area of IT there're own fads and trends. Today we see microservices, microcontainers, blockchains, tomorrow's agenda will get us something new. A certain part of IT industry tries hard to ensure that we're in a stream of constant changes. They're capitalizing on our yearning for novelty. While I'm writing this text, it's likely that a yet another whatever.js framework is published somewhere.

An enlightened developer should be above this tech chaos. Experience in a technology measured in years has ceased to mean much since a long time ago. Knack can be developed fast and could be obtained from various sources. In the long run other skills make difference.

Software development is harder than just figuring what API should be called today, and more complex than keeping in mind a checklist of actual practices and techniques. A viable software product can't be considered apart from the business it's serving to. Today software teams should give more consideration to product evolution. It's very easy to overengineer but underwork.

After all, we create not just a piece of software, but a technological process around it. What's really difficult is to build a system of methods, communications and conditions allowing a product to stand firm within a frame of expectations. Yet we can't expect a marvellous tech that will do it for us.