DEV Community: Daniil Roman

Automate Your Java Upgrades: A Practical Case Study with OpenRewrite and GitHub Actions

Daniil Roman — Fri, 10 Oct 2025 13:12:02 +0000

Tech debt grows relentlessly. Even on services you don't touch, dependencies become outdated, creating a constant maintenance burden. Manually upgrading dozens of services is slow and error-prone. What if you could automate a significant part of that process?

Remember your last big Spring Boot or Java version upgrade? How did it go? Did you spend hours renaming javax to jakarta packages across 30 or maybe 50 different services? Or perhaps you lost a whole day figuring out why a simple dependency bump broke the build? If any of this sounds familiar, you're in the right place.

What is OpenRewrite?

OpenRewrite is an open-source refactoring engine that automates code modifications at scale, enabling consistent and reliable refactoring across large codebases and significantly reducing manual effort.

OpenRewrite works by making changes to Lossless Semantic Trees (LSTs) that represent your source code and printing the modified trees back into source code. You can then review the changes in your code and commit the results. Modifications to the LST are performed in Visitors and visitors are aggregated into Recipes. OpenRewrite recipes make minimally invasive changes to your source code that honor the original formatting.

You can check this project out on GitHub

When would you use OpenRewrite?

You might be thinking, "This sounds great in theory, but what are the real-world use cases?" Let's recall some notable migrations many of us have faced recently:

Spring Boot 2.x to 3.x migration
This migration was especially remarkably by the seemingly simple task of renaming javax.* to jakarta.* namespaces. However, this change had to be applied to every single service using Spring Boot. In a typical Java and Spring ecosystem, that means changing it everywhere.

OpenRewrite offers a recipe that automates this entire process.
https://docs.openrewrite.org/recipes/java/spring/boot3/upgradespringboot_3_0

JUnit 4 to JUnit 5 Migration
Imagine you need to migrate from JUnit 4 to JUnit 5, but your codebase still has a few outdated annotations scattered around. As a part of this migration you'd need to rename @BeforeClass to @BeforeAll or @AfterClass to @AfterAll.
It doesn't sound too complicated, but it's tedious work that can be fully automated with an OpenRewrite recipe.
https://docs.openrewrite.org/running-recipes/popular-recipe-guides/migrate-from-junit-4-to-junit-5

Java Version Upgrades:
Or maybe you're upgrading to Java 21 and want to replace the deprecated new URL(String) constructor with the URI.create(String).toURL() across your entire codebase.
There's a recipe for that too: https://docs.openrewrite.org/recipes/java/migrate/upgradetojava21

In other words, we developers are constantly challenged with keeping our dependencies up to date. OpenRewrite is here to rescue us—or at least, to significantly reduce the pain.

Our experience of using OpenRewrite

What worked for us

Let's start with what went well, as we found the tool both useful and promising.

Keeping `pom.xml` in shape

In the OpenRewrite ecosystem, the magic comes from its recipes. For us, the first no-brainer was the Apache Maven best practices recipe.
It was immediately clear that we had no other tool in our stack that could consistently keep our pom.xml files in good shape.

As a simple but welcome feature, this recipe reorders the sections of a pom.xml to follow a standard pattern. This helps with readability, especially in large files.

But its real power lies elsewhere: the recipe can find and remove duplicate or unused dependencies, improving the long-term stability of a service.

As a note of caution, I admit it can be scary at first to accept an automated PR that removes dependencies. Make sure you have good test coverage before trusting any automated tool to this extent.

Refactoring test libraries

Of course, we wanted to see it work on actual Java code. As always, the safest place to try a new tool is on your tests, so that's exactly what we did. We were already in the process of standardizing on AssertJ, so we introduced three relevant recipes:

org.openrewrite.java.testing.assertj.Assertj
org.openrewrite.java.testing.mockito.MockitoBestPractices
org.openrewrite.java.testing.testcontainers.TestContainersBestPractices We ran these without specific expectations and were pleasantly surprised when they spotted and fixed several sore spots in our test code.

What didn't work

But of course, the juicy part is always where things go wrong.
In the following paragraphs, we will look at a few examples that we were not entirely satisfied with.

Complex, custom refactoring

We tried to use a recipe to fully migrate our tests from Hamcrest to AssertJ, but it simply ignored our custom matchers.

While some recipes are more powerful than others, our general feeling is that OpenRewrite struggles with highly complex or bespoke refactorings on its own.

Running Java recipes on Kotlin projects

It may seem obvious that Java recipes should only be run on Java projects. However, like many companies, we have a mix of Java and Kotlin projects, so we simply ran the recipes against all of our team's services to see what would happen. It turns out that it partially works, but it fails in enough cases to produce strange changes and broken PRs.

This makes things tricky if you want to run a uniform set of recipes across all repositories using a tool like GitHub Actions, which we'll cover next.

Newest recipes are often commercial

This might be obvious if you're already familiar with OpenRewrite, but it's worth mentioning. If you want to use OpenRewrite for Spring Boot upgrades, you'll find that the recipe for the latest version might be under a commercial license. For example, if Spring Boot 3.5 is the latest release, the open-source recipe might only support up to version 3.4. This makes perfect sense from a business perspective, but it's something to keep in mind. In short: the OpenRewrite engine is open-source, but the most cutting-edge recipes are often licensed separately.

Our automation setup with GitHub Actions

There are a few ways to run OpenRewrite recipes. If you're using Maven, you can add the rewrite-maven-plugin directly to your pom.xml. This can be configured to run during your local build or only on CI.

Example:

<project>
  <build>
    <plugins>
      <plugin>
        <groupId>org.openrewrite.maven</groupId>
        <artifactId>rewrite-maven-plugin</artifactId>
        <version>6.18.0</version>
        <configuration>
          <exportDatatables>true</exportDatatables>
          <activeRecipes>
            <recipe>org.openrewrite.staticanalysis.JavaApiBestPractices</recipe>
          </activeRecipes>
        </configuration>
        <dependencies>
          <dependency>
            <groupId>org.openrewrite.recipe</groupId>
            <artifactId>rewrite-static-analysis</artifactId>
            <version>2.17.0</version>
          </dependency>
        </dependencies>
      </plugin>
    </plugins>
  </build>
</project>

Alternatively, you can run the plugin directly from the command line:

mvn -U org.openrewrite.maven:rewrite-maven-plugin:run -Drewrite.recipeArtifactCoordinates=org.openrewrite.recipe:rewrite-static-analysis:RELEASE -Drewrite.activeRecipes=org.openrewrite.staticanalysis.JavaApiBestPractices -Drewrite.exportDatatables=true

This gives you the flexibility to run it manually, as part of a CI pipeline, or on a nightly schedule.
We chose the third option: run it via a scheduled GitHub Action on a daily basis and automatically create a PR if any changes are detected.

Why not just use the Maven plugin?

The main drawback of adding the plugin to your pom.xml is that it significantly slows down every build, even when there are no changes to be made. You could run it as a CI-only check, but that creates a frustrating workflow: the CI build would fail, and a developer would have to run the command locally to generate the changes and push another commit. This kind of friction hurts tool adoption.

Running OpenRewrite on a schedule mimics the behavior of Dependabot or Renovate. Developers don't have to actively run anything; they simply review and merge the auto-generated PRs. Ideally, only a small portion of these PRs will have failures.

Another benefit of the GitHub Action approach is centralization. We can update the recipes in a single, shared workflow file and have that change apply to all our repositories without touching a single pom.xml. And since the result is always a PR and not a direct commit, it's a completely safe operation.

The GitHub workflow in detail

Imagine you have 20 repositories. Modifying the pom.xml in every one of them just to add or change a recipe would be painful and would quickly lead to abandoning the tool. With a centralized GitHub Action, however, each repository only needs a small trigger file:

name: OpenRewrite Scheduled PR

on:
  schedule:
    - cron: '0 7 * * MON-FRI'
  workflow_dispatch: # Allows manual triggering

jobs:
  call-openrewrite-workflow:
    uses: your-organisation/your-repository/.github/workflows/reusable-openrewrite-auto-pr.yml@main
    secrets: inherit

This file references a reusable workflow, which contains the actual logic and OpenRewrite configuration. Now, whenever we add or remove a recipe, we only modify the central workflow, and all repositories pick up the change on their next scheduled run.

name: Reusable OpenRewrite Auto PR Workflow

on:
  workflow_call:
    inputs:
      ADDITIONAL_MVN_COMMAND_TO_APPLY:
        description: 'Additional OpenRewrite command to apply'
        required: false
        type: string
        default: ''

env:
  PR_BRANCH_NAME: openrewrite/auto-improvements

jobs:
  check-branch:
    name: Check if branch exists
    runs-on: ...
    outputs:
      should_run: ${{ steps.check_branch.outputs.should_run }}

    steps:
      - name: Checkout repository
        uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Check if branch exists
        id: check_branch
        run: |
          if git ls-remote --heads origin ${{ env.PR_BRANCH_NAME }} | grep -q ${{ env.PR_BRANCH_NAME }}; then
            echo "Branch ${{ env.PR_BRANCH_NAME }} already exists. Skipping workflow."
            echo "should_run=false" >> $GITHUB_OUTPUT
          else
            echo "Branch does not exist. Proceeding with workflow."
            echo "should_run=true" >> $GITHUB_OUTPUT
          fi

  openrewrite:
    name: Apply OpenRewrite recommendations

    needs: check-branch
    if: needs.check-branch.outputs.should_run == 'true'
    runs-on: ...

    steps:
      - name: Checkout repository
        uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Set up Java
        uses: actions/setup-java@v4
        with:
          distribution: 'temurin'
          java-version: '21'
          cache: 'maven'
          settings-path: ${{ github.workspace }}

      - name: Run OpenRewrite via Maven
        run: |
          ./mvnw --batch-mode -U org.openrewrite.maven:rewrite-maven-plugin:run \
            -Drewrite.recipeArtifactCoordinates=org.openrewrite.recipe:rewrite-testing-frameworks:RELEASE \
            -Drewrite.activeRecipes=org.openrewrite.staticanalysis.JavaApiBestPractices,org.openrewrite.maven.BestPractices

          ${{ inputs.ADDITIONAL_MVN_COMMAND_TO_APPLY }}

      - name: Create Pull Request
        id: create_pr
        uses: peter-evans/create-pull-request@v7
        with:
          token: ${{ secrets.GITHUB_TOKEN }}
          commit-message: "refactor: apply OpenRewrite recommendations"
          title: "refactor: apply OpenRewrite recommendations"
          add-paths: |
            .
            :!settings.xml
            :!toolchains.xml
          body: |
            This Pull Request was automatically generated by OpenRewrite.

            ## Changes Applied
            The following recipes were applied:
            - `org.openrewrite.maven.BestPractices` (Maven best practices)
            - `org.openrewrite.staticanalysis.JavaApiBestPractices` (Java API best practices)
            ${{ inputs.OPENREWRITE_RECIPES_TO_APPLY }}

            Please review the changes and merge if acceptable.
          branch: ${{ env.PR_BRANCH_NAME }}
          delete-branch: true
          labels: |
            openrewrite
            automated-pr
          draft: false

      - name: PR Status
        run: |
          echo "Pull request creation status: ${{ steps.create_pr.outputs.pull-request-operation }}"
          echo "Pull request number: ${{ steps.create_pr.outputs.pull-request-number }}"

Note:
Several unrelated and infrastructure-specific steps have been removed from the GitHub workflow described above.

Below you can see the steps of the GitHub workflow:

The main steps are:

Run OpenRewrite via Maven
Create Pull Request

This approach has proven to be robust; we've already added new recipes and removed old ones, confirming it works well in different scenarios. You can scope a shared GitHub Action to a team or an entire organization and still provide repository-specific overrides using environment variables. This allows the shared workflow to be as complex as necessary, as long as it remains maintainable.

Our results

We ran 5 distinct recipes on a schedule across our team's repositories.
The workflow was rolled out to 14 services.
In a short time, we have already merged over 40 automated PRs generated by this system.

Final thoughts

In this article, we covered the pain points OpenRewrite solves, what worked (and didn't work) for us, and how we use the open-source version with GitHub Actions to run recipes automatically across all our repositories.

So far, our experience with OpenRewrite has been very positive. It has filled a crucial gap in our toolkit, especially for keeping our Maven pom.xml files clean and consistent.

Check out the OpenRewrite documentation to find a recipe that fits your needs, and feel free to use our GitHub workflow as inspiration for your own automation.

GitHub Innovation Graph analysis

Daniil Roman — Sun, 15 Oct 2023 13:10:53 +0000

GitHub has recently unveiled its Innovation Graph, offering us a high level view of the global tech landscape. With this tool, we can assess how various countries contribute to the global tech community on GitHub. We can also delve into the shifts occurring in different regions, identifying those that are net contributors versus net consumers. Additionally, we can pinpoint the countries with the highest numbers of newly established organizations and new developers signing up for GitHub accounts.

There's a ton of data, so we'll have plenty of charts and a brief overview without digging too deep into each one.

If you want access to a Jupyter Notebook with a more detailed analysis, you can take a look at this repository.

Notes about the data

Certainly, it's important to remember that this data isn't the absolute truth about the current state of the global tech scene. However, we all know that GitHub is a big deal in our tech world. In this article, I'm working on the assumption that this data gives us a good idea of what's happening in the tech world across the globe. Plus, I'll be sharing a few conclusions that strongly support the idea that this data is quite reliable, and we can trust it.

Common numbers

First and foremost, let's begin by examining some key figures. Specifically, we'll focus on the growth of organizations and developers on GitHub, as well as the overall contributions to third countries.

In essence, the count of developers worldwide has seen a significant increase, and GitHub contributions have doubled over the past three years. Additionally, the number of organizations has grown by more than one and a half.

This information provides valuable insights into the overall growth of the global tech sector. It also allows us to identify outliers among countries that have experienced notably lower or higher growth rates compared to these data trends.

Numbers of developers and organisations

Sure, let's dig deeper into the numbers of developers and organizations.

Check out the breakdown by country for the distribution of developers and organizations worldwide.

Charts explanations

You're likely to encounter many similar charts. In these charts, the "absolute" numbers show the percentage of the total (100%) on the Y-axis, with the change in this percentage from 2020 represented on the X-axis.

For example, in the chart below, we can focus on India (IN). This means that currently, approximately 7.5% of all organizations created on GitHub are from India. This number has increased by around 0.5% since 2020.

If the values were relative instead, it would signify, for instance, that 100,000 new organizations were established in India since 2020 (on the Y-axis), with a 5% increase since that year (on the X-axis).

The colors in these charts are used for grouping purposes, although the boundaries between groups can vary widely. They are primarily for visualization and don't carry specific conclusions.

The trends for organisation growth per country:

The trends for developers growth per country:

The main takeaway here is quite straightforward: the number of organizations is increasing at a quicker pace in developed countries, while on the flip side, the number of developers is growing faster in developing countries.

Moreover, the United States, India, and China are undeniably the top leaders in both of these measures. Even though there's a noticeable declining trend in China, it's worth considering the possibility that developers and organizations might be shifting to different Git providers or relocating to other countries.

Contribution and consumption on GitHub per region

In Developed Europe, which includes the EU, the UK, and some other countries, they account for half of the total contributing to other countries. This implies that if a company is considering hiring developers from abroad, it's more likely they'll hire from this region. However, please note that this is a general global observation.

On the flip side, if you're seeking employment in a company from a foreign country, chances are it will be a company from the US or Canada.

In East Asia (encompassing China, India, Hong Kong, Japan, and Singapore), the tech sector is growing rapidly and already has a significant global presence.

Insights per region

In nearly every region, we see just two or three prominent leaders.

Furthermore, these leading figures are gradually losing their dominance in their respective regions, except for India in East Asia and Nigeria in Africa.

On average, the fastest-growing leaders in each region represent no more than 10% contribution of that region.

In contrast, Developed Europe (comprising the EU, UK, and some other countries) stands out as the most diversified region. This suggests that it might be beneficial to further dissect this region into smaller sub-regions for a more detailed analysis.

Contribution on GitHub per country

The two charts below illustrate the rate of growth for each country. They show the relative change in contribution between 2020 and 2023.

The two charts below present the current position in global contribution. They indicate the percentage of the total contribution, representing 100%, and the percentage of change between 2020 and 2023.

For instance, if we examine Canada and India, we notice that India has actually contributed a larger amount in relative terms over the past three years. However, in absolute figures, there is still a gap of 2% when compared to the global total.

Nevertheless, it's important to highlight that 2% of the global contribution is quite significant. This gap could be closed if, for example, the entire tech sector of Italy or Brazil were to relocate to India.

Consumption from third countries

From left to right, each image offers a distinct level of zoom to display the distribution from various perspectives.

Absolute values

Relative growth

The primary insight here is undoubtedly that the US is the predominant employer in this context. The US consumes nearly half of the global contributions on GitHub.

Additionally, Georgia has shown remarkable changes in contributions since 2020, which we'll investigate further in detail shortly.

The rise of Poland, Portugal and Spain in Europe

Indeed, in many charts, these three countries stand out as fast-growing contributors from a contribution point of view. Each of them should be thoroughly researched individually, as there could be entirely different reasons for their remarkable growth.

One possible assumption is that they have benefited from the fact that Europe remains a significant player in the tech sector. And these countries may offer a lower cost of living compared to other developed European nations.

Furthermore, when we focus on Portugal, for example, we can find news about special tax incentives and visas for the tech sector, which have had a substantial impact on its growth. Various articles, such as this one from TechCrunch, support this observation and demonstrate the positive impact of such policies.

China and Honk Kong

We've seen numerous graphs that clearly depict a significant and noticeable decline in China's contribution, while concurrently, Hong Kong has experienced substantial growth.

It's worth noting that reports indicate that companies are shifting from mainland China to Hong Kong, as evidenced in articles like this one.

The charts below display both contribution and consumption figures for several countries in the region, including China and Hong Kong.

Hong Kong has a remarkable level of penetration to closest countries in the region. Additionally, Hong Kong has increased its presence by nearly 100% in Singapore and Taiwan.

Russia

In early 2022, Russia invaded Ukraine, leading to a significant exodus of tech companies and professionals.
The charts presented below could provide insights into where these companies and developers may have resettled. They may also reveal common trends and connections within the Russian tech sector.

Following charts less interesting and generally just confirm assumption above.

This serves as yet another illustration and confirmation of the high mobility within the tech industry, which often surpasses that of other industries.

Georgia

Georgia has become as the primary destination for Russian tech companies and professionals.
Georgia's remarkable growth has made it one of the standout performers on global charts, particularly in terms of fast growth.

In the charts that follow, we can observe the primary connections. What's particularly interesting is that the number of connected countries has expanded significantly, growing from 4 to 36 since 2020.

Contribution to Georgia: total values changed from 3625 in 2020 to 52083 in 2023 = 1336%

Contribution from Georgia: total values changed from 105 in 2020 to 14984 in 2023 = 14170%

US

The subsequent charts reveal the resemblance between U.S. consumption and worldwide contribution to third countries, and they appear to be quite similar. This provides further confirmation of the fact that the U.S. is a major employer in the tech industry.

Conclusions

The information we have mostly deals with countries that are heavily into using GitHub. However, in Russia for example, organizations have been favoring on-premise GitLab for quite a while. So, it's essential to realize that this data doesn't provide a complete picture of the entire tech landscape. But correlation between GitHub data and real world events prove this data to be a reliable source of truth.
It's highly unlikely that the contribution of any country could increase by hundreds of percentage points without the influence of external factors.
The United States is the primary employer in the tech sector, consuming nearly half of all GitHub contributions from third countries.
Simultaneously, Europe stands as the primary activity contributor on GitHub to third countries.
For the most part, this data prove many fairly obvious assumptions, such as the faster growth of organization numbers in developed countries and the quicker increase in developer numbers in developing nations.
The tech economy is highly dynamic, and individuals and companies are quick to relocate when the need arises.
Countries with a lower cost of living have proven to be more attractive to companies, resulting in a greater increase in both contribution and consumption when compared to countries with higher living costs.

How AI helped to chat with my grandma on Telegram

Daniil Roman — Sat, 16 Sep 2023 21:35:26 +0000

Hey everyone 👋

Have you ever wondered how to keep in contact with your elderly grandma who lives in another country? What if she's never used any messaging apps and doesn't even have a smartphone?

Well, in this article, I'm going to share my story about overcoming this challenge 😁 and why I even ended up writing some code for it.

By the end of the article, you'll understand why smart assistants, messengers, and chatbots come into play here.

Idea

I didn't begin with creating anything too complex, of course. Firstly, I simply handed her my old smartphone, where I removed all the icons except for Telegram, which is my primary messaging app. I increased the Telegram icon as much as possible and centered it on the screen. I even opened the dialog with myself there so that she could easily call me with just two clicks.

And... that's it. I'm glad you took the time to read this. I put a lot of effort into this article, so thank you all, and see you next time!👋

But, of course, that wasn't everything I did. In fact, it was just 20% of the work that yielded 80% of the results. But let's delve into the other 80%.

Around that time, I introduced her to a home assistant (Yandex's smart assistant - Alisa), and all of a sudden, she started using it quite extensively. This got me thinking, "What if I could chat with my grandma through this device in some way?"

I personally prefer texting; sending a simple message is 10 times easier and faster than making a call. Plus, I might not always be available when my grandma wants to call, but I can respond to her messages within minutes whenever I have the time.

In conclusion, this idea led me to connect the home assistant (Alisa) with Telegram

System design

Why did I even think it was feasible to implement this in a timely manner? Well, I had no intention of starting from scratch using any programming language and APIs.

No, no, no... "no-code" is my go-to solution in this case.

(low code actually, but who cares)

At that time, I worked at a company where we were developing a low-code platform for chat-bots, and this problem presented itself as the perfect use case to put my skills to the test.

So, after a few iterations, I came up with this “design”.

Each Alisa icon and Telegram icon represents a separate chat-bot.

I opted to use two different chat-bots for Alisa to ensure that I don't have any NLP logic embedded within my chat-bot scenario. Instead, I've dedicated the NLP logic to the higher level within the Alisa framework, rather than incorporating it directly into the chat-bot.

But what does this mean? If I had included NLP logic within my chat-bot, let's call it an app, I would receive plain text messages from my grandma and then route her questions or requests to the appropriate state, handler, or section of my chat-bot's code.

Instead, I rely on Alisa's built-in NLP capability, and when my chat-bot is activated, it signifies that the correct request from my grandma has been successfully recognized. In other words, the right chat-bot wouldn't even be triggered otherwise.

I use Google Sheets as a simple message queue (chosen mainly for its technical convenience). The Telegram bot writes to Google Sheets, and the Alisa reading bot retrieves messages from there.

The Alisa writing bot can communicate directly with the Telegram bot through the platform.

The happy path is quite straightforward, actually.

Here's how it goes: My grandma calls the Alisa chat-bot to send me a message. I received the message in Telegram and responded to her. My response is sent to Google Sheets. Then, my grandma can occasionally call the other Alisa chat-bot to read from Google Sheets. If there are any messages for her, they'll be voiced and then removed from Google Sheets.

Pretty clear, isn't it?

The most exciting thing is that I managed to implement this idea in just one weekend.

The only minor disappointment was that I couldn't add a notification feature in Alisa (home assistant) to alert my grandma when she received a response from me. It seemed there was no way to send such notifications to the home assistant.

JAICP intro

Let's delve a bit deeper, and here's a brief introduction to the platform I built upon.

JAICP is a cloud-based, low-code platform where you can create your chat-bot using a custom DSL mixed with JavaScript code. The DSL defines the structure of the chat-bots, while JavaScript empowers your bots with intelligence.

If you're interested, you can find more information in the documentation. There's tons of useful information covering all the functionalities the platform offers.

What's particularly great in our case is that hosting costs us practically nothing. We'll have exactly one user for each chat-bot, and with such minimal traffic, it won't cost any expenses.

Implementation

We're going to dive deep into the codebase with detailed line-by-line explanations. If this isn't your cup of tea, feel free to skip this part.

Of course, you can find all the code on GitHub.

Naturally, we have to create our bots on the Telegram and Alisa side first in order to get tokens. I won't cover how to configure everything, instead, I will focus only on chat-bots coding part

Let’s have a closer look at Alisa’s bots as they’re more finely-grained compared to the Telegram bot.

Below you’ll find all the code for the writing Alisa bot and yes, it's just 14 lines of code.

theme: /

    state: BabushkaSay
        q!: $regex</start>
        a: What would you like to text to Daniil?

    state: NoMatch
        event!: noMatch
        script:
            var toSendRightNowTime = "2022-07-02T10:00:00";
            $pushgate.createEvent(toSendRightNowTime, "SetBabushkaData", 
                    {"sayData": $request.query}, 
                    "telegram",
                    $secrets.get("TLGRM_BOT_ID"),
                    $secrets.get("TLGRM_USER_ID"));
        a: Message was sent

As I mentioned earlier, I don't include any NLP logic within my chat-bots. This means that when my grandma says, "Alisa, text a message to Daniil," Alisa is responsible for recognition. I only receive the start command on the 4th line and respond with What would you like to text to Daniil? on the next line.

To retrieve all the text my grandma said, I make use of the system variable $request.query.

Additionally, $pushgate.createEvent comes in handy here, allowing us to send the event directly to the Telegram chat-bot. You can find more information about this method in the docs.

There is some configuration happening behind the scenes, but in general, there are only 14 lines of code that handle the task quite effectively.

It's almost the same for the reading Alisa's bot.

theme: /

    state: BabushkaSay
        q!: $regex</start>
        script:
            var danilAnswers = $integration.googleSheets.readDataFromCells(
                $secrets.get("INTEGRATION_ID"),
                $secrets.get("SPREADSHEET_ID"),
                "List1",
                ["A1"]
            );

            if (danilAnswers.length === 0) {
                $reactions.answer("Danil haven't had time to respond yet");
            } else {
                $reactions.answer("Danil is talking");
                var answers = danilAnswers[0]["value"];
                $reactions.answer(answers)

                                $integration.googleSheets.writeDataToCells(
                        $secrets.get("INTEGRATION_ID"),
                        $secrets.get("SPREADSHEET_ID"),
                        "List1",
                        [{value s: [""], cell: "A1"}]
                    );
            }


    state: GoogleSheetError
        a: Couldn't get the data, call to Danil

We read from Google Sheets
If there's anything in Google Sheets, we send a response to Alisa to speak.
Clear the data in Google Sheets

$integration.googleSheets provides functionality to read and write to Google Sheets (probably it’s quite obvious)

And $reactions.answer is used to send a message to the connected channel, which in our case is Alisa.

Here, we do have a few more lines of code, but everything remains at a high level and is quite readable, which is great.

Feedback

So, what happened next you might ask?
Let's recall that my granny was already a “strong” Telegram user at that point. I was excited to introduce her to my project, but before anything else, I had to sell her on the idea, to convince her to give it a try.
I spent just one weekend to implement that and all my emotions were absolutely fresh.

On the day "X", I traveled from another city to visit her. I talked to my grandma about Telegram and the Alisa home assistant to know her thoughts about new stuff. And then, I presented her with this chat-bot and the new way to chat with me anytime.

When I showed my grandma what I had created, I was bursting with pride and eagerly anticipated how she would use it. However, reality hit hard – she refused to use it because she found it too difficult.

The end.

There were two main reasons actually:

The chat-bot cold start
The platform functions much like any other lambda or serverless function, and it experiences a cold start like any other.
While this might not be a significant issue with a high volume of users who use the chat-bot frequently (as it would always be cached), or if it were a text-based chat-bot like the ones on Telegram. However, it presented a challenge in our case.
With only one user and a voice-based chat-bot, there were occasions when the chat-bot took too long to start for my grandma. She became confused when she didn't receive any responses from Alisa and eventually gave up. There were even times when the bot responded with a timeout error, which was a frustrating experience for her. She doesn’t have the mindset that if something didn't work initially, it could be retried or restarted. From her perspective, if something didn't work, it was broken beyond repair.
The Alisa’s bots voice UX
To activate an Alisa bot, you have to say something like "Alisa, enable bot XXX." Then, you have to wait for a moment to receive a response from the chat-bot itself (this is where the chat-bot's cold start occurs). In my case, the chat-bot responds with "What do you want to send to Daniil?" as a welcome phrase. Only after this, my grandma can send the message to me.
This process involved several steps for her, and it was a longer and more complicated journey compared to simply making a phone call.

What could be next steps for this chat-bot system?

So, in the end, we received feedback from customers, and it appears there's no real demand for this product 😅
Or, to put it another way, my grandma simply said she's not going to use this darn thing 🙃

But what could we potentially do to improve the situation if our MVP was to be successful?

Resolving the cold start issue is a fairly manageable task. It would, of course, require writing more code to integrate everything and creating and deploying my own service, which would need to be available 24/7 (perhaps even a very lightweight lambda function in the cloud could suffice). However, the outcome would be a straightforward service where I could store everything, possibly even locally, and seamlessly route messages between Alisa and Telegram, ensuring a smoother user experience.

The second issue is a bit trickier. We have to react to the /start event in any case, even if we don't respond with anything, and only then we can receive the text to send to Telegram.

This means we cannot eliminate the first step entirely and be ready to receive the text as soon as my grandma begins speaking.

We could make an assumption that as soon as the skill is enabled, we are ready to accept text to send because there's almost no cold start. This would eliminate the need to send any response for the /start event to notify my grandma. However, it's important to note that this can only be an assumption and not a strict rule. By making this assumption, there's a potential risk of losing some data.

Nevertheless, there are ways to address this problem and make this idea accessible even for my elderly grandmother.

What actually happened next, you may wonder?

The idea of chatting with grandma through chat-bots faded away, and we went back to good old voice calls via Telegram. However, that wasn't the end of the story 🙃

All of a sudden, she was able to call me, but for some reason, she couldn't answer my calls.
So, here's the latest chapter about how I communicate with grandma over the Internet.

As you might recall, I only have two options: 1 - Telegram and 2 - Alisa home assistant. If Telegram didn't work out, it was time to explore all available options to work with the home assistant, excluding a custom chat-bot.

And there was one option that came to light. I discovered that I could call the home assistant directly through a special messenger designed specifically for this home assistant. You can't call anyone through the home assistant, but you can answer a call and talk through the home assistant, which was quite remarkable. It was something I couldn't even have imagined before.

A quick reminder: The options available to you are, of course, strictly dependent on the ecosystem you are using.

Conclusion

So, in the end, we didn't end up using those chat-bots; we simply call each other. She calls through Telegram, and I call her back through the special messenger, and she answers the call by using the home assistant. This setup still works perfectly for us 😄

Even though the chat-bot idea didn't work out as expected, it was an incredibly interesting and educational journey. I gained hands-on experience in creating and validating a quick MVP. I highly recommend this to everyone, especially software developers. Finding your own ideas, creating, and validating end-to-end MVPs can give you tons of insights about product development in general.

In my case, I understood how crucial UX (User Experience) is and how a cold start can vanish all your efforts.

So, as a final word, I’d like to remind you of a simple and hopefully well-known thought: All aspects of software development are equally important!

And don’t forget to call your grandma 😄

Re2j instead of default regEx in Java: when and how to use it

Daniil Roman — Thu, 30 Jun 2022 12:21:43 +0000

Hi everyone! I’m Daniil and I’m a Java developer at Just AI. In this article, I’m going to tell you how we faced a backtracking issue for regex and how we solved this problem using re2j.
But most importantly, I’ll cover both advantages and disadvantages of this library.

Here’s what you’ll learn:

How we got a problem with backtracking in a regular expression.
How we dealt with it using re2j.
What not obvious issues can you face while using this library?

What is backtracking?

There’s been a lot of discussions around this problem. And for sure, I’m not the first or the last person to raise this question.

In a few words, this problem appears under specific circumstances and causes regex matching to run for a really long time.

Also, the real problem starts when you have no control over the regex your customers create and handling customer data is a very sensitive task .

You can checkout this article for more details and dive deeper into the subject.

How we addressed this issue

We’re developing JAICP - is a conversational AI platform that helps users create chatbots and often have to deal with customer information in conversations with bots.

Of course, customer data can be quite sensitive. But we really care about privacy and obfuscate confidential information.

It looks like this:

The key moment here is that customers can set their own regex to obfuscate data and create a string as a substitution.

Let’s catch problem first

Now that we know what our task is, let’s first take a look at a rather obvious solution with the standard Java regex.

We have quite a common example of obfuscation and it’s also a basic pattern for our case. It is an email obfuscation. Let's substitute them with xxxx@xxxx.xx for example. Here is an email regex pattern for substitution:

[a-z0-9!#$%&'*+/=?^_`{|}~-]+(?:\\.[a-z0-9!#$%&'*+/=?^_`{|}~-]+)*@(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\\.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?

Instead of this regex, customers can write whatever they want.

Let's see how long the following code will run with the standard java engine:

String inputText = TestUtils.getRandomLongWord("test@test.com");
Matcher matcher = Pattern.compile(TestUtils.emailRegex).matcher(inputText);
matcher.replaceAll("XXX@XXX.XX");

In this and the following examples, we’ll generate random text with target injections. In the example above, it’s an email address. It should look something like the following: ...UUID.randomUUID()+"test@test.com"+UUID.randomUUID()
We call randomUUID 500 times and inject target text every 10 calls.
I used an i7-10510U CPU 1.80GHz × 8 machine and jdk 1.8.

Execution time - 36 ms

It’s quite fast. But, let’s see what happens when the target text hasn’t been injected.

The text is the same, but now it looks like this: UUID.random()+UUID.random()

String inputText = TestUtils.getRandomLongWord("");
Matcher matcher = Pattern.compile(TestUtils.emailRegex).matcher(inputText);
matcher.replaceAll("XXX");

Execution time - 8879 ms

🤔 We seem to have a backtracking problem now.

Solution

To fix it, we’ll use the Re2j library.

This library allows for running regexes in linear time. And it doesn’t use a standard NFA approach for a regex engine, but DFA instead.

There are articles like this and this one to help you find your way around it.

It’s important to note that this library was invented exactly for the situations when we don’t know the regex to execute. If you’re facing the same issue, you definitely should give it a go.

But is Re2j so good that it can fix the 9 seconds issue? Let’s see!

To check it, we should just substitute our Pattern with com.google.re2j.Pattern and that’s it.

com.google.re2j.Pattern.compile(emailRegex)
        .matcher(TestUtils.getRandomLongWord(""))
        .replaceAll("XXX")

The new execution time is 88 ms 🔥🔥🔥

Amazing! I’m sure you are with me in wanting to call it a silver bullet and start to use this library everywhere.

But first, let’s take a closer look at a few more things.

What about the performance?

To measure performance, we will use the JMH benchmark. It’s a popular open-source tool for such purposes.

The benchmark body looks the same as in the example above:

@Benchmark
@BenchmarkMode(Mode.AverageTime)
@Fork(value = 1)
@Warmup(iterations = 2)
@Measurement(iterations = 5)
public void standartRegexWithEmailDataBenchmark(MyState myState, Blackhole blackhole) {
    myState.standartEmailRegex.matcher(myState.emailInputTextWords).replaceAll("XXX");
}

standartEmailRegex is java.util.regex.Pattern.compile(TestUtils.emailRegex)

The average replaceAll execution time for the pattern with email injection is:

Default java pattern - 0.004 second
Re2j pattern - 0.002 second

We can see little performance advantages of the re2j over the default pattern. But the main thing is the re2j result wasn’t worse then the default one. And that means we can use it even for call-intensive parts of our program.

What about the big dictionary?

The re2j performs well in practice. But it’s worth checking a case where we have a big dictionary of words to match with the target text.

Let’s take a look at an example with the US cities: Boston, Chicago, Washington...

It transfers to regex like Boston|Chicago|Washington...

Of course, we use different data at work. But as an example, I took a list of the US cities from this website. I got it using the following js code:
Array.from(document.getElementsByClassName("topic-content pt-sm-15")[0].getElementsByTagName("section")).slice(1).flatMap(x => Array.from(x.children[1].children)).map(x => x.outerText)
This resulted in a dictionary with 1,950 items. It’s smaller than the one we have in production but it’s still quite big.

So, we have a default java.util.regex Pattern:

java.util.regex.Pattern.compile(String.join("|", TestUtils.usCitiesAndTowns))

And com.google.re2j Pattern:

com.google.re2j.Pattern.compile(String.join("|", TestUtils.usCitiesAndTowns));

We will use the JMH again.

An average replaceAll execution time for the pattern with cities injection is:

Default java pattern - 0.147 second
Re2j pattern - 0.625 second

It means Re2j is 4 times slower then the default java regex😲

But what about the pattern size?

It’s not an obvious one but let’s check compiled size of our pattern.

Could something like Pattern.compile("\\d{2}") be a problem?

To see, we’ll use ObjectSizeCalculator.getObjectSize from nashorn package. It’s not available in versions newer than JDK 8, but in version 8 it’s quite a good way to measure an object size in runtime.

So, we’ll measure the following code:

Pattern.compile(String.join("|", TestUtils.usCitiesAndTowns)

Where TestUtils.usCitiesAndTowns is that big dictionary from the example above.

The final result should be something like: ObjectSizeCalculator.getObjectSize(Pattern.compile(String.join("|", TestUtils.usCitiesAndTowns)))

com.google.re2j.Pattern - 1026872 bytes (~1mb)
java.util.regex.Pattern - 197608 bytes (~0.2mb)

We got a five-fold difference!!! 🦾

The re2j pattern spends 5x more memory then the default one.

Even our big but not gigantic dictionary uses up to 1 mb of memory. But the real-life dictionary for such cases could be much bigger. And it could be duplicated to several services and instances. So, our pattern could take a lot of memory.

Conclusion

We’ve discussed the re2j from different sides and mentioned some obscure but important disadvantages.

To sum it all up:

The main goal of this library is to solve long execution issues for cases with regex coming from the outside.

When using this library make sure to test it on your data and for your cases because:

There could be opposite performance results.
The Compiled Pattern could use more memory which may cause problems.
Re2j doesn’t support all sets of regex abilities from the default java implementation.

P.S.
The entire piece of code is available on github.

Maven plugin to check dependency versions

Daniil Roman — Wed, 22 Jun 2022 06:22:49 +0000

Hi everyone! In this post I’m gonna show you why you have to check your dependency versions in your maven project and how you can do it.

Problem

Some time ago I got the bug at work that forced me to spend some hours to understand what's going on. And when I finally realised the cause I was surprised, the bug was because we had dynamic changing version for some dependency in our maven project. This dependency had the LATEST version.

And I had a thought: Why we don’t have step in our ci pipeline to check such issues? And Why I haven't even heard about such tool?

So In that moment I decided to find or create maven plugin to fail our build if we have dynamic changing versions of dependencies. Such as LATEST, RELEASE or something like [1.5,)

When could such a case happen? Just accidentially. Or by copy pasting the code. Of course it should be detected in code review. But it could easilly miss in the fast development step.

I can't imagine the situation when such dynamic changing versions could be helpful. If you have one, share this in the comments.

Why maven?

There are not only maven to build java projects. Gradle is on the way to mass adoption. And some things we can make really easily with Gradle comparing to maven. But maven is still the most popular one. Specially for big and old enterprise companies. And when I got this problem we used maven for our projects. So it is why I'm showing how to solve it using maven.

Research

I started to search existing plugins. And surprising for me I didn’t find them.
The most closest to what I wanted was requireUpperBoundDeps rule.

But the <version>LATEST</version> not a problem for a verification of this plugin rule. The plugin just verify dependency has the latest version that exists.

If 1.5 version of dependecy available and you run mvn install with this plugin, project’ll be successfully builded. But If you set 1.4 explicitly, build’ll failed.

So it totally is not what I wanted to achieve.

And also I found exactly what I was looking for but for plugins. It is a requirePluginVersions rule.

Idea

Idea is simple. Let's have regexes about incorrect dependencies versions to match. And we will just check every dependency’s version. And if it matches with any regex we’ll throw the exception to fail build.

Implementation

To implement this idea we can make maven plugin from scratch. Maven has ability to make it from maven-core library. You can find good tutorial how to make it here.

But there much more actions to make simple strings check by regex. Do we have more simple way to do it or not?

The answer is yes. We have a maven enforcer plugin where we can add custom rules.

You can find instructions how to do this here

The following code is just enough:

MavenProject mavenProject = EnforcerRuleUtils.getMavenProject(helper);
List<Dependency> dependencies = mavenProject.getDependencies();

for (Dependency d : dependencies) {
    String version = d.getVersion();
    boolean isRestrictedVersion = restrictedVersionRegexes.stream().anyMatch(p -> p.matcher(version).find());
    if (isRestrictedVersion || restrictedVersionStrings.contains(version)) {
        throw new EnforcerRuleException(String.format("Dependency %s has restricted version %s.", d, version));
    }
}

Using relates to surface plugin. You can add specific dependency and custom rule to use this functionality.

It looks like this:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-enforcer-plugin</artifactId>
    <version>3.0.0</version>
    <dependencies>
        <dependency>
            <groupId>io.github.DaniilRoman</groupId>
            <artifactId>dependency_restricted_version_checker</artifactId>
            <version>1.0</version>
        </dependency>
    </dependencies>
    <executions>
        <execution>
            <id>enforce</id>
            <configuration>
                <rules>
                    <myCustomRule implementation="io.github.daniilroman.versionchecker.RestrictDependencyVersionRule">
                        <enabled>true</enabled>
                    </myCustomRule>
                </rules>
            </configuration>
            <goals>
                <goal>enforce</goal>
            </goals>
        </execution>
    </executions>
</plugin>

And that's all. Now when you run mvn install it run enforcer plugin that run this custom rule.

As a result if your junior developer add such a dangerous change

For example:

<dependency>
    <groupId>org.testng</groupId>
    <artifactId>testng</artifactId>
    <version>RELEASE</version>
    <scope>compile</scope>
</dependency>

and you miss this in code review stage. Such checker’ll help you to protect the codebase.

Summary

In this article we showed why checking dependency versions can helpful. How to implement it for maven using enforcer plugin and how to add it in your projects.

That's all for today, stay tuned and don’t forget to check your code.

P.S. You can find the whole code on GitHub

Designing an application with Redis as a data store. What? Why?

Daniil Roman — Sat, 30 Apr 2022 07:57:21 +0000

1) Introduction

Hello everyone! Many people know what Redis is, and if you don’t know, the official site can bring you up to date.
For most Redis is a cache and sometimes a message queue.
But what if we go a little crazy and try to design an entire application using only Redis as data storage? What tasks can we solve with Redis?
We will try to answer these questions, in this article.

What we won't see here?

Every Redis data structure in detail won't be here. For what purposes you should read special articles or documentation.
Here will also be no production-ready code that you could use in your work.

What we'll see here?

We'll use various Redis data structures to implement different tasks of a dating application.
Here will be Kotlin + Spring Boot code examples.

2) Learn to create and query user profiles.

For the first, let's learn how to create user profiles with their names, likes, etc.

To do this, we need a simple key-value store. How to do it?

Simply. A Redis has a data structure - a hash. In essence, this is just a familiar hash map for all of us.

Redis query language commands can be found here and here.
The documentation even has an interactive window to execute these commands right on the page. And the whole commands list can be found here.
Similar links work for all subsequent commands that we will consider.

In the code, we use RedisTemplate almost everywhere. This is a basic thing for working with Redis in the Spring ecosystem.

The one difference from the map here is that we pass "field" as the first argument. The “field” is our hash's name.

fun addUser(user: User) {
        val hashOps: HashOperations<String, String, User> = userRedisTemplate.opsForHash()
        hashOps.put(Constants.USERS, user.name, user)
    }

fun getUser(userId: String): User {
        val userOps: HashOperations<String, String, User> = userRedisTemplate.opsForHash()
        return userOps.get(Constants.USERS, userId)?: throw NotFoundException("Not found user by $userId")
    }

Above is an example of how it might look in Kotlin using Spring's libraries.

All pieces of code from that article you can find on Github.

3) Updating user likes using Redis lists.

Great!. We have users and information about likes.

Now we should find a way how to update that likes.

We assume events can happen very often. So let's use an asynchronous approach with some queue. And we will read the information from the queue on a schedule.

Redis has a list data structure with such a set of commands. You can use Redis lists both as a FIFO queue and as a LIFO stack.

In Spring we use the same approach getting ListOperations from RedisTemplate.

We have to write to the right. Because here we are simulating a FIFO queue from right to left.

fun putUserLike(userFrom: String, userTo: String, like: Boolean) {
        val userLike = UserLike(userFrom, userTo, like)
        val listOps: ListOperations<String, UserLike> = userLikeRedisTemplate.opsForList()
        listOps.rightPush(Constants.USER_LIKES, userLike)
}

Now we're going to run our job on schedule.

We are simply transferring information from one Redis data structure to another. This is enough for us as an example.

fun processUserLikes() {
        val userLikes = getUserLikesLast(USERS_BATCH_LIMIT).filter{ it.isLike}
        userLikes.forEach{updateUserLike(it)}
}

User updating is really easy here. Give a hi to HashOperation from the previous part.

private fun updateUserLike(userLike: UserLike) {
        val userOps: HashOperations<String, String, User> = userLikeRedisTemplate.opsForHash()
        val fromUser = userOps.get(Constants.USERS, userLike.fromUserId)?: throw UserNotFoundException(userLike.fromUserId)
        fromUser.fromLikes.add(userLike)
        val toUser = userOps.get(Constants.USERS, userLike.toUserId)?: throw UserNotFoundException(userLike.toUserId)
        toUser.fromLikes.add(userLike)

        userOps.putAll(Constants.USERS, mapOf(userLike.fromUserId to fromUser, userLike.toUserId to toUser))
    }

And now we show how to get data from the list. We are getting that from the left. To get a bunch of data from the list we will use a range method.
And there is an important point. The range method will only get data from the list, but not delete it.

So we have to use another method to delete data. trim do it. (And you can have some questions there).

private fun getUserLikesLast(number: Long): List<UserLike> {
        val listOps: ListOperations<String, UserLike> = userLikeRedisTemplate.opsForList()
        return (listOps.range(Constants.USER_LIKES, 0, number)?:mutableListOf()).filterIsInstance(UserLike::class.java)
            .also{
listOps.trim(Constants.USER_LIKES, number, -1)
}
}

And the questions are:

How to get data from the list into several threads?
And how to ensure the data won't lose in case of error? From the box - nothing. You have to get data from the list in one thread. And you have to handle all the nuances that arise on your own.

4) Sending push notifications to users using pub/sub

Keep moving forward!
We already have user profiles. We figured out how to handle the stream of likes from these users.

But imagine the case when you wanna send a push notification to a user the moment we got a like.
What are you gonna do?

We already have an asynchronous process for handling likes, so let's just build sending push notifications into there. We will use WebSocket for that purpose, of course. And we can just send it via WebSocket where we get a like. But what if we wanna execute long-running code before sending? Or what if we wanna delegate work with WebSocket to another component?
We will take and transfer our data again from one Redis data structure (list) to another (pub/sub).

fun processUserLikes() {
        val userLikes = getUserLikesLast(USERS_BATCH_LIMIT).filter{ it.isLike}
                pushLikesToUsers(userLikes)
        userLikes.forEach{updateUserLike(it)}
}

private fun pushLikesToUsers(userLikes: List<UserLike>) {
  GlobalScope.launch(Dispatchers.IO){
        userLikes.forEach {
            pushProducer.publish(it)
        }
  }
}

@Component
class PushProducer(val redisTemplate: RedisTemplate<String, String>, val pushTopic: ChannelTopic, val objectMapper: ObjectMapper) {

    fun publish(userLike: UserLike) {
        redisTemplate.convertAndSend(pushTopic.topic, objectMapper.writeValueAsString(userLike))
    }
}

The listener binding to the topic is located in the configuration.
Now, we can just take our listener into a separate service.

@Component
class PushListener(val objectMapper: ObjectMapper): MessageListener {
    private val log = KotlinLogging.logger {}

    override fun onMessage(userLikeMessage: Message, pattern: ByteArray?) {
        // websocket functionality would be here
        log.info("Received: ${objectMapper.readValue(userLikeMessage.body, UserLike::class.java)}")
    }
}

5) Finding the nearest users through geo operations.

We are done with likes. But what about the ability to find the closest users to a given point.

GeoOperations will help us with this. We will store the key-value pairs, but now our value is user coordinate. To find we will use the [radius](https://redis.io/commands/georadius) method. We pass the user id to find and the search radius itself.

Redis return result including our user id.

fun getNearUserIds(userId: String, distance: Double = 1000.0): List<String> {
    val geoOps: GeoOperations<String, String> = stringRedisTemplate.opsForGeo()
    return geoOps.radius(USER_GEO_POINT, userId, Distance(distance, RedisGeoCommands.DistanceUnit.KILOMETERS))
        ?.content?.map{ it.content.name}?.filter{ it!= userId}?:listOf()
}

6) Updating the location of users through streams

We implemented almost everything that we need. But now we have again a situation when we have to update data that could modify quickly.

So we have to use a queue again, but it would be nice to have something more scalable.

Redis streams can help to solve this problem.
Probably you know about Kafka and probably you even know about Kafka streams, but it isn't the same as Redis streams. But Kafka itself is a quite similar thing as Redis streams. It is also a log ahead data structure that has consumer group and offset. This is a more complex data structure, but it allows us to get data in parallel and using a reactive approach.

See the Redis stream documentation for details.

Spring has ReactiveRedisTemplate and RedisTemplate for working with Redis data structures. It would be more convenient for us to use RedisTemplate to write the value and ReactiveRedisTemplate for reading. If we talk about streams. But in such cases, nothing will work.
If someone knows why it works this way, because of Spring or Redis, write in the comments.

fun publishUserPoint(userPoint: UserPoint) {
    val userPointRecord = ObjectRecord.create(USER_GEO_STREAM_NAME, userPoint)
    reactiveRedisTemplate
        .opsForStream<String, Any>()
        .add(userPointRecord)
        .subscribe{println("Send RecordId: $it")}
}

Our listener method will look like this:

@Service
class UserPointsConsumer(
    private val userGeoService: UserGeoService
): StreamListener<String, ObjectRecord<String, UserPoint>> {

    override fun onMessage(record: ObjectRecord<String, UserPoint>) {
        userGeoService.addUserPoint(record.value)
    }
}

We just move our data into a geo data structure.

7) Count unique sessions using HyperLogLog.

And finally, let's imagine that we need to calculate how many users have entered the application per day.
Moreover, let's keep in mind we can have a lot of users. So, a simple option using a hash map is not suitable for us because it will consume too much memory. How can we do this using fewer resources?
A probabilistic data structure HyperLogLog comes into play there. You can read more about it on the Wikipedia page. A key feature is that this data structure allows us to solve the problem using significantly less memory than the option with a hash map.

fun uniqueActivitiesPerDay(): Long {
    val hyperLogLogOps: HyperLogLogOperations<String, String> = stringRedisTemplate.opsForHyperLogLog()
    return hyperLogLogOps.size(Constants.TODAY_ACTIVITIES)
}

fun userOpenApp(userId: String): Long {
    val hyperLogLogOps: HyperLogLogOperations<String, String> = stringRedisTemplate.opsForHyperLogLog()
    return hyperLogLogOps.add(Constants.TODAY_ACTIVITIES, userId)
}

8) Conclusion

In this article, we looked at the various Redis data structures. Including not so popular geo operations and HyperLogLog.
We used them to solve real problems.

We almost designed Tinder, it is possible in FAANG after this)))
Also, we highlighted the main nuances and problems that can be encountered when working with Redis.

Redis is a very functional data storage. And if you already have it in your infrastructure, it can be worth looking at Redis as a tool to solve your other tasks with that without unnecessary complications.

PS:
All code examples can be found on github.

Write in the comments if you notice a mistake.
Leave a comment below about such a way to describe using some technology. Do you like it or not?

And follow me at Twitter:🐦@de____ro

How we built Elasticsearch index

Daniil Roman — Thu, 28 Apr 2022 12:31:42 +0000

What we wanted to do

Hey everyone. I’m Daniil and I work at Just AI where we are building a chatbot development platform. Our platform has its own DSL that enables users to create a chatbot scenario with less effort. That DSL helps describe bot behavior and enrich your bot with complex logic using javascript. Chatbot developers who work on our platform use our web IDE which supports this DSL.

Bot scenarios can have a lot of files. And of course, you’ll need to search those files to find information.
Let's take a look at what kind of search exactly we’ll get. To put it short, we wanted to get an IDE-like search. Where you can find a result using not only a part of the word but also regex, or a whole word match. And case sensitivity also should be there.

Here’s what we got:

What you will and won’t find in this article

Here you’ll discover how to build an index we finally built. Some knowledge of Elasticsearch will help you better understand this article.

While I’ll also explain a couple of Elasticsearch concepts, this article is definitely not a tutorial of Elasticsearch.

Why Elasticsearch?

You’ll probably ask “Why Elasticsearch exactly?”. But the answer is really simple. Our team just didn’t have enough experience with search engines . So the decision to choose the most popular one was really straightforward and obvious. Furthermore, our operations team had experience in maintaining Elasticsearch for Graylog. Almost everyone knows about Elasticsearch. It is the most popular search engine for searching logs and there are a lot of tutorials and articles about that search engine on the Internet.

How do we store our files in source datastore?

As I mentioned above, originally our data was programming code. For example js files and sc files that you can see above. So let's take a look at our source files in detail. We store our data in MongoDB and every single file is a separate document in the MongoDB collection.

Let’s look at an example:

theme: /

    state: Start
        q!: $regex</start>
        a: Let's start.

    state: Hello
        intent!: /hello
        a: Hello hello

    state: Bye
        intent!: /bye
        a: Bye bye

    state: NoMatch
        event!: noMatch
        a: I do not understand. You said: {{$request.query}}

MongoDB structure looks like that:

{
    "fileName": String,
    "content": BinData
}

And data in the specific document looks like that:

{
  "fileName": "main.sc",
    "content":"ewogICAgInByb2plY3QiO...0KICAgIF0KfQ=="
}

Now we understand our data and our goal. So the main question of this article is: Should we convert our original collection structure for Elasticsearch or not?

How to migrate data to Elasticsearch?

We have different tools to help us migrate our data to Elasticsearch. For example, Logstash is a good and time-proven tool to synchronize our data from different sources with Elasticsearch. You can also transform and filter data in this pipeline.

Pros:

Well-known and time-proven product.
You have to write only the config file without implementing all sync logic.
It is an external process that you can scale separately.

Cons:

Not obvious how complex the logic you can write with a config file.
Separation sync and transformation logic out of our main codebase could make our whole logic difficult to understand and maintain.

Logstash is a really great tool but in our case, it looked too complicated and had too many uncertainties. It didn’t seem a good fit.

So we decided to migrate our data and transfer our source collection structure to the index structure by ourselves.

The first try

Since we need to find the line number and match the position in line, we have only two options. The first one: every document is a single line, hence every document is small. The second one: every document is a whole file as origin in MongoDB but this document has a list of lines with the list’s numbers.

So now let’s start with a quick explanation on how Elasticsearch works. The heart of Elasticsearch is an inverted index. In other words, if you get a match you’ll get a whole matching document. That’s why we have to store some additional information to understand what line of the document we have for example.

Since we store the whole file in MongoDB, we decided to make the same for the first step in Elasticsearch. The main point was it would allow for less memory consumption compared to another case.

In this case, our Elasticsearch mappings look like this:

{
    "files_index": {
        "mappings": {
            "properties": {
                "fileName": {
                    "type": "keyword"
                },
                "lines": {
                    "type": "nested",
                    "properties": {
                        "line": {
                            "type": "text",
                            "analyzer": "ngram_analyzer"
                        },
                        "lineNumber": {
                            "type": "integer"
                        }
                    }
                }
            }
        }
    }
}

Thelines field is important for us. This field has a nested type, which has a warning on google search “don’t use this type” or “don’t make big nested fields”. So... let’s break this rule and do both )

In Elasticsearch file from the top of the article looks like this:

{
  "fileName": "main.sc",
    "lines": [
                    {"line": "require: slotfilling", "lineNumber": 1},
                    ...
                    {"line": "        a: I do not understand. You said: {{$request.query}}", "lineNumber": 19}
  ]
}

And it worked!

But... always there is a “but”.

Index updating could be quite often a task. And because a document in the index could be huge we had a lot of HTTP calls to Elasticsearch that failed by timeout in 30 seconds.

So looks like we got that problem from warnings) and we had to fix it.

Let’s make our index smaller

The option, where the document is a file, didn't work. So we have to choose the second option and make a document a line of the file.

In this case, we have another index structure:

{
    "files_index": {
        "mappings": {
            "properties": {
                "fileName": {
                    "type": "keyword"
                },
                "line": {
                    "type": "text",
                    "analyzer": "ngram_analyzer"
                },
                "lineNumber": {
                    "type": "integer"
                }
            }
        }
    }
}

And our document looks like this:

{
  "fileName": "main.sc",
    {
    "line": "require: slotfilling", 
        "lineNumber": 1
  }
}

Now Elasticsearch has started to work faster and more stable. And it helped to maximize the number of calls to Elasticsearch. Our initial fear of memory consumption for this case didn't materialize. Our index size almost hasn’t changed.

We described only our index structure but not a search query.

Default search

Let’s look at the example:

You have a document in Elasticsearch - “hello world”.

You have default settings and we want to find this document by searching “hel”. It looks like a real case doesn’t it?

So, in this case you’ll get nothing.

All because of such things as analyzer and tokenizer. You preprocess query string and index data using those things. And if you have an incorrect analyzer for example you won’t get a match and will get nothing. Default behavior is splitting text by spaces and special characters. So, that’s why you’ll get nothing by “wor”, but you’ll get what you want by “world”.

How can we find what we want using part of a word?

There is a way to become familiar with ngrams in Elasticsearch. This article on gitlab gave us confidence we found what we were looking for.

Ngram - is an ngram analyzer in Elasticsearch. You can set it in mappings for a field.

Example:

Save string “hello world” to index. For example we have min=3 and max=5 in our ngram analyzer settings.
It means we split text by 3, 4 and 5 symbols.
hel, ell, llo, lo , o w, ..., rld, ..., o wor, worl, world
And if we get a match with any of those substrings we’ll get a “hello world”.

So that’s how we did. Above you can see how we applied the ngran analyzer for the line field.

"line": {
  "type": "text",
  "analyzer": "ngram_analyzer"
}

But... Here comes another “but”.

It is a really good choice. Everything works well and fast. The only issue is that our index is going to take over a really big amount of memory.

We started looking for a solution to this issue and found it.

Wildcard

Wildcard is a kind of construction: hel.
It is also searching by substring but it is much faster than a plain regex. It works by special combination of ngram and regex and helps to find a compromise between query speed and index memory consumption. You can use this analyzer for index filed as well as for query strings.

And our final index structure looks like this:

{
    "files_index": {
        "mappings": {
            "properties": {
                "fileName": {
                    "type": "keyword"
                },
                "line": {
                    "type": "wildcard"
                },
                "lineNumber": {
                    "type": "integer"
                }
            }
        }
    }
}

Unlike ngramm analyzer we reduced our index memory consumption by some 4-5 times!

We tested results that weren't from our production data and production environment. We tested it on synthetic generated data. Data size was about 1 gigabytes.
First of all we executed POST /<index>/_forcemerge for index.
And after that using GET /_cat/indices/<index> we could see the size.

We don't have a big index at all (it’s about only 10gb) so it works really well for us. Almost every call to Elasticsearch is less than 0.1 second. But if you have a really huge Elasticsearch index this solution could be somewhat slow for you. But for us and most of such cases it is a perfect match.

Summary

In this article, we went from from an Elasticsearch newbies to understanding pros and cons of different solutions and using it in real cases.

I hope it was helpful and this article could save someone time and effort. And also I hope this article shows that Elasticsearch isn’t a black box that searches for something but it is more like a Lego constructor.

We still have so much more to tell t, because here we only talked about the structure of the index, but did not tackle solving all other tasks in order to get a search like in an IDE using Elasticsearch.

DEV Community: Daniil Roman

Automate Your Java Upgrades: A Practical Case Study with OpenRewrite and GitHub Actions

What is OpenRewrite?

When would you use OpenRewrite?

Our experience of using OpenRewrite

What worked for us

Keeping pom.xml in shape

Refactoring test libraries

What didn't work

Complex, custom refactoring

Running Java recipes on Kotlin projects

Newest recipes are often commercial

Our automation setup with GitHub Actions

Why not just use the Maven plugin?

The GitHub workflow in detail

Our results

Final thoughts

GitHub Innovation Graph analysis

Common numbers

Numbers of developers and organisations

Contribution and consumption on GitHub per region

Insights per region

Contribution on GitHub per country

Consumption from third countries

Absolute values

Relative growth

The rise of Poland, Portugal and Spain in Europe

China and Honk Kong

Russia

Georgia

US

Conclusions

How AI helped to chat with my grandma on Telegram

Idea

System design

JAICP intro

Implementation

Feedback

What could be next steps for this chat-bot system?

What actually happened next, you may wonder?

Conclusion

Re2j instead of default regEx in Java: when and how to use it

What is backtracking?

How we addressed this issue

Let’s catch problem first

Solution

What about the performance?

What about the big dictionary?

But what about the pattern size?

Conclusion

Maven plugin to check dependency versions

Problem

Why maven?

Research

Idea

Implementation

Summary

Designing an application with Redis as a data store. What? Why?

1) Introduction

What we won't see here?

What we'll see here?

2) Learn to create and query user profiles.

3) Updating user likes using Redis lists.

4) Sending push notifications to users using pub/sub

5) Finding the nearest users through geo operations.

6) Updating the location of users through streams

7) Count unique sessions using HyperLogLog.

8) Conclusion

How we built Elasticsearch index

What we wanted to do

What you will and won’t find in this article

Why Elasticsearch?

How do we store our files in source datastore?

How to migrate data to Elasticsearch?

The first try

Let’s make our index smaller

Default search

How can we find what we want using part of a word?

Wildcard

Summary

Keeping `pom.xml` in shape