DEV Community: Bohdan Stupak

Partitioning PotgreSQL Database

Bohdan Stupak — Tue, 17 Mar 2026 10:02:50 +0000

Introduction

In one of the projects I used to work on, we've employed CQRS approach with PostgreSQL as a write storage and NoSQL database as a read storage. As a safety measure, we had a special endpoint that allowed us to regenerate entire content of the read storage based on the write storage that is supposed to be the single source of truth.

The story started one day when we discovered that our safety handle started to fail with the write storage database timeout due to several millions of rows in it. Generally speaking, the solution proposed below is applicable to a wide range of data storages but we'll touch upon some Postgres-specific tooling, so let's focus on that.

Partitioning overview

Partition is the process of splitting a dataset into more manageable, smaller pieces. This, in turn yeilds reduced INSERT and SELECT costs. Partitioning can be accomplished in many ways. For instance, vertical partitioning separates data into columns. Horizontal partition, on the contrary, divides data into rows according to particular criteria (partition key). When choosing partition key we can use the following strategies:

range partitioning (splitting by date or numeric ranges)
list partitioning (splitting by categorical values like region)
hash partitioning (even distribution using a hash function)

Partitioning vs sharding

While partitioning occurs within a single database instance, sharding is a way to split the data into multiple physical instances. Thus, sharding increases operational costs due to such issues as the necessity to put the proxy, which will route requests to shards, need in distributed transactions or expensive cross-shards joins.

For that reason, we should employ sharding only when partitions exceed the limits of a single machine storage.

Choosing the right partitioning strategy

So our plan is to split the write storage into some manageable chunks so that none of those chunks will fail due to a timeout. How do we design those chunks?

The code of the safety handle reads items from the read storage in batches. I already blogged about how batching can be a neat solution for handling a large dataset.
So ideally, we would like to have our partition equal to or several times bigger than the batch size, so we could read the entire partition with an integer number of iterations.

For that reason, we have to discard the idea of list partition or range partition based on serial data (such as timestamps), as they generally offer uneven distribution of data.

The other two options are hash partitioning and range partitioning based on an integer primary key. Since both of them offer even distribution, we'll stick with a simpler range-based approach.

Using pg_partman

The go-to solution for partitioning in PostgreSQL is pg_partman extension.

First, let's enable the extension.

CREATE SCHEMA partman;
CREATE EXTENSION pg_partman WITH SCHEMA partman;

To partition existing data, we'll create a new partitioned table with the desired structure and migrate data from the old table to the new one.

ALTER TABLE public."Entities" RENAME CONSTRAINT "PK_Entities" TO "PK_Entities_old";

ALTER TABLE public."Entities" RENAME TO "Entities_old";

CREATE TABLE IF NOT EXISTS public."Entities"
(
    "Id" bigint GENERATED ALWAYS AS IDENTITY,
    "UserName" character varying(255) COLLATE pg_catalog."default" NOT NULL,
    "TimeStamp" timestamp with time zone NOT NULL,
    "Entity" jsonb,
    "OrganizationInfo" text COLLATE pg_catalog."default",
    CONSTRAINT "PK_Entities" PRIMARY KEY ("Id")
)  PARTITION BY RANGE ("Id");

Now we have to figure out how many partitions to create. To achieve this, we have to do some math. We'll take the number of existing rows, and given the row count in a single partition, we'll calculate how many partitions we need.

DO $$
DECLARE
    v_row_count bigint;
    v_partition_size bigint := 200000;
    v_premake integer;
BEGIN
    SELECT count(*) INTO v_row_count
    FROM public."Entities_old";

    v_premake := ceil(v_row_count::numeric / v_partition_size) + 1;

    PERFORM partman.create_parent(
        p_parent_table := 'public.Entities',
        p_control      := 'Id',
        p_interval     := v_partition_size::text,
        p_premake      := v_premake
    );
END $$;

A caveat here is the parameter p_premake that plays an important role, as it represents the maximum number of partitions that are available in advance when performing partitioning. By default, this parameter is 5, which means if we have 10 000 000 rows in the original table and we want to configure partitions 200 000 rows each, we'll end up with 5 partitions holding together 1 000 000 rows and another 9 000 000 rows residing in the default partition!

p_premake is rarely mentioned in most of the examples dedicated to pg_partman, as they use quite a modest amount of data for the illustration, so I spent a lot of time debugging and researching until I discovered it.

Before we wrap up with transferring data into the new table, it should be noted that pg_partman does not create new partitions automatically. This means that if we have 5 partitions with 200 000 rows each, when we insert row number 1 000 001, the 6th partition won't be created. Instead new record will fall into the default partition. For that reason, we have to perform partition maintenance, which is a built-in function of pg_partman.

To achieve this, we'll install pg_cron extension, which initiates an internal scheduler.

CREATE EXTENSION pg_cron;

SELECT cron.schedule('@hourly', $$CALL partman.partition_data_proc('public.Entities');$$);

Calling partition_data_proc allows us to create a new partition and remove data from the default partition, if any.

With that done, the last step is migration to a new table, which is pretty straightforward.

INSERT INTO public."Entities"(
    "UserName", "TimeStamp", "Entity", "OrganizationInfo")
SELECT "UserName", "TimeStamp", "Entity", "OrganizationInfo"
FROM public."Entities_old"
ORDER BY "Id";

DROP TABLE IF EXISTS public."Entities_old"

Assessing the result

Before partitioning, we had 9 million rows in the table mentioned above. Selecting a single batch from it could take up to 30 seconds. Now, with a little bit of SQL glue, we were able to reduce this time to the fractions of a second, yielding orders of magnitude improvement.

Using RAG architecture for generative tasks

Bohdan Stupak — Wed, 28 May 2025 13:26:38 +0000

Large language models are used widely across the industry these days. Yet still, many people are skeptical about their capabilities as they are quite prone to hallucination. For that reason, in this article, I decided to use LLM in a case where there are no incorrect answers: generating artistic text. However, even in the case of art which is highly subjective, there are still some quality gates (or shall I say personal preferences?).

So how do I trick a model into generating text that suits my artistic taste? While prompt engineering can get you this far, I decided to provide LLM with data that would represent my view of fine art. For this, purpose I decided to try out Retrieval Augmented Generation (RAG) architecture.

You can take a look at the entire code in this repository

Enter RAG

RAG is a technique that combines a large language model with a vector database. The idea is that the model will be able to retrieve relevant information from a database and use it to generate a response. One of the most prominent use cases is letting LLM operate across the knowledge base thus creating an expert system.

So how does it work?

It consists of the following stages:

Indexing. The data we want to use as our knowledge base is converted into LLM embeddings: numerical representations in the form of vectors.
Retrieval. Given a user query, a document retriever is first called to select the most relevant documents that will be used to augment the query.
Augmentation. The model feeds this relevant retrieved information into the LLM via prompt engineering of the user's original query.
Generation. Finally, the LLM generates output based on both the query and the retrieved documents.

Indexing data

To be able to use LLM locally, I used Ollama. I won't dive deep into its setup as you can get quite detailed info on their site.

As I don't expect big dataset for my task, I'll embed data of interest in memory.

open Microsoft.KernelMemory.AI.Ollama
open Microsoft.KernelMemory
open Microsoft.SemanticKernel

let ollamaUri = "http://localhost:11434/"
let chatModelName = "llama3.1"
let embedderModelName = "all-minilm"

let kernelMemoryConfig =
    let config = new OllamaConfig()
    config.Endpoint <- ollamaUri
    config.TextModel <- new OllamaModelConfig(chatModelName)
    config.EmbeddingModel <- new OllamaModelConfig(embedderModelName, 2048)
    config

let memory =
    (new KernelMemoryBuilder())
        .WithOllamaTextGeneration(kernelMemoryConfig)
        .WithOllamaTextEmbeddingGeneration(kernelMemoryConfig)
        .Build()

Now, once I set up memory kernel, I can embed data

let importPreparedData =
    async {
        let files = Directory.GetFiles(poemsPath, "*", SearchOption.AllDirectories)
        for j in 1..files.Length do
            let text = File.ReadAllText(files.[j-1])
            let! importResult = memory.ImportTextAsync(text) |> Async.AwaitTask
            Console.WriteLine(importResult)
    }

Generating actual text looks like this

let generateText =
    async {
        do! importData
        let! finalPoem = (memory.AskStreamingAsync(propmpt)
        |> TaskSeq.fold
            (fun acc answer ->
                Console.WriteLine(answer.RelevantSources.Count)
                Console.WriteLine(answer.Result)
                acc + answer.Result
            ) "" |> Async.AwaitTask)
        return Utils.truncateText finalPoem 100
    }

Enriching dataset

As often happens, results in the domain of machine learning depend on the quality and quantity of data provided. This case is not an exception. With the small dataset result looks chunky and often repetitive, as it generates data based solely on what's embedded in memory.

Honestly, as this domain is quite new to me this was a big revelation to me, as I expected LLM to use not only the data provided but also its own generative capabilities.

So how do I add more variativity? With generating synthetic data and embedding it alongside the original dataset.

let rawLlama = Kernel.CreateBuilder().AddOllamaTextGeneration(chatModelName, new Uri(ollamaUri)).Build()

let importSyntheticData =
    async {
        for i in 1..syntheticDataCount do
            let! builtInPoem = rawLlama.InvokePromptAsync(syntheticPropmpt) |> Async.AwaitTask
            let! importResult = memory.ImportTextAsync(builtInPoem.ToString()) |> Async.AwaitTask
            Console.WriteLine(importResult)
    }

let importData =
    async {
        do! importSyntheticData
        do! importPreparedData
    }

Prompt engineering

Even when one has a good enough dataset at your disposal, experience with LLM still depends greatly on how well the question is formulated.

I found a good metric to control it is relevant document count that comes together with LLMs response. Maximizing this value allowed me to find the prompt that utilized most of the data I provided.

While initially, I used one prompt for both embedding synthetic data and generating the final output, eventually, I found that it's better to use different prompts for each task. While generating synthetic data I use the most restrictive prompt I have, in order to guide LLM to generate something that will be on par with seeded data.

On the other hand, the prompt for generating the final output is quite relaxed, since I applied all the restrictions I wanted during the generation phase, so now I want to squeeze the most from the data I have.

Conclusion

While subject to fierce debate from the philosophical standpoint, from the technical standpoint, employing LLMs for generative tasks is the perfect use case. In this article, we used a couple of techniques to guide the model into generating the output we strive for, without spending huge effort on training our own model.

As a conclusion, I'll leave the fragment generated with my approach. I think it is vague and enigmatic enough to be passed as a modern poem.

On black earth, Crimean wormwood blooms,
As tears flow from the icon of the Mother of God.
The vizier of Solomon appears on city shop windows,
Like a guardian spirit abandoned.

Describing musical domain with F#

Bohdan Stupak — Fri, 16 Aug 2024 13:16:18 +0000

One of my recent projects was to create software that would automatically generate music based on a predefined set of rules. The degree of randomness I've planned to introduce would let me create different melodies every time, while the set of rules I was planning to create would ensure that it still would sound nice. You can access the complete source code here. Below we'll dive more deeply into the details regarding what it does.

Domain description

The rules I was planning to expand upon are the concept of functional harmony. This concept is based on the idea that each chord has its own function. The functionality of a given chord is based on where the chord “wants” to go next because a harmonic progression has two dimensions: the chord’s pitches and how they are interacting (interval hierarchy); and its function in the overall harmonic context. So, functional harmony goes through the cycle of creating and releasing tension and as a result, we have stable and unstable moments that vary in different degrees of intensity.

The three most important functions are:

The Tonic; can either be or feel very stable, and generally is the final chord of a piece of music or a section
The Sub-Dominant; prepares the harmonic cadence and introduces some degree of instability
The Dominant; the most unstable chord that wants to resolve to another chord

Encoding the domain

Now let's encode this knowledge into code. I've chosen F# for this task since its type system is quite handy for expressing all sorts of domains.

Let's start from the basics and describe what chords we have in our palette.

type ChordQuality =
    | Major
    | Minor

There are many more chord qualities, but this is just enough for our needs.

Now, let's describe the knowledge we've obtained from the previous paragraph.

type HarmonyItem =
    | Tonic
    | SubDominant
    | Dominant

The transitions between them will look as below

type HarmonyTransition =
    | Dublicate
    | IncreaseTension
    | MaximizeTension
    | DecreaseTension
    | Resolve

Now let's see how the transitions are applied

let applyCommand command chord =
    match command with
    | Dublicate -> dublicate chord
    | IncreaseTension -> increaseTension chord
    | DecreaseTension -> decreaseTension chord
    | MaximizeTension -> maximizeTension chord
    | Resolve -> resolve chord

let dublicate harmonyItem =
    harmonyItem

let increaseTension harmonyItem =
    match harmonyItem with
    | Tonic -> SubDominant
    | SubDominant -> Dominant
    | Dominant -> Dominant

let decreaseTension harmonyItem =
    match harmonyItem with
    | Tonic -> Tonic
    | SubDominant -> Tonic
    | Dominant -> SubDominant

let maximizeTension harmonyItem =
    Dominant

let resolve harmonyItem =
    Tonic

With that said let's have a look at what hides behind each item in our functional progression. So basically each chord will have a quality and it's offset in notes from the root note.

type HarmonyItemValue = {
    value: int
    chordQuality: ChordQuality
}

let getHarmonyItemValue item =
    match item with
    | Tonic -> { value = 0; chordQuality = Major }
    | SubDominant -> { value = 5; chordQuality = Major }
    | Dominant -> { value = 7; chordQuality = Major }

Given this, we can create an array of pitches from each harmony item.

type Pitch = {
    midiNote: int
    duration: float
}

let createChordFromRootNote rootNote item =
    let itemValue = getHarmonyItemValue item
    match (itemValue.value, itemValue.chordQuality) with
    | (value, Major) -> [|
        {
            midiNote = rootNote + value
            duration = 1.0
        };
        {
            midiNote = rootNote + value + 4
            duration = 0.125
        };
        {
            midiNote = rootNote + value + 7
            duration = 1.0
        }|]
    | (value, Minor) -> [|
        {
            midiNote = rootNote + value
            duration = 1.0
        };
        {
            midiNote = rootNote + value + 4
            duration = 0.125
        };
        {
            midiNote = rootNote + value + 7
            duration = 1.0
        }|]

Generating the progression

So to create different progressions each time we need to add some randomness to the process. To achieve that we'll have some degree associated with the probability of each transition. Let's say we're in our tonic chord and we have the probability of 0.1 that we'll stay there for the next chord, while the probabilities of increasing tension are equal among themselves and are total to 0.45 each. In such a case let's assign a threshold for each transition. Say Tonic will be 0.1, SubDominant will be 0.55 which is the tonic threshold of 0.1 + tonic probability and Dominant will be 1.0 which is the probability of a full group of events. In such a case once we generate a random number between 0.0 and 1.0 we can select the minimal item that has a threshold greater than the given random number.

Here's how it looks in the code.

type HarmonyTransitionProbability = {
    transition: HarmonyTransition
    coinThreshold: float
}

let regenerateHarmonyTransitionProbability currentHarmonyItem =
    match currentHarmonyItem with
    | Tonic ->
        [|
            { transition = Dublicate; coinThreshold = 0.1 };
            { transition = IncreaseTension; coinThreshold = 0.55 };
            { transition = MaximizeTension; coinThreshold = 1.0 };
        |]
    | SubDominant ->
        [|
            { transition = Dublicate; coinThreshold = 0.1 };
            { transition = IncreaseTension; coinThreshold = 0.55 };
            { transition = Resolve; coinThreshold = 1.0 };
        |]
    | Dominant ->
        [|
            { transition = Dublicate; coinThreshold = 0.1 };
            { transition = Resolve; coinThreshold = 0.9 };
            { transition = DecreaseTension; coinThreshold = 1.0 };
        |]

let rnd = Random()

let generateNextChord currentChord coin =
    let probabilityMap = regenerateHarmonyTransitionProbability currentChord
    let command = (Array.filter (fun x -> coin <= x.coinThreshold) probabilityMap).[0].transition
    applyCommand command currentChord

let generateProgression (initialChord: HarmonyItem) (length: int) : HarmonyItem array =
    let rec generate (currentChord: HarmonyItem) (remaining: int) (progression: HarmonyItem list) =
        if remaining = 0 then
            List.toArray (List.rev progression)
        else
            let coin = rnd.NextDouble()
            Console.WriteLine(coin)
            let nextChord = generateNextChord currentChord coin
            generate nextChord (remaining - 1) (nextChord :: progression)
    generate initialChord (length - 1) [initialChord]

Domain evolution

So far we have covered only some basic concepts. But even some more mainstream progressions such as Axis of Awesome 4 chord wamp operate on the concept of substitutes. Substitutes are the duplicates of harmonic functions we already know but are not as distinct as their counterpart. So let's introduce them in our domain as well.

To me, this was the most enjoyable part of expressing my domain in F# since I had to remember to add it in two places: harmony items and transitions between them.

type HarmonyItem =
    | Tonic
    | TonicSubstitute1
    | TonicSubstitute2
    | SubDominant
    | Dominant

type HarmonyTransition =
    | Dublicate
    | IncreaseTension
    | MaximizeTension
    | DecreaseTension
    | DecreaseTensionToFisrtSubstitute
    | DecreaseTensionToSecondSubstitute
    | Resolve
    | ResolveToFirstSubstitute
    | ResolveToSecondSubstitute

At this point, in any place where I apply pattern matching, the compiler issues me a warning about incomplete pattern matching. So I just add missing cases until the compiler is satisfied and voila: a new version of the domain is complete. To some extent, this reminds me of lean on the compiler technique from the all-time classic "Working effectively with legacy code".

Generating sound

At this point, we can produce the array of pitches that are midi notes. To create sound from these notes I've used a specialized programming language called SuperCollider. I won't dive much into details here, but you may have a look at the code if you're interested. Beware, there are quite a lot of branches there and all of them contain some interesting code.

Conclusion

I've been a proponent of F# for quite a long time. So instead of expanding once again on the power of its type system, I'll just leave here a link to one of my favorite tracks that was created with the code in this article.

Composing poésie concrète with AWS Step Function

Bohdan Stupak — Tue, 27 Feb 2024 09:54:11 +0000

Concept

Observing signs of russo-Ukrainian war fatigue I decided to come up with a poem called "Is the war over?". I've labeled it as poésie concrète drawing an analogy with musique concrète - genre where music is composed of non-musical pieces of sound.

In the same way, my poem is composed of search engine results for "russian war crimes" in the latest week which are then processed by a sentiment analysis model to extract sentences that highlight russian atrocities most properly. After ML-processing sentences assembled in the poem. Each Sunday the poem is regenerated with new and new war crimes.

The war will be over on a day when the search engine will return no results and the poem will be blank.

One may argue that search engine might return results long after hostilities will end. This is exactly the point. Many people in Ukraine will have to live with the aftermath of war for their entire lives. Consider war veterans, traumatized children, and families who lost their close ones.

You may access the page that leads to the poem here.

High-level architecture

I've decided to proceed with the serverless offering since it allows me to pay per execution and execution figures are low for this one. While my experience mostly connected with .NET stack for this project I've decided to go with Javascript since its web-based capabilities exceed any other language I'm familiar with.

The high-level architecture diagram looks as below.

The entire process is launched by EventBridge Scheduler which launches the chain of Lambdas each following its own responsibility:

Crawling the search engine
Extract article content from a web page
Analyze the sentiment of each sentence inside the article
Assemble the poem from the sentences with the strongest sentiment and put it in S3 bucket that is served to the client via CloudFront.

Since the source code is stored in Github I decided to deploy them via Github actions. In the article below we'll focus on some points of interest found in the code.

Crawling the search engine

The algorithm behind Google Crawler service is:

Make a request to https://www.google.com/search?q=russian+war+crimes&tbs=qdr:w
Traverse html page for links to web pages.

Once approaching this task I was under the impression that I'd leverage document API to traverse the HMTL. However, it relies on a browser which was not the case for Lambda. JSDom came to my rescue.

With its help extracting necessary values is as simple as

const dom = new jsdom.JSDOM(data);
const anchors = dom.window.document.querySelectorAll('a[data-ved]');

First deploy

I decided to design the project for deployment from the start so the next step was to introduce continuous build on GitHub.

name: Google Crawler Build

on:
  push:
    branches: [ "*" ]
  pull_request:
    branches: [ "master" ]

jobs:
  build:

    runs-on: ubuntu-latest

    strategy:
      matrix:
        node-version: [20.x]
        # See supported Node.js release schedule at https://nodejs.org/en/about/releases/

    steps:
    - uses: actions/checkout@v3
    - name: Use Node.js ${{ matrix.node-version }}
      uses: actions/setup-node@v3
      with:
        node-version: ${{ matrix.node-version }}
        cache: 'npm'
        cache-dependency-path: "./src/google-crawler/package-lock.json"
    - run: cd ./src/google-crawler && npm ci
    - run: cd ./src/google-crawler && npm test
    - run: cd ./src/google-crawler && npm run lint

I think the code is pretty self-explanatory, however, let's look through some points.

Here we rely on ubuntu-latest environment and node-version: [20.x].

First of all, we check out the source code with actions/checkout@v3. For npm to work correctly we have to specify the path to package-lock.json file with cache-dependency-path: "./src/google-crawler/package-lock.json". Apart from restoring packages with npm ci we also run unit-tests and linter which are necessary quality gates for our codebase.

The deployment looks as follows

name: Google Crawler Deploy

on:
  push:
    branches: [ "master" ]
jobs:
  lambda:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        node-version: [20.x]
        # See supported Node.js release schedule at https://nodejs.org/en/about/releases/

    steps:
      - uses: actions/checkout@v3
      - name: Use Node.js ${{ matrix.node-version }}
        uses: actions/setup-node@v3
        with:
          node-version: ${{ matrix.node-version }}
          cache: 'npm'
          cache-dependency-path: "./src/google-crawler/package-lock.json"
      - uses: aws-actions/configure-aws-credentials@v2
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: eu-central-1
      - run: cd ./src/google-crawler && npm ci
      - run: cd ./src/google-crawler && zip -r lambda1.zip ./
      - run: cd ./src/google-crawler && aws lambda update-function-code --function-name=google-crawler --zip-file=fileb://lambda1.zip

It looks pretty similar to the build job, however, we also zip the code and deploy it via aws lambda update-function-code command.

Extracting article content

To extract article content from the web page I've used Readability package. Here's how I download article content from the web page and split it into sentences.

const res = await fetch(url);
const html = await res.text();
const doc = new jsdom.JSDOM(html);
const reader = new readability.Readability(doc.window.document);
const article = reader.parse();
const sentences = splitIntoSentences(article.textContent);

Calling one lambda from another

There are many advices on how to synchronously call one lambda from another over the internet. AWS documentation, however, is more prohibitive on that matter and for a good reason:

While this synchronous flow may work within a single application on a server, it introduces several avoidable problems in a distributed serverless architecture:

Cost: with Lambda, you pay for the duration of an invocation. In this example, while the Create invoice functions runs, two other functions are also running in a wait state, shown in red on the diagram.

Error handling: in nested invocations, error handling can become much more complex. Either errors are thrown to parent functions to handle at the top-level function, or functions require custom handling. For example, an error in Create invoice might require the Process payment function to reverse the charge, or it may instead retry the Create invoice process.

Tight coupling: processing a payment typically takes longer than creating an invoice. In this model, the availability of the entire workflow is limited by the slowest function.

Scaling: the concurrency of all three functions must be equal. In a busy system, this uses more concurrency than would otherwise be needed.

One of the alternatives is to use AWS Step Functions to orchestrate the execution of the lambdas. And turns out that my problem is a nice example of distributed map. Consider:

We extract all necessary links
We map each link in parallel by extracting its content and analyzing its sentiment.
We reduce it into a single S3 bucket.

Here's the definition of the entire system.

{
  "Comment": "A Step Functions workflow that processes an array of strings concurrently",
  "StartAt": "Extract links from google",
  "States": {
    "Extract links from google": {
      "Type": "Task",
      "Resource": "<google crawler arn>",
      "ResultPath": "$",
      "Next": "ProcessArray",
      "Retry": [
        {
          "ErrorEquals": [
            "States.ALL"
          ],
          "IntervalSeconds": 1,
          "MaxAttempts": 2,
          "BackoffRate": 2
        }
      ]
    },
    "ProcessArray": {
      "Type": "Map",
      "ItemsPath": "$",
      "MaxConcurrency": 10,
      "Iterator": {
        "StartAt": "Extract article content",
        "States": {
          "Extract article content": {
            "Type": "Task",
            "Resource": "<article extractor arn>",
            "InputPath": "$",
            "Next": "Analyze sentiment",
            "Retry": [
              {
                "ErrorEquals": [
                  "States.ALL"
                ],
                "IntervalSeconds": 1,
                "MaxAttempts": 2,
                "BackoffRate": 2
              }
            ],
            "Catch": [
              {
                "ErrorEquals": [
                  "States.ALL"
                ],
                "Next": "Analyze sentiment"
              }
            ]
          },
          "Analyze sentiment": {
            "Type": "Task",
            "Resource": "<sentiment analyzer arn>",
            "InputPath": "$",
            "End": true,
            "Retry": [
              {
                "ErrorEquals": [
                  "States.ALL"
                ],
                "IntervalSeconds": 1,
                "MaxAttempts": 2,
                "BackoffRate": 2
              }
            ]
          }
        }
      },
      "Next": "Reducer",
      "Retry": [
        {
          "ErrorEquals": [
            "States.ALL"
          ],
          "IntervalSeconds": 1,
          "MaxAttempts": 2,
          "BackoffRate": 2
        }
      ],
      "Catch": [
        {
          "ErrorEquals": [
            "States.ALL"
          ],
          "Next": "Reducer"
        }
      ]
    },
    "Reducer": {
      "Type": "Task",
      "Resource": "<reducer arn>",
      "InputPath": "$",
      "ResultPath": "$",
      "End": true,
      "Retry": [
        {
          "ErrorEquals": [
            "States.ALL"
          ],
          "IntervalSeconds": 1,
          "MaxAttempts": 2,
          "BackoffRate": 2
        }
      ]
    }
  }
}

Here we enter into the map phase executing step with "Type": "Map", and both are Article extractor and Sentiment analyzer serve as Iterator. Once the map phase is done we enter reduce phase via "Next": "Reducer".

Another thing worth mentioning is increasing the reliability of our system by adding error handling. The most obvious way is adding retries via

"Retry": [
  {
    "ErrorEquals": [
      "States.ALL"
    ],
    "IntervalSeconds": 1,
    "MaxAttempts": 2,
    "BackoffRate": 2
  }
]

We also use another tactic - letting a single map instance fail instead of failing the entire map phase.

"Catch": [
  {
    "ErrorEquals": [
      "States.ALL"
    ],
    "Next": "Analyze sentiment"
  }
]

Using layers to optimize monorepo structure

At this point, the structure of our repository looks suboptimal with package.json and separate build step for each function. What's more: a separate package.json means a separate node_modules folder which leads to much disk space going to waste since a lot of modules are duplicates.

This won't scale once we will add more functions. There is however a way to build and package all the dependencies at once using Lambda layers. This approach lets us package all the dependencies into a separate layer and treat it for our functions as a common runtime.

We'll reorganize our repository to look like this:

Let's have a look at a separate action that deploys the layer:

name: Deploy Modules Layer

on:
  workflow_call:
    secrets:
      AWS_ACCESS_KEY_ID:
        required: true
      AWS_SECRET_ACCESS_KEY:
        required: true

jobs:
  layer:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        node-version: [20.x]
        # See supported Node.js release schedule at https://nodejs.org/en/about/releases/

    steps:
      - uses: actions/checkout@v3
      - name: Use Node.js ${{ matrix.node-version }}
        uses: actions/setup-node@v3
        with:
          node-version: ${{ matrix.node-version }}
          cache: 'npm'
          cache-dependency-path: "./src/package-lock.json"
      - uses: aws-actions/configure-aws-credentials@v2
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: eu-central-1
      - run: cd ./src && npm ci
      - run: cd ./src && zip -r layer.zip node_modules
      - run: cd ./src && aws lambda publish-layer-version --layer-name poeme-concrete-modules --zip-file fileb://layer.zip

Nothing special is happening here apart from the fact that now we are using aws lambda publish-layer-version. Now let's jump to consuming the deployed layer when we deploy our functions.

name: Article Extractor Deploy

on:
  push:
    branches: [ "master" ]
jobs:
  layer:
    uses: ./.github/workflows/modules-layer-deploy.yml
    secrets: inherit

  lambda:
    runs-on: ubuntu-latest
    needs: layer
    strategy:
      matrix:
        node-version: [20.x]
        # See supported Node.js release schedule at https://nodejs.org/en/about/releases/

    steps:
      - uses: actions/checkout@v3
      - name: Use Node.js ${{ matrix.node-version }}
        uses: actions/setup-node@v3
        with:
          node-version: ${{ matrix.node-version }}
          cache: 'npm'
          cache-dependency-path: "./src/package-lock.json"
      - uses: aws-actions/configure-aws-credentials@v2
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: eu-central-1
      - run: cd ./src && npm ci
      - run: cd ./src/article-extractor && zip -r lambda1.zip ./
      - run: cd ./src/article-extractor && aws lambda update-function-code --function-name=article-extractor --zip-file=fileb://lambda1.zip
      - run: echo "layer-arn=$(aws lambda list-layer-versions --layer-name poeme-concrete-modules --region eu-central-1 --query 'LayerVersions[0].LayerVersionArn')" >> $GITHUB_ENV
      - run: aws lambda update-function-configuration --function-name=article-extractor --layers="${{ env.layer-arn }}"

Here couple of things are worth noting.

First of all, is how we rely on deploy layers job.

jobs:
  layer:
    uses: ./.github/workflows/modules-layer-deploy.yml
    secrets: inherit

The thing that might be unobvious to newcomers is how we use secrets: inherit to pass secrets down to the layer deploy action. One might naturally assume that it will infer secrets from the Github storage, however, this is not true and child action infers secrets from parent workflow.

Another important thing is forcing newly deployed function to use the latest version of the published layer. We achieve this in two steps:

Querying for the latest layer version and storing it inside the environment variable

echo "layer-arn=$(aws lambda list-layer-versions --layer-name poeme-concrete-modules --region eu-central-1 --query 'LayerVersions[0].LayerVersionArn')" >> $GITHUB_ENV

Using stored value to configure update function configuration

aws lambda update-function-configuration --function-name=article-extractor --layers="${{ env.layer-arn }}"

Accessing secrets

When it comes to choosing a sentiment analysis engine the natural choice is Amazon Comprehend. Why I didn't stick with it? I didn't like the results.

Instead, I've chosen text2data service. At the end of the day it's like calling any other third-party service via HTTP so in this section I'll briefly cover retrieving secrets needed to call this API.

import { SecretsManagerClient, GetSecretValueCommand } from "@aws-sdk/client-secrets-manager";

async function getSentimentAnalysisApiKey() {
    const secret_name = "SENTIMENT_ANALYSIS_API_KEY";

    const client = new SecretsManagerClient({
        region: "eu-central-1",
    });

    let response;

    try {
        response = await client.send(
            new GetSecretValueCommand({
                SecretId: secret_name,
                VersionStage: "AWSCURRENT"
            })
        );
    } catch (error) {
        console.log(error);
        throw error;
    }

    return response.SecretString;
}

Writing down the result to S3

Cloudfront serves HTML content from S3 bucket. So in order for the poem to be published we need to generate HTML and store it inside the bucket.

To generate the HTML we insert sentences inside mustache template

const formatted =
        poem
          .map(p => `<p>${p}</p>`)
          .join("\n");
const html = renderTemplate(formatted);

const renderTemplate = (poem) => {
  const template = fs.readFileSync('./template.html', 'utf8');

  return Mustache.render(template, {
    poem: poem
  });
}

The point of interest in the template is that we have to use triple curly brackets in order for the inserted HTML not to be escaped

<html>
    //omitted for brevity
    <body>
        <article>
{{{poem}}}
        </article>
    </body>
</html>

Now we can store the HTML in S3 with the code below:

const putParams = {
  Bucket: 'poeme-concrete',
  Key: 'index.html',
  Body: html,
  ContentType: 'text/html',
};

await s3.putObject(putParams).promise();

Conclusion

Usually, at this point I write a summary of technologies touched on in the article. This time, however, I encourage you to read through the poem and occasionally revisit it. You may brush it off as something too disturbing but for many people in Ukraine it's a grim reality. I am no exception since I have spent the last two years battling to cure my now-4-year-old son's PTSD and speech disorder. Still, I'm in a more lucky position because I have a roof over my head and live in a relatively peaceful region of the country.

And if the course of the last 30 years has taught us anything it's that you can never stop the aggressor by leaving him unpunished. russia didn't stop after Transnistria, Ichkeria, Abkhazia, Crimea, and parts of Donetsk and Luhansk regions, and won't stop now if it feels that it can wage this war unpunished.

This is why I encourage you to take a stance and support the Ukrainian military via these funds I've been supporting personally as well.

Dynamic polymorphism: key concept to master OOP

Bohdan Stupak — Thu, 06 Jul 2023 18:40:40 +0000

The world of object-oriented programming is a bit confusing. It requires mastering a lot of things: SOLID principles, design patterns to name a few. This gives birth to a lot of discussions: are design patterns still relevant, is SOLID intended solely for object-oriented code? It is said that one should prefer composition to inheritance but what is the exact rule of thumb when one should choose one or another?

Since numerous opinions on this matter were voiced I don't think that mine will be the final but nevertheless, in this article, I'll present the system that helped me in my everyday programming using C#. But before we jump to that let's have a look at another question. Consider the code.

public class A
{
    public virtual void Foo()
    {
        Console.WriteLine("A");
    }
}

public class B : A
{
    public override void Foo()
    {
        Console.WriteLine("B");
    }
}

public class C : A
{
    public void Foo()
    {
        Console.WriteLine("C");
    }
}

var b = new B();
var c = new C();
b.Foo();
c.Foo();

Can you tell what the code will output in each case? If you've answered correctly that respective output will be "B" and "C" then why does override keyword matter?

Enter dynamic polymorphism

Polymorphism is believed to be one of the pillars of object-oriented programming. But what exactly it means? Wikipedia tells us that polymorphism is the provision of a single interface to entities of different types or the use of a single symbol to represent multiple different types.

I don't expect you to grasp this definition from the first time so let's have a look at some examples.

string Add(string input1, string input2) => string.Concat(input1, input2);
int Add(int input1, int input2) => input1 + input2;

Above is an example of ad-hoc polymorphism which refers to polymorphic functions that can be applied to arguments of different types, but that behave differently depending on the type of the argument to which they are applied. So why is polymorphism so important for object-oriented code? This snippet does not provide a clear answer to this question. Let's look through more examples.

class List<T> {
    class Node<T> {
        T elem;
        Node<T> next;
    }
    Node<T> head;
    int length() { ... }
}

This is an example of parametric polymorphism and to be honest it looks more functional than object-oriented. Let's have a look at the final example.

interface IDiscountCalculator
{
    decimal CalculateDiscount(Item item);
}

class ThanksgivingDayDiscountCalculator : IDiscountCalculator
{
    public decimal CalculateDiscount(Item discount)
    {
        //omitted
    }
}

class RegularCustomerDiscountCalculator : IDiscountCalculator
{
    public decimal CalculateDiscount(Item discount)
    {
        //omitted
    }
}

This is an example of dynamic polymorphism which is the term for applying polymorphism at the runtime (mostly via subtyping). And if you've tried to memorize all these design patterns before the interview or some SOLID principles you may notice the shape of something familiar. Let's see how dynamic polymorphism manifests itself in these concepts.

Dynamic polymorphism and design patterns

Most design patterns (strategy, command, decorator, etc) rely on injecting the abstract class or the interface and choosing the implementation of it at runtime. Let's have a look at some class diagrams to make sure it's the case.

Above is the diagram of strategy pattern where Client works with abstraction and its concrete implementation is chosen during the runtime.

And here's decorator.

In this case, wrapper accepts wrappee which is an instance of abstraction and its implementation may vary during the runtime.

Dynamic polymorphism and SOLID

When asking about SOLID during the interview the regular answer I hear is "S stands for single responsibility and O stands for uhm...". On the contrary, I argue that the latter four letters of this acronym are more important since they represent the set of preconditions for your dynamic polymorphism to run smoothly.

For instance, open-closed principle represents a way of thinking in which you tackle every new problem as the subtype for your abstraction. Recall IDiscountCalculator example. Imagine the case when you have to add another discount (say for father's day). To satisfy open-closed principle you have to add another subclass FathersDayDiscountCalculator that performs the calculation.

Let's move on to the Liskov substitution principle. Imagine the situation that breaks it: we have to check whether the user is actually a father and it's a matching date. So we add the public method which checks whether the user is eligible.

class FathersDayDiscountCalculator : IDiscountCalculator
{
    public decimal CalculateDiscount(Item discount)
    {
        //omitted
    }

    public bool IsEligible(User user, DateTime date)
    {
        //omitted
    }
}

Now the calling code will face some complications

private IReadOnlyCollection<IDiscountCalculator> _discountCalculators;
public decimal CalculateDiscountForItem(Item item, User user)
{
    decimal result = 0;
    foreach (var  discountCalculator in _discountCalculators)
    {
        if (discountCalculator is FathersDayDiscountCalculator)
        {
            var fathersDayDiscountCalculator = discountCalculator as FathersDayDiscountCalculator;
            if (fathersDayDiscountCalculator.IsEligible(user, DateTime.UtcNow))
            {
                result += fathersDayDiscountCalculator.CalculateDiscount(item);
            }
        }
        else
        {
            result += discountCalculator.CalculateDiscount(item);
        }
    }
    return result;
}

Pretty verbose, isn't it? So in order to satisfy Liskov substitution principle we have to force all our implementations to exhibit the same public contract provided by the abstraction. Otherwise, it will complicate the application of dynamic polymorphism.

Another thing that complicates the application of dynamic polymorphism is having abstraction too broad. Imagine that we've made IsEligible the part of our interface and now all concrete classes implement it. The calling code is greatly simplified.

private IReadOnlyCollection<IDiscountCalculator> _discountCalculators;
public decimal CalculateDiscountForItem(Item item, User user)
{
    decimal result = 0;
    foreach (var  discountCalculator in _discountCalculators)
    {
        result += discountCalculator.CalculateDiscount(item);
    }
    return result;
}

But now imagine (I know the example is a bit contrived but just for the sake of the argument!) that one of the implementations threw NotImplementedException because it doesn't make sense for this particular type of discount. At this point, you may forsee the problem CalculateDiscountForItem failing with the runtime exception.

That is what interface-segregation principle is about: don't make abstractions too broad so that concrete types won't have trouble with implementing them thus complicating your dynamic polymorphism with unnecessary NotImplementedExceptions.

And by this time you may observe dependency inversion principle in action. In the example above we deal with the collection of abstractions and have no idea of their runtime types.

Prefer composition over inheritance

I won't dive a lot into why composition is preferable. There are numerous examples of how inheritance complicates things. But now when you have a question about what are legit cases for inheritance here's an answer for you: when it facilitates dynamic polymorphism.

Virtual and override

At this point, those of you who didn't know the answer correctly to the question at the beginning of the article might have a suspicion that it was a tricky question. And indeed while the behavior is similar when we use var keyword, differences start to arise when we apply dynamic polymorphism. For that matter let's convert both instances to parent type.

A b = new B();
A c = new C();

Now the output will be "B" and "A" respectively. Hot memorize this? The goal of override keyword is to facilitate dynamic polymorphism. So think of it this way: when we inject abstraction we expect to work with the concrete type realization and as override facilitates this goal so implementation of B will be invoked.

Why this matters?

So now you know how to memorize all these pesky interview questions. But the most curious of you may be asking: what are the benefits of such a style of programming? Why do we strive to apply dynamic polymorphism in our object-oriented codebases?

Imagine we have two methods somewhere in our codebase.

public string GetCurrencySign(string currencyCode)
{
    return currencyCode switch
    {
        "US" => "$",
        "JP" => "¥",
        _ => throw new ArgumentOutOfRangeException(nameof(currencyCode)),
    };
}

public decimal GetRoundUpAmount(decimal amount, string currencyCode)
{
    return currencyCode switch
    {
        "US" => Math.Floor(amount + 1),
        "JP" => Math.Floor(amount / 100 + 1) * 100,
        _ => throw new ArgumentOutOfRangeException(nameof(currencyCode))
    };
}

Now imagine we have to add another country support. Doesn't look like a big deal but imagine these two methods hiding in one of those "real-world" codebases with thousands of classes and hundreds of thousands of lines of code. Most likely you'll forget all the places where you should add country support. This is exactly what Shotgun surgery code smell is.

How do we fix it? Let's extract all the information related to country code in a single place.

public interface IPaymentStrategy
{
    string CurrencySign { get; }
    decimal GetRoundUpAmount(decimal amount);
}

Now when we have to add a new country code we are forced to implement the interface above so we definitely won't forget anything. We use factory to return instances of IPaymentStrategy.

public string GetCurrencySign(string currencyCode)
{
    var strategy = _strategyFactory.CreateStrategy(currencyCode);
    return strategy.CurrencySign;
}

In the example above we've fixed a code smell by applying dynamic polymorphism. Occasionally we've managed to satisfy some SOLID principles (namely Open-Closed by crafting new functionality with extensions instead of modification) and applying design patterns. Bunch of cool enterprise stuff for your CV by applying just a single OOD principle!

Conclusion

Software engineers, just like most of us, tend to follow a lot of principles without questioning their rationale. When this is done principles tend to get distorted and diverted from their original goal. So by questioning what was the original goal we might apply these principles as they were intended to be applied.

In this article, I've argued that one of the core principles beyond OOD was the application of dynamic polymorphism and a lot of principles (SOLID, design patterns) are just mnemonics built around it.

Visualizing your photos on a map with React Native

Bohdan Stupak — Wed, 01 Mar 2023 18:45:32 +0000

that time, I had a Nokia Lumia phone with a nice widget that displayed the locations of the photos I've taken on a map. I enjoyed looking at it realizing how big is the world and how much I have to travel in order to discover it.

While I have a similar widget in my Android phone I wasn't satisfied with its look and feel. Another issue for me was that the last time I touched fron-end was 4 years ago and since then I hadn't much experience with modern SPA-frameworks. The desire to bridge that gap and to have the app I wanted motivated me to create my own.

The ultimate result looks like this.

In case you want to try out the app you can download it from the Play Market. The source code is available on Github in case you want to follow along with the code in the article.

Constraints and tradeoffs

The reason behind choosing my tech stack was to get to know modern SPA frameworks better. So naturally, my choice was React Native. As a novice in mobile development, I decided to start with Expo. It is a nice way to start the project since it contains a couple of templates as well as a nice debugging toolset.

What I've learned is that the convenience of Expo is not free. Here are some disadvantages:

Implicit integration with Facebook SDK
Depreciation of some crucial libs such as expo-ads-admob
Deprecation of classic build.

However, these shortcomings can be remediated with EAS build.

Visualization algorithm

In order to provide the best possible look and feel we need to give a user a feeling that something is happening while the application is enumerating the photos and obtaining their geodata. So rendering sequence looks as below:

Display a splash screen
Replace it with a screen that resembles a splash screen but additionally provides information about loading progress.
Once photos are processed, display a map with the points.

Let's briefly look at these points.

Displaying progress information

The root component code is provided below

SplashScreen.preventAutoHideAsync().catch(() => {
  /* reloading the app might trigger some race conditions, ignore them */
});

const App = () => {
  return (
    <AnimatedAppLoader/>
  );
}

Pretty straightforward. We prevent the automatic hiding of a splashscreen and render a single component. Let's dive deeper into the code of this component.

  return (
      <View style={{ flex: 1 }}>
        <AnimatedSplashScreen/>
      </View>
    )
  }

The point of our interest is AnimatedSplashScreen component.

const AnimatedSplashScreen = () => {
    const textAnimation = useMemo(() => new Animated.Value(0), []);
    const [isAppReady, setAppReady] = useState(false);
    const [isTextAnimationIsReady, setTextAnimationIsReady] = useState(false);
    const [loadingText, setLoadingText] = useState("");

    useEffect(() => {
      if (!isAppReady && isTextAnimationIsReady) {
        Animated.timing(textAnimation, {
          toValue: 1,
          duration: 200,
          easing: Easing.inOut(Easing.exp),
          useNativeDriver: true,
        }).start();
      }
    }, [isAppReady, isTextAnimationIsReady]);

    const loadLocations = async () => {
      let markersArray : MediaLibrary.Location[] = [];
      let hasMoreData = true;
      try {
        while (hasMoreData) {
          //loading imahe locations is omitted by now
          let now = Date.now();
          let delta = now - timeStart;
          if (delta > 9000) {
            setLoadingText(`We've processed ${cursor.endCursor} items. There's more work though...`)
          } else if (delta > 5000) {
            setLoadingText("Working on it...")
          } else if (delta > 2000) {
            if (!isTextAnimationIsReady) {
              setTextAnimationIsReady(true);
            }
            setLoadingText("Hold on! We're doing some magic just for you...")
          }
        }
      } catch (e) {
        console.log(e)
      } finally {
        setAppReady(!hasMoreData);
      }
    }

    const onImageLoaded = useCallback(async () => {
      await SplashScreen.hideAsync();
      await loadLocations();
    }, []);

    return (
      <View style={{ flex: 1 }}>
        {isAppReady  && (<MainScreen/>)}
        {!isAppReady && (<Animated.View
          pointerEvents="none"
          style={[
            {
              backgroundColor: Constants.manifest?.splash?.backgroundColor
            },
          ]}>
          <Animated.Text>
            {loadingText}
          </Animated.Text>
          <Animated.Image
            source={require("../assets/splash.png")}
            onLoadEnd={onImageLoaded}
            fadeDuration={0}
          />
        </Animated.View>)}
      </View>
    );
  }

What happens here is that we replace our splash screen with Animated.Image that looks exactly like the splash screen. Once the image is ready we call onImageLoaded callback. Now when our image is ready we can hide the splashscreen and call loadLocations. This method performs the work needed to enumerate location of our photos and meanwhile takes charge of animation to keep the user engaged.

For now, everything we need to know about loadLocations method is that it does image processing batch by batch which is indicated by while (hasMoreData). This means that inside each loop iteration, we can check how much time elapsed and trigger text animation accordingly. Once it's time to display text animation we update the state with setTextAnimationIsReady(true);. This in turn triggers Animated.timing. Which changes the opacity of Animated.Text.

Once we're done we update the state with setAppReady and substitute our animation component with MainScreen.

Loading image location

In order to enumerate the contents of our gallery we'll use Expo Medialibrary.

let medialibraryRequest : MediaLibrary.AssetsOptions = {}

const loadLocations = async () => {
  let markersArray : MediaLibrary.Location[] = [];
  let hasMoreData = true;
  try {
    let { status } = await MediaLibrary.requestPermissionsAsync();
    let markersSet : Set<MediaLibrary.Location> = new Set();
    const albums = await MediaLibrary.getAlbumsAsync();
    const cameraAlbum = albums.find(p => p.title === "Camera");
    medialibraryRequest.album = cameraAlbum;
    while (hasMoreData) {
      let cursor = await MediaLibrary.getAssetsAsync(medialibraryRequest);
      await populateLocationsIntoSet(cursor, markersSet);
      hasMoreData = cursor.hasNextPage;
      medialibraryRequest.after = cursor.endCursor
    }
    markersArray = [...markersSet]
    setMarkers(markersArray);
  } catch (e) {
    console.log(e)
  } finally {
    setAppReady(!hasMoreData);
  }
}

const populateLocationsIntoSet = async (
  cursor : MediaLibrary.PagedInfo<MediaLibrary.Asset>,
  markersSet : Set<MediaLibrary.Location>) => {
    const allowedTypes : MediaLibrary.MediaTypeValue[] = [
      MediaLibrary.MediaType.photo,
      MediaLibrary.MediaType.video
    ]
    const markersArray = await Promise.all(cursor.assets.map(async element => {
      if (!allowedTypes.includes(element.mediaType)) {
        return;
      }
      let image = await MediaLibrary.getAssetInfoAsync(element);
      return image.location;
    }));
    if (markersArray.length === 0) {
      return;
    }
    let nonNullLocations = markersArray.filter(p => p != undefined) as MediaLibrary.Location[];
    nonNullLocations.forEach(markersSet.add, markersSet);
}

We start off with requesting permissions to access the media library using requestPermissionsAsync. Then we iterate "Camera" album contents with the method getAssetsAsync that acts as a cursor obtaining items batch by batch. Since the resulting items contain only basic information about the item we need to perform additional request to obtain geolocation. We do this inside populateLocationsIntoSet using getAssetInfoAsync method.

Displaying locations on a map

To display locations on a map I use react-native-maps.

const MainScreen = ({markers}: MainScreenProps) => {
    let map = useRef<MapView>(null);

    const fitAllMarkers = () => {
      const boundingBox = markers
      if (markers.length === 1) {
        markers.push({latitude: markers[0].latitude+0.1, longitude: markers[0].longitude+0.1})
      }
      map.current?.fitToCoordinates(boundingBox, {
        edgePadding: DEFAULT_PADDING,
        animated: true,
    })};

    return (
       <View style={styles.container}>
        <MapView ref={map}
          style={styles.map}
          onMapLoaded={fitAllMarkers}>
          {markers.map((item : MediaLibrary.Location) => (
            <Marker
              key={Math.random()}
              coordinate={{
                latitude: item.latitude,
                longitude: item.longitude,
              }}
              icon={require('../assets/marker.png')}>
            </Marker>
          ))}
        </MapView>
    );
  }

In the code above we use MapView component and for each item inside markers array we place a Marker on the map. Once the map is loaded we focus it on the area where the markers are set with fitToCoordinates method.

Sharing your map

I guess you can never imagine the modern world without people sharing stuff on social networks. So the crucial piece of functionality I wanted to embed in my application is "Share" button.

To share the results we'll require the combination of react-native-view-shot that captures the state of our screen once the map is rendered and expo-sharing to share the screenshot we've captured.

const captureAndShareScreenshot = async () => {
  const uri = await captureRef(map, {
      format: "png",
      quality: 1
  })
  await Sharing.shareAsync("file://" + uri);
};

Checking for connection status

The downside of react native maps is that it requires internet connection to load the tiles. That's why we would like to gracefully notify user if the internet connection is present instead of displaing broken maps.

In order to probe for internet connection we'll require NetInfo.

Let's revisit AnimatedAppLoader component.

const AnimatedAppLoader = () => {
  const [isConnected, setConnected] = useState(false)
  const [isConnectionProbeFinished, setConnectionProbeFinished] = useState(false)

  const fetchConnection = async () => {
    const connectionStaus = await NetInfo.fetch();
    setConnected(connectionStaus.isConnected === true)
    if (!connectionStaus.isConnected) {
      await SplashScreen.hideAsync();
    }
    setConnectionProbeFinished(true)
  }

  useEffect(() => {
     fetchConnection()
  }, [])

  const refresh = useCallback(async () => {
    await fetchConnection()
  },[])

  return (
      <View style={{ flex: 1 }}>
        {isConnected && isConnectionProbeFinished && <AnimatedSplashScreen/>}
        {!isConnected && isConnectionProbeFinished && <OfflineScreen refresh={refresh}/>}
      </View>
    )
  }

The point I'd like to highlight is the passing refresh inside the OfflineScreen component. Doing so allows us to perform another connection probe from inside the child component.

const OfflineScreen = ({refresh}: any) => {
    return (
        <View style={styles.container}>
            <Text style={styles.text}>
                Locate! needs an internet connection to function properly. {'\n'}
                Refresh this page once you enable it.
            </Text>
            <Button
                icon="refresh"
                mode="contained"
                buttonColor="#E81E25"
                textColor="#FFF"
                theme={CustomTheme}
                style={styles.button}
                onPress={() => refresh()}>
                    Refresh
            </Button>
        </View>
    )
}

Conclusion

Summarizing my journey I would say that React Native and Expo, in particular, are pretty convenient tools that allow you to develop mobile applications without the need to master Android/IOS specific technologies. With the use of the widely popular React framework, I've managed to develop the application I've wanted for myself for a pretty long time.

As per my personal experience publishing the application is a pretty tedious process and I'm not sure that I would like to try it again but that is the part I'll omit in this article.

Applying custom similarity calculation in Elasticsearch

Bohdan Stupak — Mon, 19 Dec 2022 16:20:48 +0000

Presently we've been discussing improving elasticsearch autocomplete functionality. For most cases, this is enough. However, in our case, things turned out to be not that rosy and we'll have a look at why.

Setup

In our system we single document type per index. Since we have quite a lot of types in our system and the number of types and their mappings can be changed by the end-user, we have unpredictable number of indices. Currently, this figure stands at 17 different indices. Naturally, documents are not distributed equally across all types meaning that some indices contain more records than others.

The key feature of your project is the ability to perform a search across logical a group of indices. This and the data skew across indices leads to some relevance implications that will become obvious to you when we discuss our problem in detail.

Another thing worth mentioning is that the user search queries are rather specific. The aim of our project is to dissect a large number of documents to get one or two relevant documents. So we don't expect vague queries like "Ukraine" that might yield thousands of documents. This will be important when we'll move on to the solution we propose.

The standard way of measuring similarity

By default, Elasticsearch uses BM25 similarity in order to rank search results. There are three main factors that determine a document’s score:

Term frequency (TF) — The more times that a search term appears in the field we are searching in a document, the more relevant that document is.
Inverse document frequency (IDF) — The more documents that contain a search term in the field that we are searching, the less important that term is.
Field length — If a document contains a search term in a field that is very short (i.e. has few words), it is more likely relevant than a document that contains a search term in a field that is very long (i.e. has many words).

Let's look more deeply at IDF. Inverse Document Frequency is the calculation function that takes Rare Globally words into account. The rarer the word is in the whole corpus, the more important it is.

Let's have a look at the extract from explain section to get a better understanding of how it's calculated.

{
    "value": 0.5389965,
    "description": "idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:",
    "details": [
    {
        "value": 3,
        "description": "n, number of documents containing term",
        "details": []
    },
    {
        "value": 5,
        "description": "N, total number of documents with field",
        "details": []
    }
    ]
},

This means that IDF is inverse to the number of documents in the index. In our case, it leads to danger. Recall that we perform a search on multiple indices. Imagine two different indices one with 2K documents and the other one with 20K documents both containing exactly one match for a given query. This means that IDF for the first index will be higher and thus the result from the first index will be more relevant!

For the end user, however, the number of indices and the number of items in them is an irrelevant implementation detail. What the user cares about is how well the output result matches the query string input.

Discarding IDF

Taking into account that the user has no interest in data skew across indices and places specific enough queries we've decided that IDF is irrelevant in our case. How can we discard it from our search calculation formula?

The answer is scripted similarity.

Let's reimplement BM25 but with constant IDF.

PUT /index
"settings": {
    "index": {
        "similarity": {
            "discarded_idf": {
                "type": "scripted",
                    "script": {
                        "source": "double tf = Math.sqrt(doc.freq); double idf = 1.0; double norm = 1 / Math.sqrt(doc.length); return query.boost * tf * idf * norm;"
                }
            }
        }
    }
}

And that's pretty it: original BM25 formula but with IDF that is constant for each search across all indices. With this done we can calculate similarity that suits our use-case better.

Conclusion

Elasticsearch has a wide range of tools for your search query to return the most relevant results possible. Once you exhaust dedicated datatypes and rich query DSL you might tweak similarity calculation as well by using either a set of predefined algorithms or rolling out your own.

Implementing clean architecture in Go

Bohdan Stupak — Wed, 02 Nov 2022 18:49:15 +0000

It has been written a lot about the clean architecture. Its main value is the ability to maintain free from side effects domain layer that allows us to test core business logic without leveraging heavy mocks.

This is accomplished by writing dependency-free core domain logic and external adapters (be it database storage or API layer) that rely on the domain and not vice versa.

In this article, we'll have a look at how clean architecture is implemented with a sample Go project. We'll cover some additional topics such as containerization and implementing OpenAPI specification with Swagger.

While I'll highlight the points of interest in the article you may take a look at the entire project at my Github

Project requirements

We are required to provide an implementation of a REST API to simulate a deck of cards.

We will need to provide the following methods to your API ho handle cards and decks:

Create a new Deck
Open a Deck
Draw a Card

Create a new Deck

It would create the standard 52-card deck of French playing cards, It includes all thirteen ranks in each of the four suits: clubs (♣), diamonds (♦), hearts (♥) and spades (♠). You don't need to worry about Joker cards for this assignment.

the deck to be shuffled or not — by default the deck is sequential: A-spades, 2-spades, 3-spades... followed by diamonds, clubs, then hearts.
the deck to be full or partial — by default it returns the standard 52 cards, otherwise the request would accept the wanted cards like this example ?cards=AS,KD,AC,2C,KH

The response needs to return a JSON that would include:

the deck id (UUID)
the deck properties like shuffled (boolean) and total cards remaining in this deck (integer)

{
    "deck_id": "a251071b-662f-44b6-ba11-e24863039c59",
    "shuffled": false,
    "remaining": 30
}

Open a Deck

It would return a given deck by its UUID. If the deck was not passed over or is invalid it should return an error. This method will "open the deck", meaning that it will list all cards by the order it was created.

The response needs to return a JSON that would include:

the deck id (UUID)
the deck properties like shuffled (boolean) and total cards remaining in this deck (integer)
all the remaining cards cards (card object)

{
    "deck_id": "a251071b-662f-44b6-ba11-e24863039c59",
    "shuffled": false,
    "remaining": 3,
    "cards": [
        {
            "value": "ACE",
            "suit": "SPADES",
            "code": "AS"
        },
                {
            "value": "KING",
            "suit": "HEARTS",
            "code": "KH"
        },
        {
            "value": "8",
            "suit": "CLUBS",
            "code": "8C"
        }
    ]
}

Draw a Card

We would draw a card(s) of a given Deck. If the deck was not passed over or invalid it should return an error. A count parameter needs to be provided to define how many cards to draw from the deck.

The response needs to return a JSON that would include:

all the drawn cards cards (card object)

{
    "cards": [
        {
            "value": "QUEEN",
            "suit": "HEARTS",
            "code": "QH"
        },
        {
            "value": "4",
            "suit": "DIAMONDS",
            "code": "4D"
        }
    ]
}

Designing the domain

Since the domain is an integral part of our application we'll start designing our system from the domain.

Let's encode our Shape and Rank types as iota. If you're acquainted with other languages you might think of it as an enum which is pretty neat since our task assumes some sort of build-in order so we might leverage underlying numerical value just for that matter.

type Shape uint8

const (
    Spades Shape = iota
    Diamonds
    Clubs
    Hearts
)

type Rank int8

const (
    Ace Rank = iota
    Two
    Three
    Four
    Five
    Six
    Seven
    Eight
    Nine
    Ten
    Jack
    Queen
    King
)

With that done we can encode Card as a combination of its shape and rank

type Card struct {
    Rank  Rank
    Shape Shape
}

One of the functions of domain-driven design is making illegal states unrepresentable but since all combinations of ranks and shapes are valid creating a card is pretty straightforward

func CreateCard(rank Rank, shape Shape) Card {
    return Card{
        Rank:  rank,
        Shape: shape,
    }
}

Now let's have a look at the deck

type Deck struct {
    DeckId   uuid.UUID
    Shuffled bool
    Cards    []Card
}

The deck will exhibit three operations: create a deck, draw cards and count remaining cards.

func CreateDeck(shuffled bool, cards ...Card) Deck {
    if len(cards) == 0 {
        cards = initCards()
    }
    if shuffled {
        shuffleCards(cards)
    }

    return Deck{
        DeckId:   uuid.New(),
        Shuffled: shuffled,
        Cards:    cards,
    }
}

func DrawCards(deck *Deck, count uint8) ([]Card, error) {
    if count > CountRemainingCards(*deck) {
        return nil, errors.New("DrawCards: Insuffucient amount of cards in deck")
    }
    result := deck.Cards[:count]
    deck.Cards = deck.Cards[count:]
    return result, nil
}

func CountRemainingCards(d Deck) uint8 {
    return uint8(len(d.Cards))
}

Notice that when drawing cards we check whether we have a sufficient amount of cards to perform the operation. In order to signal in case we're unable to proceed we take advantage of Go multiple return values feature.

At this point we may observe one of the key benefits of clean architecture: the core domain logic has no external dependencies which greatly simplifies unit-testing. While most of them are trivial and we'll omit them for the sake of brevity, let's have a look at those that verify whether the deck is shuffled

func TestCreateDeck_ExactCardsArePassed_Shuffled(t *testing.T) {
    jackOfDiamonds := CreateCard(Jack, Diamonds)
    aceOfSpades := CreateCard(Ace, Spades)
    queenOfHearts := CreateCard(Queen, Hearts)
    cards := []Card{jackOfDiamonds, aceOfSpades, queenOfHearts}
    deck := CreateDeck(false, cards...)
    deckCardsCount := make(map[Card]int)
    for _, resCard := range deck.Cards {
        value, exists := deckCardsCount[resCard]
        if exists {
            value++
            deckCardsCount[resCard] = value
        } else {
            deckCardsCount[resCard] = 1
        }
    }
    for _, inputCard := range cards {
        value, found := deckCardsCount[inputCard]
        assert.True(t, found, "Expected all cards to be present")
        assert.Equal(t, 1, value, "Expected cards not to be duplicate")
    }
}

Obviously, we cannot verify the order of shuffled cards. What we can do instead is to verify that the shuffled deck satisfies properties of interest which are that we have every card present and we have no duplicate cards in our deck. Such a technique closely resembles property-based testing.

As a side note, it's worth mentioning that in order to eliminate boilerplate assertion code we take advantage of testify library.

Providing the API

Let's start off with defining routes.

func main() {
    r := gin.Default()
    r.POST("/create-deck", api.CreateDeckHandler)
    r.GET("/open-deck", api.OpenDeckHandler)
    r.PUT("/draw-cards", api.DrawCardsHandler)
    r.Run()
}

Some of the readers may be confused by the fact that the create-deck endpoint according to the requirements listed above accepts parameters as a part of URL request and might consider making this endpoint accept GET requests instead of POST. However, an important prerequisite of GET requests is that they exhibit idempotency with is not the case for this endpoint. This is the exact reason why we stick with POST.

Handlers follow the same patterns. We parse query parameters, based on them we create a domain entity, perform operations upon it, update the storage and return specialized DTO. Let's have a look at more details.

type CreateDeckArgs struct {
    Shuffled bool   `form:"shuffled"`
    Cards    string `form:"cards"`
}

type OpenDeckArgs struct {
    DeckId string `form:"deck_id"`
}

type DrawCardsArgs struct {
    DeckId string `form:"deck_id"`
    Count  uint8  `form:"count"`
}

func CreateDeckHandler(c *gin.Context) {
    var args CreateDeckArgs
    if c.ShouldBind(&args) == nil {
        var domainCards []domain.Card
        if args.Cards != "" {
            for _, card := range strings.Split(args.Cards, ",") {
                domainCard, err := parseCardStringCode(card)
                if err == nil {
                    domainCards = append(domainCards, domainCard)
                } else {
                    c.String(400, "Invalid request. Invalid card code "+card)
                    return
                }
            }
        }
        deck := domain.CreateDeck(args.Shuffled, domainCards...)
        storage.Add(deck)
        dto := createClosedDeckDTO(deck)
        c.JSON(200, dto)
        return
    } else {
        c.String(400, "Ivalid request. Expecting query of type ?shuffled=<bool>&cards=<card1>,<card2>,...<cardn>")
        return
    }
}

func OpenDeckHandler(c *gin.Context) {
    var args OpenDeckArgs
    if c.ShouldBind(&args) == nil {
        deckId, err := uuid.Parse(args.DeckId)
        if err != nil {
            c.String(400, "Bad Request. Expecing request in format ?deck_id=<uuid>")
            return
        }
        deck, found := storage.Get(deckId)
        if !found {
            c.String(400, "Bad Request. Deck with given id not found")
            return
        }
        dto := createOpenDeckDTO(deck)
        c.JSON(200, dto)
        return
    } else {
        c.String(400, "Bad Request. Expecing request in format ?deck_id=<uuid>")
        return
    }
}

func DrawCardsHandler(c *gin.Context) {
    var args DrawCardsArgs
    if c.ShouldBind(&args) == nil {
        deckId, err := uuid.Parse(args.DeckId)
        if err != nil {
            c.String(400, "Bad Request. Expecing request in format ?deck_id=<uuid>")
            return
        }
        deck, found := storage.Get(deckId)
        if !found {
            c.String(400, "Bad Request. Expecting request in format ?deck_id=<uuid>&count=<uint8>")
            return
        }
        cards, err := domain.DrawCards(&deck, args.Count)
        if err != nil {
            c.String(400, "Bad Request. Failed to draw cards from the deck")
            return
        }
        var dto []CardDTO
        for _, card := range cards {
            dto = append(dto, createCardDTO(card))
        }
        storage.Add(deck)
        c.JSON(200, dto)
        return
    } else {
        c.String(400, "Bad Request. Expecting request in format ?deck_id=<uuid>&count=<uint8>")
        return
    }
}

Defining OpenAPI specification

The way we should treat OpenAPI specification is not solely as a fancy docs generator (although it is sufficient for the sake of our article) but also as a standard for describing REST APIs that simplifies their consumption for the clients.

Let's start off with declarative comments decorating our main method. These comments will be used later to automatically generate Swagger Specification. Here you can look up the format.

// @title Deck Management API
// @version 0.1
// @description This is a sample server server.
// @termsOfService http://swagger.io/terms/

// @contact.name API Support
// @contact.url http://www.swagger.io/support
// @contact.email support@swagger.io

// @license.name Apache 2.0
// @license.url http://www.apache.org/licenses/LICENSE-2.0.html

// @host localhost:8080
// @BasePath /
// @schemes http
func main() {

The same goes for our handler. Let's take a look at one of them as an example.

// CreateDeckHandler godoc
// @Summary Creates new deck.
// @Description Creates deck that can be either shuffled or unshuffled. It can accept the list of exact cards which can be shuffled or unshuffled as well. In case no cards provided it returns a deck with 52 cards.
// @Accept */*
// @Produce json
// @Param shuffled query bool  true  "indicates whether deck is shuffled"
// @Param cards    query array false "array of card codes i.e. 8C,AS,7D"
// @Success 200 {object} ClosedDeckDTO
// @Router /create-deck [post]
func CreateDeckHandler(c *gin.Context) {

Now let's pull Swagger libraries

go get -v github.com/swaggo/swag/cmd/swag
go get -v github.com/swaggo/gin-sagger
go get -v github.com/swaggo/files

Now we're going to generate the specification

swag init -g main.go --output docs

This command will generate needed files inside the docs folder.

The next step is updating our main.go file with the necessary imports

_ "toggl-deck-management-api/docs"
swaggerFiles "github.com/swaggo/files"
ginSwagger "github.com/swaggo/gin-swagger"

And the endpoint

url := ginSwagger.URL("/swagger/doc.json")
r.GET("/swagger/*any", ginSwagger.WrapHandler(swaggerFiles.Handler, url))

With all that done, now we can run our application and see documentation generated by Swagger.

Containerizing API

Last but not least is how we are going to deploy our application. The traditional way of doing this is installing the runtime on a dedicated server and running the application over the runtime installed.

Containerization is the convenient way to package runtime along with the application which might be convenient if we want to utilize auto-scale functionality and we might not have all the needed servers with the environment installed at our disposal.

Docker is the most popular containerization solution so we'll take advantage of it. To do so we'll create dockerfile at the root of our project.

The first thing we'll do is chose the runtime image we'll base our application upon

FROM golang:1.18-bullseye

After that, we'll copy the source into the workdirectory and build it

RUN mkdir /app
COPY . /app
WORKDIR /app
RUN go build -o server .

The last step is exposing the port to the outside world and running the application

EXPOSE 8080
CMD [ "/app/server" ]

Now, granted Docker is installed on our machine, we can run the app with

docker build -t <image-name> .
docker run -it --rm -p 8080:8080 <image-name>

Conclusion

In this article, we've covered the holistic process of writing clean architecture API in Go. Starting from the well-tested domain, providing an API layer for it, documenting it using OpenAPI standard, and packaging our runtime together with the app and thus simplifying the deployment process.

Property-based tests and clean architecture are perfect fit

Bohdan Stupak — Tue, 11 Oct 2022 15:31:04 +0000

It has been written a lot about the clean architecture. Its main value is the ability maintain free from side effects domain layer that allows us to test core business logic without leveraging heavy mocks.

However, when it comes to designing tests for out pure domain logic quite often we don't tend to be so picky. Unit testing contains many traps such as overspecified software. But even when it comes to testing pure functions which may seem as a pretty straightforward process we may encounter some pitfalls.

One of them is that when writing unit tests we rely on some sort of arbitrary magic numbers. While we may guarantee that our function works correctly at given points we can't guarantee that it works at every point. An alternative would be to check whether the function satisfies some criteria continuously.

And this is what property-based testing is aimed at. Instead of verifying output at hardcoded input points, it checks the properties of the function you define with a multitude of generated values.

Let's look at the code example to see how that works. Below is the example from my project Kyiv Station Walk. You can see the function that takes a collection of checkpoints from the domain and transforms it so it conforms to the rules of the presentation layer.

let removeRedundantCheckpoints (checkPoints : Location[]) =
    let checkPointsMaxCount = 5
    let isStartOrEndOfTheRoute (checkPoints : Location[]) i =
       i = 0 || i = checkPoints.Length - 1
    let euclidianDistance c1 c2 =
        Math.Pow(float(c1.lattitude - c2.lattitude), float(2)) + Math.Pow(float(c1.longitude - c2.longitude), float(2))
    if checkPoints.Length <= 5 then
        checkPoints
    else
        checkPoints
        |> Array.mapi(fun i c ->
            if isStartOrEndOfTheRoute checkPoints i then
                {
                    index = i
                    checkPoint = c
                    distanceToNextCheckPoint = float(1000000)
                }
            else
                {
                    index = i
                    checkPoint = c
                    distanceToNextCheckPoint = euclidianDistance checkPoints.[i+1] c
                }
        )
        |> Array.sortByDescending(fun i -> i.distanceToNextCheckPoint)
        |> Array.take(checkPointsMaxCount)
        |> Array.sortBy(fun i -> i.index)
        |> Array.map(fun i -> i.checkPoint)

We can supply some arbitrary arrays of checkpoints and check the output or instead we can think about some properties that our function should satisfy. Here are these properties expressed in code.

open FsCheck.Xunit
open RouteModels

module RemoveRedundantCheckpointsTests =

    let ``result array contains no more than 5 items`` input mapFn =
        let res = mapFn input
        Array.length res <= 5

    [<Property>]
    let maxLength x =
        ``result array contains no more than 5 items`` x removeRedundantCheckpoints

    let ``result contains first point from input`` (input: Location[]) (mapFn : Location[] -> Location[]) =
        if Array.length input = 0 then
            true
        else
            let res = mapFn input
            res.[0] = input.[0]

    [<Property>]
    let firstItem x =
        ``result contains first point from input`` x removeRedundantCheckpoints

    let ``result contains last point from input`` (input: Location[]) (mapFn : Location[] -> Location[]) =
        if Array.length input = 0 then
            true
        else
            let res = mapFn input
            res.[res.Length-1] = input.[input.Length-1]

    [<Property>]
    let lastItem x =
        ``result contains last point from input`` x removeRedundantCheckpoints

    let ``result contains only points from input`` input mapFn =
        let res = mapFn input
        Array.length (Array.except input res) = 0

    [<Property>]
    let onlyInput x =
        ``result contains only points from input`` x removeRedundantCheckpoints

As you can see from the imports statement we're relying on FsCheck to generate some random values for us.

Later in the code, we declare a higher-order function that accepts the mapper function and input array and returns a boolean condition that checks whether the property is satisfied. Double backticks is a convenient F# feature that allows us to express property in a natural language.

The test is decorated with Property attribute and accepts input generated by FsCheck as well as removeRedundantCheckpoints function which is subject to change. With such a setup we can check whether the function under tests satisfies provided properties with the multitude of random values generated by a library.

Conclusion

When it comes to testing a lot of teams really put the same effort into designing a test suite as into the application code. And even those who do rarely consider something outside of the traditional testing pyramid. Still, property-based testing represents a nice option for pure logic that usually resides in the domain layer or in the mapping layers of your application.

Crafting generative art with javascript generators

Bohdan Stupak — Wed, 28 Sep 2022 12:53:44 +0000

Apart from career programming and management tasks, I spend my free time doing creative coding. Most of it is programming music with SuperCollider but recently I was also involved in some visual art.

Initial project structure

The idea behind the project is to study the relation between colors by randomly morphing them into each other. The program which does this is the array of program frames which are the color and number of frame executions. In the initial iteration, the capacity of the array is 5.

function generateProgram() {
    var numberOfFrames = 500;
    var numberOfIterations = 5000;
    var previousIteration = 0;
    var program = [];
    for (var i = 0; i < numberOfFrames; i++) {
        var currentIteration = randomInt(numberOfIterations) + previousIteration;
        program[i] = {
            color: randomHsl(),
            latestIteration: currentIteration
        }
        previousIteration = currentIteration;
    }
    return program;
}

We're generating colors in HSL format since random colors in RGB look pretty similar and the result wouldn't be as pleasing aesthetically.

Program execution is the infinite loop that selects a couple of random points on the canvas and blends them with the color of the current program frame. Once we've reached the target number of iterations we switch to the next program frame.

function executeProgram(program) {
    var points = getInitialCanvas(program);
    drawOnCanvas(points);
    var iteration = 0;
    var currentProgramFrameIndex = 0;
    var currentProgramFrame = program[currentProgramFrameIndex];
    function run() {
        setTimeout(function() {
            for (var k = 0; k < 10; k++) {
                window.requestAnimationFrame(run);
                iteration++;
                var i = randomInt(height)-1;
                var j = randomInt(width)-1;
                points[i][j] = blendColors(points[i][j], currentProgramFrame.color);
                if (iteration == currentProgramFrame.latestIteration
                        && currentProgramFrameIndex !== program.length - 1) {
                    currentProgramFrameIndex++;
                    currentProgramFrame = program[currentProgramFrameIndex];
                }
                drawPointOnCanvas(i, j, points[i][j]);
            }
        }, 1);
    }
    run();
}

We can observe that we're instructing the browser to schedule animation using window.requestAnimationFrame method. This method accepts a callback which will be executed when it is time to paint the animation. So by calling run method as the callback we're effectively creating the infinite loop.

In order for our infinite loop not to block repainting of the canvas, we call each iteration inside the setTimeout function.

Using generators

The obvious downside of this approach is that we have a limited amount of frames. Instead, we want our animation to last forever. One of the possible ways is to provide each new program frame via generators

Changes are pretty straightforward. Instead of generating the program as an array of frames, we define a generator function that will return a new frame when requested.

function* generateProgram()  {
    while (true) {
        var numberOfIterations = 5000;
        var program = [];
        var currentIteration = randomInt(numberOfIterations)
        program = {
            color: randomHsl(),
            latestIteration: currentIteration
        }
        previousIteration = currentIteration;
        yield program;
    }
}

To request the frame we instantiate the generator object by calling the function and then calling next method.

var programGenerator = generateProgram()
var program = programGenerator.next().value;

Now we can request each program frame, once the previous is completed.

function executeProgram(program) {
    var points = getInitialCanvas(program);
    drawOnCanvas(points);
    var iteration = 0;
    function run() {
        setTimeout(function() {
            for (var k = 0; k < 10; k++) {
                window.requestAnimationFrame(run);
                iteration++;
                var i = randomInt(height)-1;
                var j = randomInt(width)-1;
                points[i][j] = blendColors(points[i][j], program.color);
                if (iteration == program.latestIteration) {
                    program = programGenerator.next().value
                }
                drawPointOnCanvas(i, j, points[i][j]);
            }
        }, 1);
    }
    run();

}

Sidenote: Fixing the memory leak

When you pay attention to this program long enough you might notice that it starts consuming more and more memory.

Upon examining memory profile in my Firefox devtools I can notice that the primary source of allocation is setTimeout function.

Turns out that calls to requestAnimationFrame are stacked to map of animation frame callbacks. And if the browser is not keeping up with the growth of callback's map it's starts growing up in size consuming more and more memory.

So the fix is as simple as increasing timeout duration.

Leveraging lazy evaluation

Bohdan Stupak — Wed, 22 Jun 2022 15:38:21 +0000

I'm usually skeptical about leetcode tasks since this is not something you'll encounter in your daily line-of-business code. But the idea behind this particular task has fascinated me. So I'll break it down here.

I'll provide a bit simplified version of the task so you could capture the gist of it more easily.

Write an API that generates fancy sequences using the append, addAll, and multAll operations.

Implement the Fancy class:

    Fancy() Initializes the object with an empty sequence.
    void append(val) Appends an integer val to the end of the sequence.
    void addAll(inc) Increments all existing values in the sequence by an integer inc.
    void multAll(m) Multiplies all existing values in the sequence by an integer m.
    int getIndex(idx) Gets the current value at index idx (0-indexed). If the index is greater or equal than the length of the sequence, return -1.

Example 1:

Input
["Fancy", "append", "addAll", "append", "multAll", "getIndex", "addAll", "append", "multAll", "getIndex", "getIndex", "getIndex"]
[[], [2], [3], [7], [2], [0], [3], [10], [2], [0], [1], [2]]
Output
[null, null, null, null, null, 10, null, null, null, 26, 34, 20]

Explanation
Fancy fancy = new Fancy();
fancy.append(2);   // fancy sequence: [2]
fancy.addAll(3);   // fancy sequence: [2+3] -> [5]
fancy.append(7);   // fancy sequence: [5, 7]
fancy.multAll(2);  // fancy sequence: [5*2, 7*2] -> [10, 14]
fancy.getIndex(0); // return 10
fancy.addAll(3);   // fancy sequence: [10+3, 14+3] -> [13, 17]
fancy.append(10);  // fancy sequence: [13, 17, 10]
fancy.multAll(2);  // fancy sequence: [13*2, 17*2, 10*2] -> [26, 34, 20]
fancy.getIndex(0); // return 26
fancy.getIndex(1); // return 34
fancy.getIndex(2); // return 20

Constraints:
1 <= val, inc, m <= 100
0 <= idx <= 105
At most 105 calls total will be made to append, addAll, multAll, and getIndex.
Number of read operation surpasses number of mutations

The naive solution would be to perform each operation on all elements of an array and then get element at desired index. However this solution is suboptimal since we may need only a couple of items from the collection so there's clearly no need to calculate each item for such a case.

Alternatively, we might want to compute each item of the array on-demand leveraging some sort of lazy evaluation. In a such case, we need to ensure that we don't calculate the same item twice. For such an occasion we need to store the result of the calculation in an intermediary data structure. map would be the best fit for such structure since it allows us to get the desired element at O(1) time. The element index will serve as a key while the value of the map is the result of the calculation.

So the types are declared as follows:

type Operation struct {
    operationCode   int8
    operand         int
    valuesLastIndex int32
}

type Fancy struct {
    values     []int8
    operations []Operation
    cache      map[int]int
}

When appending values we're just increasing array of values

func (this *Fancy) Append(val int) {
    this.values = append(this.values, int8(val))
}

When adding or multiplying we're increasing array of operations. No evaluation happens at this point in time.

func (this *Fancy) AddAll(inc int) {
    this.operations = append(this.operations, Operation{-2, inc, int32(len(this.values))})
    this.cache = make(map[int]int)
}

func (this *Fancy) MultAll(m int) {
    this.operations = append(this.operations, Operation{-1, m, int32(len(this.values))})
    this.cache = make(map[int]int)
}

Evaluation occurs on-demand. We either try to hit a cache or calculate it if we miss a cache

func (this *Fancy) GetIndex(idx int) int {

    if idx < 0 || idx >= len(this.values) {
        return -1
    }

    if val, ok := this.cache[idx]; ok {
        return val
    }

    var vv uint64

    vv = uint64(this.values[idx])

    for _, v := range this.operations {
        if idx >= int(v.valuesLastIndex) {
            continue
        }

        switch os := v.operationCode; os {
        case -2:
            vv += uint64(v.operand)
        case -1:
            vv *= uint64(v.operand)
        }
    }

    if this.cache == nil {
        this.cache = make(map[int]int)
    }

    this.cache[idx] = int(vv)

    return int(vv)
}

As some of you might have noticed each mutation operation leads to clearing the cache. So this method might be ineffective when the number of mutations surpasses the number of reads. So (as with every other solution in software engineering) it is advisable to evaluate your constraints before applying methods from your toolbox.

Evaluating cache

One thing that is worth noting is that cache we construct occupies space in memory. That means that we should evaluate using it so it won't turn into a huge memory leak. For the same reason, you should have cache expiration strategy for most of your projects.

Since we know that the maximum number of operations is 10^5 we can deduce that cache size will reach its peak we won't perform any additions or multiplications thus clearing the cache but instead half of the operations will be appending items and the other one will be evaluating them thus calculating new item each time.

func BenchmarkMemoryConsumption() {
    var m1, m2 runtime.MemStats
    runtime.GC()
    runtime.ReadMemStats(&m1)
    const MaxOperations int = 100000
    fancy := Constructor()
    for i := 0; i < MaxOperations/2; i++ {
        fancy.Append(100)
    }
    for i := 0; i < MaxOperations/2; i++ {
        fancy.GetIndex(i)
    }
    runtime.ReadMemStats(&m2)
    fmt.Println("total:", m2.TotalAlloc-m1.TotalAlloc)
    fmt.Println("mallocs:", m2.Mallocs-m1.Mallocs)
}

I've found built-in benchmarking functionality providing allocs per second metric irrelevant for this case so instead I've came up with the benchmark as above.
The output is

total: 3082976
mallocs: 1986

3MBs for the cache of maximum size? Easy for most of the line-of-business applications.

Conclusion

In case we're presented with the perspective of doing a lot of wasteful computations in a read-intensive scenario we can omit that by restoring to computing values when necessary using lazy evaluation and caching. Still, when using a cache its size and expiration strategy should be taken into account.

It's not about how you inject your services, it's about how you test them

Bohdan Stupak — Mon, 02 May 2022 14:01:52 +0000

It has been written a lot about the value of unit testing and still a lot of developers may have witnessed codebases with unit tests being too brittle or rarely discovering actual defects in software. Also some have questioned default architectural style supposed to make code testable. These are the reasons why a lot of developers openly question unit-testing while others just silently sabotage the process of writing unit-tests.

In this article I offer my take on the aforementioned issues.

You can check out the code provided in this repository. During the course of the article I present two different designs which reside in two different branches, so feel free to explore them both.

Setup

Let us take a look at simple controller with service injected. Although I use the term "controller" I don't use any frameworks for the sake of example simplicity. Still you might imagine that we're talking about one of the popular MVC frameworks.

public class ItemService
{
    public string Serialize(Item input)
    {
        switch (input.Type)
        {
            case ItemType.String:
                return input.Value;
            case ItemType.Geo:
                return SeriazlieGeo(input.Value);
            case ItemType.Range:
                return SerializeRange(input.Value);
            default:
                throw new ArgumentOutOfRangeException(nameof(input.Type));
        }
    }

    private string SerializeRange(string value)
    {
        var items = value.Split(new[] { '-', ' ' }, StringSplitOptions.RemoveEmptyEntries);
        if (items.Length != 2)
        {
            throw new ArgumentException(nameof(value));
        }
        return $"gte:{items[0]},lte:{items[1]}";
    }

    private string SeriazlieGeo(string value)
    {
        var items = value.Split(new[] { ',', ' ' }, StringSplitOptions.RemoveEmptyEntries);
        if (items.Length != 2)
        {
            throw new ArgumentException(nameof(value));
        }
        return $"lat:{items[0]},lon:{items[1]}";
    }
}

public class ItemController
{
    private readonly ItemService _serializer;

    public ItemController(ItemService serializer)
    {
        _serializer = serializer;
    }

    public string Process(Item item)
    {
        return $"Output is: {_serializer.Serialize(item)}";
    }
}

Now let's cover this code with tests.

public class ItemsServiceTests
{
    private ItemController CreateSut()
    {
        return new ItemController(new ItemService());
    }

    public static IEnumerable<object[]> SerializeTestData => new List<object[]>
    {
        new object [] { new Item
        {
            Type = ItemType.String,
            Value = "test value"
        }, "Output is: test value" },
        new object [] { new Item
        {
            Type = ItemType.Geo,
            Value = "45,54"
        }, "Output is: lat:45,lon:54" },
        new object [] { new Item
        {
            Type = ItemType.Range,
            Value = "45-54"
        }, "Output is: gte:45,lte:54" },

    };

    [Theory]
    [MemberData(nameof(SerializeTestData))]
    public void Serialize(Item item, string expectedResult)
    {
        //arrange
        var sut = CreateSut();

        //act
        var actualResult = sut.Process(item);

        //assert
        Assert.Equal(actualResult, expectedResult);
    }
}

So is the controller written in a testable fashion? Sure, we've covered it with tests completely! Can we improve it somehow? I don't think so. But let us examine options we can find in some codebases that I've encountered myself as well.

Introducing interface

Some developers who are striving for loose coupling may object that injecting concrete realization into the controller is a violation of SOLID principles. Namely the principle of dependency inversion. So let's adhere to these principles and introduce an interface and inject it into service.

public interface IItemService
{
    string Serialize(Item input);
}

public class ItemController
{
    private readonly IItemService _serializer;

    public ItemController(IItemService serializer)
    {
        _serializer = serializer;
    }

    public string Process(Item item)
    {
        return $"Output is: {_serializer.Serialize(item)}";
    }
}

Now given we don't change out tests we see them still passing. This means that we've executed refactoring and our test suite assured us that we didn't break things. Exactly why we write our unit tests!

Did adhering to SOLID somehow improved our code in this particular case? I don't think so. It was already concise and testable. Did it make it worse? Some of us (and me as well) who believe that code is a liability, not an asset think so. But is it critical? Frankly speaking, even when I'm in charge of processes but my team feels that strictly conforming laws of SOLID has its values, I'd rather adhere to the team will than try breaking it.

Side note: abstracting away volatile dependencies

Critical readers may tell that I'm battling a strawman here and they might be right. While there is not much sense in abstracting away dependency by the interface in the example I've provided this is quite a useful technique, the real benefit to testability comes when you abstract away volatile dependencies. By the term "volatile dependencies" I mean such that provide observable side effects (such as databases, email providers, etc). Using these dependencies directly in your test suite can make your tests unstable since they rely on external resources. So it perfectly makes sense to replace them with a test double that inherits injected interface.

However other techniques are possible. One of them is extracting such side-effectful interactions into separate modules while unit-testing pure logic. I won't dive into details much on that matter since there already exists excellent explanation of this technique.

It's worth noting that some frameworks come with a built-in option to abstract-away volatile dependencies not based on interfaces. One such example is EFCore Inmemory Provider.

Refactoring(?) unit-tests

So since both designs are quite fine what is the point? As some of you might have guessed from the article our attention will be devoted to the unit-test suite. As the name implies unit-tests are designed to test separate units contrary to integration tests which test multiple units in integration.

The thinking I find in many codebases is the natural unit of code is class so tests from the example are refactored as follows

public class ItemControllerTests
{
    private ItemController CreateSut()
    {
        var itemServiceMock = new Mock<IItemService>(MockBehavior.Strict);
        itemServiceMock.Setup(e => e.Serialize(It.IsAny<Item>())).Returns("serialized");
        return new ItemController(itemServiceMock.Object);
    }

    [Fact]
    public void ProcessWrapsSerializedOutput()
    {
        //arrange
        var sut = CreateSut();

        //act
        var res = sut.Process(new Item { });

        //assert
        Assert.Equal("Output is: serialized", res);
    }
}

public class ItemsServiceTests
{
    private ItemService CreateSut()
    {
        return new ItemService();
    }

    public static IEnumerable<object[]> SerializeTestData => new List<object[]>
    {
        new object [] { new Item
        {
            Type = ItemType.String,
            Value = "test value"
        }, "test value" },
        new object [] { new Item
        {
            Type = ItemType.Geo,
            Value = "45,54"
        }, "lat:45,lon:54" },
        new object [] { new Item
        {
            Type = ItemType.Range,
            Value = "45-54"
        }, "gte:45,lte:54" },

    };

    [Theory]
    [MemberData(nameof(SerializeTestData))]
    public void Serialize(Item item, string expectedResult)
    {
        //arrange
        var sut = CreateSut();

        //act
        var actualResult = sut.Serialize(item);

        //assert
        Assert.Equal(actualResult, expectedResult);
    }
}

Now we test both ItemService and also check that the correct method of ItemService is called inside ItemController. Is this design actually better? Let's take a look.

Evolution of design

You may have noticed that some of our business logic (if you can apply this term to such an over-simplistic example) namely wrapping serialized item with supplementary text resides inside ItemController. Let's say we want to follow thin controllers principle and extract this code inside the service.

public string Process(Item item)
{
    var serializedOutput = _serializer.Serialize(item);
    return _serializer.Wrap(serializedOutput);
}

Since we've refactored our code let's run test suite to check if we didn't break things.

What happened?

Moq.MockException : IItemService.Wrap("serialized") invocation failed with mock behavior Strict.
All invocations on the mock must have a corresponding setup.

On the contrary, if we stick to our original testing strategy when testing multiple classes at once, our test suite will be green.

Turns out, the testing style from "Refactoring unit-tests" section represents a case of overspecified software. Instead of focusing on verifying the behavior, we're verifying implementation details which are subject to change. Such tests don't provide any additional confidence in our code but are brittle which causes dissatisfaction with unit-tests in general. Think about it this way: would your stakeholders ever care that you call the method of your ItemService two times or exactly once?

That leads us to a conclusion that when we speak about testing a unit we should think about a unit of behavior, not a unit of code! Such tests allow us to focus on important aspects of the system under test thus increasing the value of our test suite.

Side note: respecting SRP

Critical readers may observe that after we introduced another method to ItemService it started to violate the single responsibility principle. In this case, it was made solely to illustrate a case of brittle tests but generally speaking you should always take care when working with classes that have the suffix Service or Manager in their names as this is the first flag that indicates that responsibility of the class is defined not clearly enough.

Conclusion

The main goal of this article is to show that we as engineers should understand the benefits and shortcomings of the principles we're applying instead of just blindly following them. For that reason, we've taken a tour over service composition and harnessing a test suite. As we have seen following principles without understanding what they are about, leads us to brittle design. And while our first reaction may be to question principles themselves the main thing we really have to ask ourselves is whether we're applying them correctly.