DEV Community: Yonatan Karp-Rudin

XKCD: From Tile to Picture - Seeing the Full Image

Yonatan Karp-Rudin — Fri, 28 Jun 2024 22:00:54 +0000

Introduction

If there's one thing I can say about myself, it's that I love challenges related to software programming. They don't need to be meaningful or even useful, but I enjoy the challenge. That's why when I saw this message on my LinkedIn, I had to take on the challenge!

If you're unaware of what XKCD is, here's what Wikipedia has to say about it:

xkcd, sometimes styled XKCD, is a webcomic created in 2005 by American author Randall Munroe. The comic's tagline describes it as "a webcomic of romance, sarcasm, math, and language". Munroe states on the comic's website that the name of the comic is not an initialism but "just a word with no phonetic pronunciation".

The subject matter of the comic varies from statements on life and love to mathematical, programming, and scientific in-jokes. Some strips feature simple humor or pop-culture references. It has a cast of stick figures, and the comic occasionally features landscapes, graphs, charts, and intricate mathematical patterns such as fractals.

I had never seen this specific comic before (see https://xkcd.com/1110), but I was up for the challenge. I opened the link and discovered a simple 4-frame comic:

The catch is, the last frame is actually interactive and allows the reader to move inside it:

With that knowledge in hand, the task was clear: to download the entire image within the last frame of the comic!

Step 1: Trying the Easy Way

The first attempt to solve the problem was taking the easy way. I checked the RSS feed of XKCD for this specific comic. To do so, I used the link https://xkcd.com/1110/info.0.json.

The result was as follows:

{
  "month": "9",
  "num": 1110,
  "link": "",
  "year": "2012",
  "news": "",
  "safe_title": "Click and Drag",
  "transcript": "[[A character is dangling from a balloon. All text appears in rectangular bubbles.]]\nCharacter: From the stories\nCharacter: I expected the world to be sad\nCharacter: And it was\n\nCharacter: And I expected it to be wonderful.\n\nCharacter: It was.\n\n((The last panel, unusually, is infinitely large, and this transcript is not wide enough to contain it. The part you can see in a normal browser window goes as follows.))\n[[ The same character is dangling above the ground, next to an intricately drawn tree with no leaves. ]]\nCharacter: I just didn't expect it to be so \nbig\n.\n\n{{Title text: Click and drag.}}",
  "alt": "Click and drag.",
  "img": "https://imgs.xkcd.com/comics/click_and_drag.png",
  "title": "Click and Drag",
  "extra_parts": {
    "pre": "",
    "headerextra": "",
    "post": "\n<div class=\"map\"><div class=\"ground\"></div></div>\n<script type=\"text/javascript\" src=\"/s/a3c5de.js\"></script>\n<script type=\"text/javascript\" src=\"/s/d28668.js\"></script>\n",
    "imgAttr": ""
  },
  "day": "19"
}

However, checking the image URL ended in very disappointing results:

Step 2: Downloading the Image

Finding the Image Size

To try and understand how to download the image, the first thing I did was to check the website's source code, hoping it would help me find the full image.

While checking the source code, I actually found a very interesting piece of code:

<div class="map" style="...">
    <div class="ground" style="..."></div>
    <img class="tile2n1w" src="http://imgs.xkcd.com/clickdrag/2n1w.png" style="...">
    <img class="tile1n1w" src="http://imgs.xkcd.com/clickdrag/1n1w.png" style="...">
    <img class="tile2n2w" src="http://imgs.xkcd.com/clickdrag/2n2w.png" style="...">
    <img class="tile1n2w" src="http://imgs.xkcd.com/clickdrag/1n2w.png" style="...">
    <img class="tile2n3w" src="http://imgs.xkcd.com/clickdrag/2n3w.png" style="...">
    <img class="tile1n3w" src="http://imgs.xkcd.com/clickdrag/1n3w.png" style="...">
    <img class="tile3n3w" src="http://imgs.xkcd.com/clickdrag/3n3w.png" style="...">
    ...
</div>

This part of the code was updated whenever I moved on the map and reached the end of a tile in the frame. After playing with it a bit, I could see a clear pattern in the file names. They contain a number followed by the letter n (for north) or s (for south), indicating whether the image is above or below the central line of the image, then a number again, and the letter w (for west) or e (for east).

So, for example, the image 1n2w.png is located in the central row of the image, two tiles to the left. That was a great discovery! Now, I can try to see the limits of the image by calling the server and checking the results.

After some experimentation, I found out that the maximum available tile to the right is 48 (http://imgs.xkcd.com/clickdrag/1n48e.png), and the maximum available tile to the left is 33 (http://imgs.xkcd.com/clickdrag/1n33w.png) — wow, that's a big image!

As for the height of the image, this was a much greater challenge. Apparently, tiles that do not contain any image are not stored on the server and are rendered as a color in the browser. This means that some columns might have 5 tiles, while others have 20. Moreover, in theory, it could contain "islands" of tiles that are not connected to anything but themselves. See the image below as an example:

Ok, so what do we do?

When in doubt, brute force is always a good solution! Since the number of tiles is (probably) not too big, I can spend some finite reasonable amount of time trying to download a tile, if it's not there, just continue to the next one. For this task, I decided to try and download a 50x50 tile matrix.

Downloading the Tiles

At this point, I started to be lazy. I asked my best friend ChatGPT to help me out and create a code that would download the tiles for me using Kotlin. The code was pretty ugly, but if you're curious, you can find the final results on GitHub at the end of the article.

As usual, I had to alter the code and add some protection in case a tile didn't exist so the code would not crash. The problem was, that downloading the tiles took too much time, and I was eager to continue to the next part.

What can we do? Well, we can leverage the power of parallelism, and in our context, Kotlin's coroutines!

Instead of trying to fetch the entire matrix in one thread, I divided it into four equal segments and started to download the tiles of each segment:

launch(Dispatchers.IO) {
    launch { fetcher.repeatInDirection(latitude = "n", longitude = "w") }
    launch { fetcher.repeatInDirection(latitude = "n", longitude = "e") }
    launch { fetcher.repeatInDirection(latitude = "s", longitude = "w") }
    launch { fetcher.repeatInDirection(latitude = "s", longitude = "e") }
}

A few minutes passed by, and I had all 160 tiles on my machine. Success! 🎉

Building the Image from the Tiles

The next part was a bit more challenging: taking the different tiles and combining them into one big image. First, it required locating each of the tiles in its position in the final image. Again, being the lazy software engineer I am, I asked ChatGPT to generate a code for me that would build the image from those files based on the file name encoding.

Running the code given by ChatGPT ended with the exact same results I opened this article with, the library I used (BufferedImage by awt) throwing the following exception:

Exception in thread "main" java.lang.IllegalArgumentException: Dimensions (width=165888 height=28672) are too large
    at java.desktop/java.awt.image.SampleModel.<init>(SampleModel.java:131)
    at java.desktop/java.awt.image.SinglePixelPackedSampleModel.<init>(SinglePixelPackedSampleModel.java:144)
    at java.desktop/java.awt.image.Raster.createPackedRaster(Raster.java:914)
    at java.desktop/java.awt.image.Raster.createPackedRaster(Raster.java:546)
    at java.desktop/java.awt.image.DirectColorModel.createCompatibleWritableRaster(DirectColorModel.java:1032)
    at java.desktop/java.awt.image.BufferedImage.<init>(BufferedImage.java:324)
    at com.yonatankarp.xkcd.clickanddrag.Combiner.combineAll(Combiner.kt:40)
    at com.yonatankarp.xkcd.clickanddrag.MainKt.main(Main.kt:61)

Looking at the codebase of the library, the issue was clear:

if (size > Integer.MAX_VALUE) {
    throw new IllegalArgumentException("Dimensions (width=" + w + " height=" + h + ") are too large");
}

Indeed, the image had too many pixels in it. The first thing I did to fix the issue was to scale down the end image. Instead of using the original tile size (2048x2048), I scaled it down by 4 to the size of 512x512, which resulted in this file:

Now, I could immediately notice a bug: tiles that are below the center of the image should have been colored black, while tiles that are above the center should have been white, resulting in a weird image. After a quick fix, I finally received this image (not before inverting the colors, of course):

Hurray! However, zooming into the image resulted in pixelated and unreadable results. The mission was not achieved just yet.

Slicing the Cake

I searched on Google for a while, but every result I saw indicated some limitation on the image size, caused by the fact that the image processing was done in memory for all libraries. I consulted Reddit and LinkedIn but without any good outcome.

To solve this problem, I added yet another fix to the program: each row in the image was generated as a separate PNG file, and I hoped to use graphic editing software like Gimp to combine them back into one huge image.

With tons of patience and over 50GB of memory used to combine all the images, I finally managed to build the image! Great success!

I finished my day happy and went to bed. 💪

The Bug

The next morning, I decided to look at the output image and finally read the entire image. It didn't take more than 30 seconds for me to find a bug in the results. Between tiles 1n1e and 1n1w, there was a one-tile space that should not have been there:

The fix was very simple, but I didn't want to spend yet another 30 minutes combining 14 slices of images. To minimize the pain, all I had to do was calculate the maximum number of rows that I could fit within the size of MAX_INT pixels. A quick calculation showed that the number of rows I can fit is 3 (167,936 x 2048 * 3), which resulted in only 5 pieces to fit together.

I re-ran the code, combined all the pieces, and happily exported the file as a PNG with a size of ~30MB.

Step 3: Share the Results

I wanted to share the results with people, but the image was just way too big to upload. To resolve this issue, I decided to create a new repository to hold the final result image, the Gimp intermediate image, and all the tiles, including all the source code that was used during this hack project, on GitHub.

If you want to check it out, visit GitHub Repository. If you like it, give it a ⭐️!

If you enjoyed this article and want to read more, please consider following my blogs at yonatankarp.com and kotlinbackend.com. I look forward to sharing more insights with you!

Conclusion

Challenge yourself. Think outside the box. The naive solution can sometimes be the best. But most importantly, have fun!

Taking on the challenge of downloading and rebuilding the entire "Click and Drag" comic from XKCD was a rewarding and educational experience. Through perseverance and creative problem-solving, I navigated various technical obstacles and learned valuable lessons about large-scale image processing & the power of parallel computing.

This project underlines the importance of thinking outside the box and not shying away from seemingly scary tasks. The satisfaction of overcoming each obstacle and seeing the final, complete image was immensely gratifying. It serves as a reminder that even the most complex problems can be solved with a combination of curiosity, resourcefulness, and persistence.

I encourage you to take on your own challenges, whether they are in software programming or another field. Embrace the learning process, seek help when needed, and most importantly, enjoy the journey. After all, the best solutions often come from simply having fun and pushing the boundaries of what you think is possible.

Thank you for following along with my adventure. If you're interested in the details or want to try it yourself, feel free to visit the GitHub repository and explore further. Happy coding!

Design Patterns - Prototype

Yonatan Karp-Rudin — Mon, 20 Nov 2023 09:49:55 +0000

TL;DR: The Prototype design pattern creates new objects by cloning a prototypical instance, allowing you to modify the copy without creating an object from scratch. This pattern is useful when object creation is expensive or when classes to instantiate are specified at runtime.

Intent

Specify the kinds of objects to create using a prototypical instance, and create new objects by copying this prototype.

Explanation

First, it should be noted that the Prototype pattern is not used to gain performance benefits. It's only used for creating new objects from prototype instances.

Real-world example

Remember Dolly? The sheep that was cloned! Let's not get into the details but the key point here is that it is all about cloning.

In plain words

Create an object based on an existing object through cloning.

Wikipedia says

The prototype pattern is a creational design pattern in software development. It is used when the type of objects to create is determined by a prototypical instance, which is cloned to produce new objects.

In short, it allows you to create a copy of an existing object and modify it to your needs, instead of going through the trouble of creating an object from scratch and setting it up.

Programmatic Example

In Kotlin, the prototype pattern is recommended to be implemented as follows. First, create an interface with a method for cloning objects. In this example, Prototype interface accomplishes this with its clone method. We will then use the copy() function of data class to create a clone of our class where we can override specific fields of the class if needed.

abstract class Prototype<T> {
  abstract fun clone(): T
}

Our example contains a hierarchy of different creatures. For example, let's look at Beast and OrcBeast classes.

abstract class Beast : Prototype<Beast>()

data class OrcBeast(private val weapon: String) : Beast() {

  override fun clone() = copy()

  override fun toString() = "Orcish wolf attacks with $weapon"
}

We don't want to go into too many details, but the full example contains also base classes Mage and Warlord and there are specialized implementations for those for elves in addition to orcs.

To take full advantage of the prototype pattern, we create HeroFactory class to produce different kinds of creatures from prototypes.

class HeroFactory(
  private val mage: Mage,
  private val warlord: Warlord,
  private val beast: Beast,
) {
  fun createMage() = mage.clone()

  fun createWarlord() = warlord.clone()

  fun createBeast() = beast.clone()
}

Now, we can show the full prototype pattern in action producing new creatures by cloning existing instances.

var factory = HeroFactory(
  ElfMage("cooking"),
  ElfWarlord("cleaning"),
  ElfBeast("protecting")
)

var mage = factory.createMage()
var warlord = factory.createWarlord()
var beast = factory.createBeast()

logger.info(mage.toString())
logger.info(warlord.toString())
logger.info(beast.toString())

factory = HeroFactory(
  OrcMage("axe"),
  OrcWarlord("sword"),
  OrcBeast("laser")
)

mage = factory.createMage()
warlord = factory.createWarlord()
beast = factory.createBeast()

logger.info(mage.toString())
logger.info(warlord.toString())
logger.info(beast.toString())

Here's the console output from running the example.

Elven mage helps in cooking
Elven warlord helps in cleaning
Elven eagle helps in protecting
Orcish mage attacks with axe
Orcish warlord attacks with sword
Orcish wolf attacks with laser

Class diagram

Applicability

Use the Prototype pattern when a system should be independent of how its products are created, composed, represented and

When the classes to instantiate are specified at run-time, for example, by dynamic loading.
To avoid building a class hierarchy of factories that parallels the class hierarchy of products.
When instances of a class can have one of only a few different combinations of state. It may be more convenient to install a corresponding number of prototypes and clone them rather than instantiating the class manually, each time with the appropriate state.
When object creation is expensive compared to cloning.

I hope you enjoyed this journey and learned something new. If you want to stay updated with my latest thoughts and ideas, feel free to register for my newsletter. You can also find me on LinkedIn or Twitter. Let's stay connected and keep the conversation going!

Code Examples

All code examples and tests can be found in the Kotlin Design Patterns repository

Credits

Design Patterns - Builder

Yonatan Karp-Rudin — Fri, 03 Nov 2023 08:01:30 +0000

TL;DR: Separate the construction of a complex object from its representation so that the same construction process can create different representations.

Intent

Separate the construction of a complex object from its representation so that the same construction process can create different representations.

Explanation

Real-world example

Imagine a character generator for a role-playing game. The easiest option is to let the computer create the character for you. If you want to manually select the character details like profession, gender, hair color, etc. the character generation becomes a step-by-step process that completes when all the selections are ready.

In plain words

Allows you to create different flavors of an object while avoiding constructor pollution. Useful when there are several flavours of an object. Or when there are a lot of steps involved in the creation of an object.

Wikipedia says

The builder pattern is an object creation software design pattern with the intentions of finding a solution to the telescoping constructor anti-pattern.

Having said that let me add a bit about what telescoping constructor anti-pattern is. At one point or the other, we have all seen a constructor like below:

data class Hero(
  val profession: Profession,
  val name: String,
  val hairType: HairType?,
  val hairColor: HairColor?,
  val armor: Armor?,
  val weapon: Weapon?
)

As you can see the number of constructor parameters can quickly get out of hand, and it may become difficult to understand the arrangement of parameters. Plus this parameter list could keep on growing if you would want to add more options in the future. This is called a telescoping constructor anti-pattern.

Programmatic Example

The sane alternative is to use the Builder pattern. First of all, we have our hero that we want to create:

data class Hero(
  val profession: Profession,
  val name: String,
  val hairType: HairType?,
  val hairColor: HairColor?,
  val armor: Armor?,
  val weapon: Weapon?
) {
  private constructor(builder: Builder) : this(
    builder.profession,
    builder.name,
    builder.hairType,
    builder.hairColor,
    builder.armor,
    builder.weapon
  )
}

Then we have the builder:

class Builder(profession: Profession?, name: String?) {
  val profession: Profession
  val name: String
  var hairType: HairType? = null
  var hairColor: HairColor? = null
  var armor: Armor? = null
  var weapon: Weapon? = null

  init {
    require(!(profession == null || name == null)) { "profession and name can not be null" }
    this.profession = profession
    this.name = name
  }

  fun withHairType(hairType: HairType?): Builder = apply {
    this.hairType = hairType
  }

  fun withHairColor(hairColor: HairColor?): Builder = apply {
    this.hairColor = hairColor
  }

  fun withArmor(armor: Armor?): Builder = apply {
    this.armor = armor
  }

  fun withWeapon(weapon: Weapon?): Builder = apply {
    this.weapon = weapon
  }

  fun build() = Hero(this)
}

Then it can be used as:

val mage = Hero.Builder(Profession.MAGE, "Riobard")
  .withHairColor(HairColor.BLACK)
  .withWeapon(Weapon.DAGGER)
  .build()

However, Kotlin provides an alternative to the Builder pattern with named arguments and default parameter values:

data class Hero(
    val profession: Profession,
    val name: String,
    val hairType: HairType? = null,
    val hairColor: HairColor? = null,
    val armor: Armor? = null,
    val weapon: Weapon? = null
)

Then it can be used as:

val mage = Hero(
    profession = Profession.MAGE,
    name = "Riobard",
    hairColor = HairColor.BLACK,
    weapon = Weapon.DAGGER
)

Not only that the code simpler, but we are also enforcing the required parameters at compile time.

Class diagram

Applicability

Use the Builder pattern when

The algorithm for creating a complex object should be independent of the parts that make up the object and how they're assembled
The construction process must allow different representations for the object that's constructed

Tutorials

Code Examples

All code examples and tests can be found in the Kotlin Design Patterns repository

Credits

Design Patterns - Abstract Factory

Yonatan Karp-Rudin — Thu, 26 Oct 2023 07:29:08 +0000

TL;DR: Abstract Factory pattern provides an interface for creating families of related or dependent objects without specifying their concrete classes. This pattern is useful when the system needs to be independent of how its products are created, composed, and represented, and when there are multiple families of products that need to be used together.

Also known as

Intent

Provide an interface for creating families of related or dependent objects without specifying their concrete classes.

Explanation

Real-world example

To create a kingdom we need objects with a common theme. The elven kingdom needs an elven king, elven castle, and elven army whereas the orcish kingdom needs an orcish king, orcish castle, and orcish army. There is a dependency between the objects in the kingdom.

In plain words

A factory of factories; is a factory that groups the individual but related/dependent factories together without specifying their concrete classes.

Wikipedia says

The abstract factory pattern provides a way to encapsulate a group of individual factories that have a common theme without specifying their concrete classes

Programmatic Example

Translating the kingdom example above. First of all, we have some interfaces and implementations for the objects in the kingdom.

internal interface Castle {
  val description: String
}

internal interface King {
  val description: String
}

internal interface Army {
  val description: String
}

// Elven implementations ->

internal class ElfCastle : Castle {
  override val description = "This is the elven castle!"
}

internal class ElfKing : King {
  override val description = "This is the elven king!"
}

internal class ElfArmy : Army {
  override val description = "This is the elven army!"
}

// Orcish implementations similarly -> ...

Then we have the abstraction and implementations for the kingdom factory.

internal interface KingdomFactory {
  fun createCastle(): Castle
  fun createKing(): King
  fun createArmy(): Army
}

internal class ElfKingdomFactory : KingdomFactory {
  override fun createCastle() = ElfCastle()
  override fun createKing() = ElfKing()
  override fun createArmy() = ElfArmy()
}

internal class OrcKingdomFactory : KingdomFactory {
  override fun createCastle() = OrcCastle()
  override fun createKing() = OrcKing()
  override fun createArmy() = OrcArmy()
}

Now we have the abstract factory that lets us make a family of related objects i.e. elven kingdom factory creates an elven castle, king and army, etc.

val factory = ElfKingdomFactory()
val castle = factory.createCastle()
val king = factory.createKing()
val army = factory.createArmy()

castle.description
king.description
army.description

Program output:

This is the elven castle!
This is the elven king!
This is the elven Army!

Now, we can design a factory for our different kingdom factories. In this example, we created FactoryMaker, responsible for returning an instance of either ElfKingdomFactory or OrcKingdomFactory.

The client can use FactoryMaker to create the desired concrete factory which, in turn, will produce different concrete objects (derived from Army, King, Castle).

In this example, we also used an enum to parameterize which type of kingdom factory the client will ask for.

internal data class Kingdom(
  val king: King,
  val castle: Castle,
  val army: Army,
) {
  object FactoryMaker {
    enum class KingdomType {
      ELF,
      ORC
    }

    fun makeFactory(type: KingdomType): KingdomFactory {
      return when (type) {
        KingdomType.ELF -> ElfKingdomFactory()
        KingdomType.ORC -> OrcKingdomFactory()
      }
    }
  }
}

Now we can use the abstract factory to create the kingdoms

logger.info("elf kingdom")
val elfKingdom = createKingdom(KingdomType.ELF)
logger.info(elfKingdom.army.description)
logger.info(elfKingdom.castle.description)
logger.info(elfKingdom.king.description)

logger.info("orc kingdom")
val orcKingdom = createKingdom(KingdomType.ORC)
logger.info(orcKingdom.army.description)
logger.info(orcKingdom.castle.description)
logger.info(orcKingdom.king.description)

Program output:

elf kingdom
This is the elven castle!
This is the elven king!
This is the elven Army!
orc kingdom
This is the orc castle!
This is the orc king!
This is the orc Army!

Class diagram

Applicability

Use the Abstract Factory pattern when

The system should be independent of how its products are created, composed, and represented
The system should be configured with one of the multiple families of products
The family of related product objects is designed to be used together, and you need to enforce this constraint
You want to provide a class library of products, and you want to reveal just their interfaces, not their implementations
The lifetime of the dependency is conceptually shorter than the lifetime of the consumer.
You need a run-time value to construct a particular dependency
You want to decide which product to call from a family at runtime.
You need to supply one or more parameters only known at run-time before you can resolve a dependency.
When you need consistency among products
You don't want to change existing code when adding new products or families of products to the program.

Example use cases

Selecting to call to the appropriate implementation of FileSystemAcmeService or DatabaseAcmeService or NetworkAcmeService at runtime.
Unit test case writing becomes much easier
UI tools for different OS

Consequences

Dependency injection in Java hides the service class dependencies that can lead to runtime errors that would have been caught at compile time.
While the pattern is great when creating predefined objects, adding new ones might be challenging.
The code becomes more complicated than it should be since many new interfaces and classes are introduced along with the pattern.

Tutorials

Abstract Factory Pattern Tutorial

Code Examples

All code examples and tests can be found in the Kotlin Design Patterns repository

Related patterns

Factory Method
Factory Kit

Credits

Design Patterns - Factory

Yonatan Karp-Rudin — Tue, 24 Oct 2023 15:53:19 +0000

TL;DR: Factory pattern provides a static method encapsulated in a class to hide implementation logic, allowing client code to focus on usage rather than object initialization. It simplifies object creation, promotes loose coupling, and supports easy switching between object types without modifying existing code.

Also known as

Factory
Simple Factory

Intent

Providing a static method encapsulated in a class called the factory, to hide the implementation logic and make client code focus on usage rather than initializing new objects.

Explanation

Real-world example

Imagine an alchemist who is about to manufacture coins. The alchemist must be able to create both gold and copper coins and switching between them must be possible without modifying the existing source code. The factory pattern makes it possible by providing a static construction method which can be called with relevant parameters.

Wikipedia says

A factory is an object for creating other objects formally a factory is a function or method that returns objects of a varying prototype or class.

Programmatic Example

We have an interface Coin and two implementations GoldCoin and CopperCoin.

internal interface Coin {
    val description: String
}

internal class GoldCoin : Coin {
    override val description = "This is a gold coin."
}

internal class CopperCoin : Coin {
    override val description = "This is a copper coin."
}

The enumeration above represents the types of coins that we support (GoldCoin and CopperCoin).

internal enum class CoinType(val constructor: () -> Coin) {
    COPPER({ CopperCoin() }),
    GOLD({ GoldCoin() }),
}

Then we have the static method getCoin to create coin objects encapsulated in the factory class CoinFactory.

internal object CoinFactory {
    fun getCoin(type: CoinType): Coin = type.constructor()
}

Now on the client code, we can create different types of coins using the factory class.

logger.info("The alchemist begins his work.")
val copper = CoinFactory.getCoin(CoinType.COPPER)
val gold = CoinFactory.getCoin(CoinType.GOLD)
logger.info(copper.description)
logger.info(gold.description)

Program output:

The alchemist begins his work.
This is a copper coin.
This is a gold coin.

Class Diagram

Applicability

Use the factory pattern when you only care about the creation of an object, not how to create and manage it.

Pros

Allows keeping all objects created in one place.
Allows to write loosely coupled code. Some of its main advantages include better testability, easy-to-understand code, swappable components, scalability, and isolated features.

Cons

The code becomes more complicated than it should be.

Code Examples

All code examples and tests can be found in the Kotlin Design Patterns repository

Related patterns

Credits

Java Design Patterns

Our Errors, Your Gain: A System Design Cautionary Tale

Yonatan Karp-Rudin — Mon, 23 Oct 2023 06:28:23 +0000

TL;DR: This article shares my experience with a temporary solution that turned into a long-term issue. I discuss the evolution of a service and how it improved its performance by over 450% in a few hours. The story highlights the importance of addressing temporary solutions and constantly evaluating system design to optimize performance.

Introduction

About four years ago, around the time I joined my previous employer, my manager at the time introduced me to one of the services my team owned. He opened the explanation with the sentence, "As the saying goes, nothing is more permanent than a temporary solution". More than five years have passed since this service was initially introduced, and it's still running in production.

In this article, I share a specific use case of the service and how we improved its performance by over 450% in just a few hours.

The Original System Design

Let's start by briefly looking at the system's original design. The system included a database that serves as the source of the data, a poller service that checks for changes in the source database every five seconds, and in case a change is found, it aggregates the data and stores it in a different database. Lastly, a Reader service is responsible for reading the aggregated data from the database and serving it to our clients.

As you might already imagine, integration via a database is almost always a bad idea. It can lead to data inconsistency, lack of real-time updates, difficulty in making changes in the database schema, and more. But as we just said, temporary solution - right? Our system specifically suffered mainly from data inconsistency. The main root cause for the issues was been that the entities in the database might not yet be finalized by the time they were polled. In such a case, the poller would ignore the data and would never return to fix it later.

Addressing Data Inconsistency

Eventually, the data-owning team began publishing events indicating that the data was finalized and ready for use. We amended our poller service, so it would start to listen to those events and store the IDs of the entities that are ready to be picked in a table that it would later poll. This ensures no data is missed. With the new change, our system now looks as follows:

You might ask: " Why didn't you change the implementation to rely completely on the events?". Well... First, the events lacked the required information. The only reason the data exists in the database, to begin with, is because of domain mixing. The mix in domains has been so deep that multiple teams worked very hard for a very long time to split those domains.

The Human Error

At this point, it's worth stopping and discussing the human aspect behind the existence of this solution. The poller service was introduced as a technical solution for a technical problem. We needed access to data that we didn't have, and with the help of this service, we had that access (even if it caused a violation of multiple domain boundaries).

However, as we made the poller stable it fell lower and lower in the priority list of the company to replace this design with a proper solution. Not only that, but the maintenance of the service moved between teams in the company like a hot potato. In my time in the company, I actually had the pleasure of working on this system in 3 different contexts and teams.

The lack of real ownership of the system, higher company priorities, and management's general agreement that the solution wasn't great but worked, led to the repeated de-prioritization of refactoring this part of the system.

Introducing New Features and Challenges

The system functioned and was left unchanged for some time. Until one day, we had to introduce another new feature. The feature required to generate information for our clients about the previous day. The data should be available every day at 5 a.m. (client's local time).

To solve this problem, we first introduced a new service responsible for managing day opening/closing. Next, the poller service made a REST call to the new service for each entity in the database and assigned the ID of the current day. Lastly, the Read service aggregated the data from the database and used it to generate the required information.

Our system now looked like this:

As you can imagine, using REST calls between the poller and the new service created a strong coupling. Every time we had to maintain the database of the day management service, our poller service received an internal error response from the REST call and did not assign an ID to the entity. We could resolve this situation by using asynchronous communication between the services. In such a case, the Day management service would notify all consumers whenever a new day is opened/closed and let them do as they wish with the information. Such a solution would allow scalability of the system, resiliency, and avoid coupling between the services (or as my favorite phrase says, building a distributed monolith).

Yet another issue, when the clients activated the feature, we did not assign the ID of the day to already existing entities. The impact of that was that the data from the first day was incomplete at best, or completely wrong at worst. This caused many calls to our support agents and wasted quite a significant amount of the team's time debugging the issue.

Performance Issues and Materialized Views

We rolled out the feature to 1000 clients. After about two weeks, we discovered that we had severe performance issues with our new feature. The issue was related to the SQL query we used to build the required information, which included multiple JOINs.

Since the data doesn't have to be updated during the day but only in the morning, we decided to come up with a new approach. We introduced a materialized view that was holding all the required information for the feature pre-calculated and refreshed the view every morning.

If you're unfamiliar with the materialized view in SQL, it is a database object that contains the results of a query, stored in a table-like structure. It is used to cache the results of complex queries for faster access and improved performance. Materialized views are periodically updated by refreshing the stored data, either manually or automatically, to keep the view in sync with the underlying data.

That means that from that point on, our poller started to refresh the materialized view every day. That was supposed to be a temporary solution until we find a more suitable database for our needs (where did I hear this before?)

At this point, I would like to stop and point out a few flaws in the above design with the requirements we had:

Since we rolled out in multiple time zones, and we had only one view, we could not serve the information for our client at 5 a.m. anymore; the refresh had to happen when the client with the latest timezone hit this time.
Materialized view does not support partial refresh, which means that every day we had to refresh all the data that was included in the feature, for all of our clients.

The solution kind of worked for a while. However, as the number of clients using the feature grew, and the amount of data we collected for each of them also grew, the refresh of the materialized view started to become heavy. It became so heavy that at some point it took almost 5 hours to refresh the view.

Not only did our API not match its target, but it also consumed a lot of resources from our database, which caused a lot of headaches for our database admins.

The Efficient Solution and its Benefits

Some time passed by, and a colleague of mine reached out to me and told me, "I think that I know how to fix the issue!". Instead of refreshing all the data every day, let's create a new table that would hold the pre-calculated data, we will run a job that would be executed periodically, and calculate only the delta since the previous calculation. The idea was genius by its simplicity!

We decided to make a POC for the above suggestion and check if it can be implemented. In six hours, we deployed a working POC in our staging environment, tested it, and with an additional day of work we went all the way to production.

After some testing, we set our refresh period to 1 minute and noticed that the query takes now around 1.4 seconds to be executed. With that, our saga came to an end.

As you can see in the graph below, our CPU usage for the database dropped dramatically after the changes!

With those changes, we regained:

Minimal resource consumption by the database.
Near-real-time information availability for clients instead of a 5-hour delay
Simpler, easier-to-understand codebase.

To Be Continue?

While the story of the poller does not end here, my story in the company does. After almost 4.5 years I contacted my first manager in the company and told him that I had failed to sunset the poller. However, there is good news on the horizon. A new dedicated team starting to work now on re-writing the entire system from the ground, basing it on the right data sources, and hopefully soon enough the poller story will come to an end! 🤞

Conclusion

This cautionary tale demonstrates the importance of addressing temporary solutions and constantly evaluating system design to optimize performance. By implementing a more efficient approach to handling growing data and client needs, the team significantly improved their system's performance, resource consumption, and overall simplicity.

Remember, temporary solutions are never temporary!

Every tech debt decision you make will come back and hunt you if you are not going to come back and fix it later.

Observability in Action Part 3: Enhancing Your Codebase with OpenTelemetry

Yonatan Karp-Rudin — Tue, 17 Oct 2023 07:43:34 +0000

This blog post was moved to https://kotlinbackend.com/observability-in-action-how-to-use-opentelemetry/

Observability in Action Part 2: Enhancing Your Codebase with OpenTelemetry

Yonatan Karp-Rudin — Thu, 05 Oct 2023 12:23:02 +0000

This blog post was moved to https://kotlinbackend.com/observability-in-action-how-to-use-opentelemetry/

Observability in Action Part 1: Enhancing Your Codebase with OpenTelemetry

Yonatan Karp-Rudin — Thu, 05 Oct 2023 10:25:51 +0000

This blog post was moved to https://kotlinbackend.com/observability-in-action-how-to-use-opentelemetry/

Design Patterns - Factory Method

Yonatan Karp-Rudin — Fri, 29 Sep 2023 08:30:09 +0000

TL;DR : Factory Method is a design pattern that defines an interface for creating objects, allowing subclasses to decide which class to instantiate, thus delegating instantiation logic to child classes.

Also known as

Virtual Constructor

Intent

Define an interface for creating an object, but let subclasses decide which class to instantiate. Factory Method lets a class defer instantiation to subclasses.

Explanation

Real-world example

Blacksmith manufactures weapons. Elves require Elvish weapons and orcs require Orcish weapons. Depending on the customer at hand the right type of blacksmith is summoned.

In plain words

It provides a way to delegate the instantiation logic to child classes.

Wikipedia says

In class-based programming, the factory method pattern is a creational pattern that uses factory methods to deal with the problem of creating objects without having to specify the exact class of the object that will be created. This is done by creating objects by calling a factory method either specified in an interface and implemented by child classes, or implemented in a base class and optionally overridden by derived classesrather than by calling a constructor.

Programmatic Example

Take our blacksmith example above. First of all, we have a Blacksmith interface and some implementations for it:

internal interface Blacksmith {
  fun manufactureWeapon(weaponType: WeaponType): Weapon
}

internal class ElfBlacksmith : Blacksmith {
  override fun manufactureWeapon(weaponType: WeaponType): Weapon =
    ELF_ARSENAL.getOrElse(weaponType) {
      throw IllegalArgumentException("Weapon type $weaponType is not supported by elf blacksmith.")
    }
}

internal class OrcBlacksmith : Blacksmith {
  override fun manufactureWeapon(weaponType: WeaponType): Weapon =
    ORC_ARSENAL.getOrElse(weaponType) {
      throw IllegalArgumentException("Weapon type $weaponType is not supported by the orc blacksmith.")
    }
}

When the customers come, the correct type of blacksmith is summoned and the requested weapons are manufactured:

// Orc
var blacksmith: Blacksmith = OrcBlacksmith()

var weapon: Weapon = blacksmith.manufactureWeapon(WeaponType.SPEAR)
logger.info("$blacksmith manufactured ${weapon.weaponType.title}")

weapon = blacksmith.manufactureWeapon(WeaponType.AXE)
logger.info("$blacksmith manufactured ${weapon.weaponType.title}")

// Elf
blacksmith = ElfBlacksmith()

weapon = blacksmith.manufactureWeapon(WeaponType.SPEAR)
logger.info("$blacksmith manufactured ${weapon.weaponType.title}")

weapon = blacksmith.manufactureWeapon(WeaponType.AXE)
logger.info("$blacksmith manufactured ${weapon.weaponType.title}")

Program output:

The orc blacksmith manufactured spear
The orc blacksmith manufactured axe
The elf blacksmith manufactured spear
The elf blacksmith manufactured axe

Class diagram

Applicability

Use the Factory Method pattern when:

The class cannot anticipate the class of objects it must create.
The class wants its subclasses to specify the objects it creates.
Classes delegate responsibility to one of several helper subclasses, and you want to localize the knowledge of which helper subclass is the delegate.

Code Examples

All code examples and tests can be found in the Kotlin Design Patterns repository

Credits

Design Patterns - Singleton

Yonatan Karp-Rudin — Tue, 26 Sep 2023 07:58:44 +0000

TL;DR: Ensure a class has only one instance and provide a global access point to it with the Singleton pattern, which is useful for coordinating actions across a system.

Intent

Ensure a class only has one instance, and provide a global point of access to it.

Explanation

Real-world example

There can only be one ivory tower where the wizards study their magic. The same enchanted ivory tower is always used by the wizards. The ivory tower here is a singleton.

In plain words

Ensures that only one object of a particular class is ever created.

Wikipedia says

In software engineering, the singleton pattern is a software design pattern that restricts the instantiation of a class to one object. This is useful when exactly one object is needed to coordinate actions across the system.

Programmatic Example

Kotlin makes it externally easy to create singletons by introducing the language keyword object. By using this keyword, we will get an implementation of the Singleton pattern from the compiler that contains all of our requirements.

data object IvoryTower

One of the key differences between a class and a object is that the latter does not allow constructor arguments. If your implementation needs initialization for your Singleton (e.g. to load data) you can use an init block.

data object IvoryTower {
  init {
    logger.info("Initializing Ivory Tower...")
  }
}

Note that if the Singleton object is never invoked, it won't run its initialization block at all. This is called lazy initialization.

Then in order to use:

val ivoryTower1 = IvoryTower
val ivoryTower2 = IvoryTower
logger.info("ivoryTower1={}", ivoryTower1)
logger.info("ivoryTower2={}", ivoryTower2)

The output on the console:

09:21:31.107 [main] INFO Singlton -- Initializing Ivory Tower...
09:21:31.109 [main] INFO Singlton -- ivoryTower1=com.yonatankarp.singleton.IvoryTower@1207938418
09:21:31.111 [main] INFO Singlton -- ivoryTower2=com.yonatankarp.singleton.IvoryTower@1207938418

Class diagram

Applicability

Use the Singleton pattern when

There must be exactly one instance of a class, and it must be accessible to clients from a well-known access point
When the sole instance should be extensible by subclassing, and clients should be able to use an extended instance without modifying their code

Some typical use cases for the Singleton

The logging class
Managing a connection to a database
File manager

Consequences

Violates the Single Responsibility Principle (SRP) by controlling their creation and lifecycle.
Encourages using a globally shared instance which prevents an object and resources used by this object from being deallocated.
Creates tightly coupled code. The clients of the Singleton become difficult to test.
It makes it almost impossible to subclass a Singleton.

Code Examples

All code examples and tests can be found in the Kotlin Design Patterns repository

Credits

Kotlin Code Smell 36 - Object Coupling

Yonatan Karp-Rudin — Mon, 04 Sep 2023 13:41:17 +0000

TL;DR: Respect object boundaries: avoid coupling to data and prioritize interfaces and behavior.

When you view your objects merely as data holders, you risk violating their encapsulation.

Problem

Information Hiding Violation
Encapsulation Violation
Coupling

Solution

Always couple to interfaces and behavior, not data.

Sample Code

Wrong

data class Point(var x: Double, var y: Double)

class DistanceCalculator {
    fun distanceBetween(origin: Point, destination: Point): Double {
        return sqrt(
            (destination.x - origin.x).pow(2) +
                    (destination.y - origin.y).pow(2)
        )
    }
}

Right

data class Point(
    private val radius: Double,
    private val theta: Double
) {
    val x: Double get() = radius * cos(theta)
    val y: Double get() = radius * sin(theta)
}

class DistanceCalculator {
    fun distanceBetween(origin: Point, destination: Point): Double {
        return sqrt(
            (destination.x - origin.x).pow(2) +
                    (destination.y - origin.y).pow(2)
        )
    }
}

Conclusion

If your classes are polluted with setters, getters and public methods you will certainly have ways to couple to their accidental implementation.

Credits

Code Smell 55 - Object Orgy by Maximiliano Contieri