DEV Community: Stanislav Kozlovski

One-shotting a Diskless Kafka in Python

Stanislav Kozlovski — Wed, 29 Apr 2026 16:19:55 +0000

Talk is cheap, show me the code - Linus Torvalds

In 2026, code is cheap too - design is what matters.

StreamNative recently open-sourced a formally-verified protocol for implementing a leaderless log. Their announcement blog sent a message similar to the opening quote (h/t @sijieg) - that in the age of AI coding harnesses, what matters more is the design/protocol of a system rather than its particular implementation.

I wanted to put that to the test, so I took their protocol, took a linearizable metadata store (which the protocol requires) and got cracking:

git clone git@github.com:oxia-db/oxia-client-python.git
git clone https://github.com/oxia-db/oxia oxia-server
git clone https://github.com/lakestream-io/leaderless-log-protocol

/code/diskless-python-kafka (main) $ ls  

leaderless-log-protocol oxia-client-python      oxia-server

/code/diskless-python-kafka (main) $ codex # the magic begins

The One Shot

My prompt was simple:

Using the Oxia python client (in this folder), and a running Oxia server (again in this folder), please implement a leaderless log protocol python agent for writing data. (only writing. no compaction yet). Use the leaderless-log-protocol spec in the folder here. In particular, the 1-leaderless-log-protocol.md should tell you all you need to know. The 0-coordination-delegated-pattern.md can share info on Oxia/the coordination store. Implement everything in one single file.

This was enough to implement a working leaderless log distributed system (with just its write functionality). Two prompts later, I implemented the read path and the compaction path.

But it wasn't optimal - the published leaderless log specification only details how to ensure correctness for a single partition. It doesn't detail how to batch many topic partitions into a single mixed WAL S3 object for cost efficiency (what WarpStream and every other Diskless Kafka do).

Preserving correctness while batching and following the protocol wasn't hard though.

The core thing was more or less implemented in one 5 hour usage limit of Codex ($20 plan) with gpt-5.4 xhigh.

I then started spending tokens on "productionizing" it. A load-testing harness, an observability stack and subsequent performance optimizations. This took me around 2-3 days of hacking, and a lot more tokens from parallel Codex sessions.

Here's how my terminal looked:

It's important to work on unrelated stuff in parallel so as to limit the eventual merge conflicts.

How Diskless Works

The leaderless log protocol will be familiar to anybody who's read about Diskless Kafka before. The key differentiator from regular Kafka is that:

no leaders exist: every broker accepts writes for every partition
mixed-partition segment files: each broker buffers data and then unloads it all in one big fat blob on S3 that contains multi-partition data
compaction is critical: eventually, a compaction process splits that big blob into per-partition blobs optimized for sequential reads

Key benefits of this architecture are:

cost - it can be 90% cheaper in high throughput situations because no inter-AZ network fees are incurred.
operational simplicity - because brokers are stateless (all data is in S3), they're easier to manage and scale.

Here's how my write path looked like:

client
  |
  | POST /produce
  v
+---------------------------+
| HTTP broker               |
| topic_partitions[]        |
+---------------------------+
  |
  | aggregate + batch
  | flush at 8 MiB or 500 ms
  v
+---------------------------+
| LeaderlessLogWriter       |
|                           |
+---------------------------+
  |
  | 1) write one shared WAL blob
  +-------------------------------> S3
  |                                 llog/wal-shared/{uuid}
  |
  | 2) for each partition:
  |    reserve offsets + persist sparse-index
  v
+----------------------------------------------+
| Oxia                                         |
| orders[0] offsets 1..2 -> shared WAL object  |
| orders[1] offsets 1..1 -> same WAL object    |
+----------------------------------------------+
  |
  | 3) respond with per-partition offsets/results
  v
client

An HTTP Python broker accepts incoming POST /produce requests whose payload is a simple JSON map of partition name to a list of records for that partition.
The broker buffers requests until it either reaches 8 MiB of pending data, or the wall clock time from the first request has surpassed 500ms. When either triggers, it begins to commit the data.
First, it commits the mixed topic-partition data to S3 in one big 8 MiB blob. The data is durably persisted in S3 at this point - but it doesn't have offsets applied yet.
Then, for each partition, it goes to Oxia (the distributed key-value metadata store) and persists the offsets there. This now "seals" our S3 file as a legit record of Kafka record data. Our metadata points to it.
The broker responds to the client's produce request.

Step 4) is more complex than it looks, and is critical in ensuring safety of the distributed protocol. Let me expand on it:

The Oxia Offset Commit

💡 Oxia is the distributed strongly-consistent key-value store we chose as our metadata store (article here)

The offset assignment in Oxia consists of multiple steps. A single meta/control key (per partition) acts as the centralized sequencer -- it says what the latest offset is.

meta/control = {
    "log_state": "OPEN",
    "sequence_counter": 48,
    "pending": null
  }

When a writer goes to commit a new bunch of offsets for a partition there (after the mixed multi-partition S3 blob has been persisted), it increments the offset counter AND populates the pending field to reference the latest mixed S3 blob that holds these offsets:

{
    "log_state": "OPEN",
    "sequence_counter": 73, // + 25
    "pending": {
      "start_offset": 48,
      "end_offset": 72,
      "msg_count": 25,
      "data_key": "s3://bucket/llog/wal-shared/abc123",
      ...
    }
  }

This is done with a Compare-and-Swap (CAS) write to Oxia.

💡 Oxia assigns versions for EVERY write operation, which lets you achieve strongly-consistent conditional updates via compare and swap operations.

The next step for that writer is to move the pending data to the index/ key hierarchy in Oxia (for that partition). That is where the definitive [record-offset -> S3] data location mapping is stored. An entry in that key space looks like this:

// key: llog/orders/partitions/0/index/00000000000000000072
// hint: 00000000000000000072 is the end offset
  {
    "type": "WAL",
    "msg_count": 25,
    "data_key": "s3://bucket/llog/wal-shared/blob-c",
    "encoding": "bytes-batch-v1",
    "byte_offset": 2048,
    "byte_length": 12000,
    "created_at_ms": 1760000002000
  }

where:

orders/partitions/0 - denotes partition-0 of the orders topic
00000000000000000072 - a part of the key name, is the END offset of the records in that index entry
data_key - denotes the full S3 path for that blob file.
byte_offset/byte_length - denotes the exact location inside the S3 blob file where the records are consecutively laid out. Since a read may only want a single record from that blob file, it would be inefficient to have it read the whole blob to get the record. Instead, this mapping allows for byte-ranged GETs to S3 that download those particular records and not a byte more.

After it's written there, the pending field of meta/control gets deleted.

Offset Summary

So again, the path is:

write the index entry into meta/control.pending
write the index entry into index/{END_OFFSET}
delete the pending field of meta/control.

These 3 steps are not atomic. The writer process can fail in the middle of any step.

The key safety property which guarantees data stays consistent is the following - a writer NEVER overrides meta/control.pending.

It only writes into it if it's empty (which we can guarantee via the CAS write).

If it is NOT empty, that implies that a previous writer process failed to complete the steps. The new writer takes up this responsibility and performs steps [2, 3] itself before it writes its own index entry.

The Read Path

Now that we have our files stored in S3 and our metadata stored in Oxia, reads can be performed from literally any broker. Our brokers are completely stateless.

When a broker receives a request to fetch starting offset 40 from partition 0 of topic orders, it deterministically knows that the place to figure out which S3 file stores that data is somewhere in Oxia under the key space of llog/orders/partitions/0/index/.

But which exact key is it? If you've noticed, our indexing is sparse.

Assuming our batch size is 50 records per index (i.e the mixed S3 blob had each partition store 50 records in it), Oxia may hold two index keys (per partition) for a hundred records. In this example, they would denote two end offsets - 50 and 100:

llog/orders/partitions/0/index/00000000000000000050
  { ... S3 file, S3 byte offset, etc ... }
 llog/orders/partitions/0/index/00000000000000000100
  { ... S3 file, S3 byte offset, etc ... }

Assume a pathological scenario - a Fetch request comes in for offsets 40-60 (desiring data from both index entries).

The reader issues a so-called Ceiling Get to Oxia. This gets the key-value entry whose key is the lowest one that is above or equal to the supplied parameter. In other words:

ceiling_get(0) # => 50
ceiling_get(40) # => 50
ceiling_get(50) # => 50
ceiling_get(51) # => 100
ceiling_get(99) # => 100

💡(remember this behavior because it's critical to how compaction works)

Because all keys hold end offsets, our reader requesting a ceiling get of 40-60 issues ceiling_get(40) and knows that the entry it received - end offset 50 - holds at least some of the records it wants. When it realizes it ends at record 50, it'll issue a ceiling get of 51 and get the next index entry 100.

Knowing both S3 file locations, the reader performs byte-ranged GETs to fetch that data.

Easy peasy!

Compaction

Last but definitely not least - compaction. If you haven't yet noticed, this data model can result in pretty slow and expensive reads:

Oxia will accumulate a lot of index keys
S3 will accumulate a lot of small files
Readers who want a lot of consecutive record data need to scan multiple Oxia keys and read from multiple S3 files

Just to crunch some numbers - assume our cluster has 10 brokers, assume we persist two WAL blobs a second per broker (the default 500ms per batch), and assume a mixed WAL blob has just ~20 partitions' worth of data -- that's:

34,560,000 sparse index key entries a day
1,728,000 S3 files a day

Each partition would have 1,728,000 index key entries per day alone. Assuming each partition in a mixed WAL blob has ~200 records in it, each index entry itself would also just point to 200 records.

If we could compact each S3 file to instead store, say, 100,000 records per partition and each index entry to denote 5000 records, we'd go down to a more manageable:

3456 S3 files per partition a day
69,120 index entries per partition a day

Or:

69,120 S3 files a day
1,382,400 sparse index key entries a day

So how can we do that?

The Compaction Path

The Compactor is a separate service that reads and mutates Oxia/S3. There is no need for it to talk to the broker that serves reads/writes because its process is asynchronous, and locking is guaranteed through Oxia. The compactor is therefore free to scale separately and not interfere with the broker.

The Compactor works on one partition at a time. To ensure other compactors don't step on each other, it claims a so-called Ephemeral Record in Oxia - this acts as a lightweight distributed lock.

// llog/orders/partitions/0/meta/compactor-claim
  {
    "compactor_id": "compactor-1",
    "claimed_at_ms": 1760000010000
  }

💡 An ephemeral record is one whose lifecycle is tied to a particular client. It stays alive as long as the client heartbeats. If the client dies, the record is deleted by Oxia.

The Compactor keeps a compaction cursor per partition, denoting up to what offset it has compacted:

// llog/orders/partitions/0/meta/compaction-cursor
  {
    "offset": 1
  }

🤫 This single-offset implies we do a one-pass compaction only, which can be inefficient. A better implementation would support multiple passes of compaction, creating ever-larger files with each pass. (up to a limit)

Starting from the last compacted offset, it starts reading /index entries for that partition and its record data from S3. It groups up many such records into a newly-created single partition-exclusive blob file and uploads it to S3.

It then creates a single /compaction key entry in Oxia to persist its progress:

// llog/orders/partitions/0/meta/compaction
  {
    "state": "WRITING_COMPACTED_INDEX",
    "start_offset": 1,
    "end_offset": 100,
    "data_key": "s3://leaderless-log-wal/llog/orders/partitions/0/data/compacted/8b8e9c9df7d94d5f8f2b7b6d3e6a1234",
    // ^ the newly-compacted S3 file
  }

This meta/compaction key acts as the single source-of-truth of the current on-going compaction. The key either has data in it - which means a compaction is on-going, or it's empty - which means no compaction is happening right now.

At this point, we've compacted the data into a new read-optimized file in S3.

The next step is to override the metadata - our /index entries. Those still point to the old mixed S3 blobs when they should actually be pointing to the new compacted file.

Instead of naively overwriting every index key entry at this stage, the protocol only overwrites the max end offset index entry:

// llog/orders/partitions/0/index/00000000000000000100
  {
    "type": "COMPACTED",
    "data_key": "s3://leaderless-log-wal/llog/orders/partitions/0/data/compacted/8b8e9c9df7d94d5f8f2b7b6d3e6a1234",
    // ^ the newly-compacted S3 file
    ...
  }

The rest of the index entries will be deleted.

Remember - readers issue Ceiling GETs to find the end offset of an index entry -- and our many index entries just got merged into one big entry. So naturally, we will be left with one big (compacted) index entry whose end offset is the largest offset in it.

Before they get deleted, the state update has to be persisted:

// llog/orders/partitions/0/meta/compaction
 {
    "state": "DELETING_OLD",
    ...
  }

💭 It's important to durably persist progress. Were the compaction node to die, the fail-over to a new compactor would be faster.

The compactor then deletes all the old index entries for that partition from Oxia.

Once the old index entries are deleted, the compaction state is advanced again:

// llog/orders/partitions/0/meta/compaction
 {
    "state": "UPDATING_CURSOR",
    ...
  }

And the compaction cursor is updated:

// llog/orders/partitions/0/meta/compaction-cursor
  {
    "offset": 101
  }

And then the meta/compaction record is deleted:

// llog/orders/partitions/0/meta/compaction
NULL

The Golden Age of Programming 💛

The funny thing is that I did not come up with these paths, nor did I implement them.

I retroactively learned about how it works in detail.

By pointing my agent to the battle-tested, formally-verified protocol that got shared by StreamNative - my agent implemented everything without burdening me with complex distributed system problems.

It was the subsequent prompts that made it explain things to me which helped me learn.

It is extremely fun to toy around with AI coding when you know what you're doing. The key thing is to:

have a strong foundation in the domain you're working on -- in this case, understand distributed systems at some decent level
have enough experience so as to have proper intuition on where the AI may have screwed up or done something inefficient

👉 The most fun I had was during our iteration over the system's performance. I was aiming to hit a simple 32 MB/s write rate on a single broker. I couldn't.

First, I simply didn't have enough clients sending enough data to reach 32 MB/s per broker (duh...). So I added more (192). Throughput didn't budge but latency grew (285ms → 2074ms). Hm...
Second, I thought we were overloading Oxia with too many requests. Since the number of Oxia operations scales with the number of partitions (around 3-5 ops/s per partition), I figured 128 partitions (up to 910 ops/s) was a tad too much -> lowered partitions to 32. Got some improvement, esp. around latency. (2.6 MB/s -> 4.24 MB/s (up 61%); 1997ms -> 786ms (down 61%)); Still low though. Can't be it.
Oxia exhibited decent latency (max ~5ms per op), so it didn't make sense it would take long. The issue was dumber than I thought. Given Python & the AI, the Oxia metadata requests were all SERIAL. The code would serially send hundreds of requests, always waiting for the previous one to finish. Parallelisation fixed that. ~7.41 MB/s and 523ms - good progress. The bottleneck moved to the client.
Increased the number of HTTP clients again. The way the test was structured, each client would send at most one request at a time. With the given latency per request, and the size of the request - 192 requests in flight weren't enough to reach the target throughput. Increased it to 512. Much higher throughput! (18MB/s, up 162%). But latency also went up - 890ms.
Another dumb server bottleneck - lock contention. The path that checks if a partition exists was using the same lock as the write lock, meaning each request was blocked on the one writing. That made no sense. Removed the lock & added another one -- then we really got a perf boost - 28MB/s and 181ms (yes, latency went down 80%). That particular stage (locking) was taking 532ms... we got it down to 0.09ms.

📱 All these steps were done through my phone in a park. 🌲 When you've got the testing harness right (export results in agent-readable JSON) and you've got a decent intuition of where the system may be slow -- querying the agent is a piece of cake.

Having the AI automate all these tedious and ultra-boring processes was a godsend. I could get 100x more done in a day than I would have pre-AI.

Through this AI coding exercise, I also found a small shard placement bug in Oxia that I fixed, and a feature gap in the Python client that also got fixed.

The Results

Testing this on real S3 and EC2, I got:

100 MB/s writes
100 MB/s reads

the cluster-wide data in and data out throughput rates

inside a single EC2 instance running 5 brokers, Oxia & compactors.

All for less than $0.60/hour of S3 API costs.

The cost deflation of this architecture is real. The equivalent would have cost at least $16.4/hour of cross-AZ network costs in AWS.

But it doesn't come entirely for free. Hitting the real S3 meant much higher latencies than what the local MinIO gave me:

Average writes for 10MB objects were ~200ms, whereas p99 went up to the multi-second threshold.

And herein lies the big tradeoff that this leaderless log architecture brings - higher end-to-end latency.

💡 end-to-end latency - measures the time from which an event was published from a producer application to the time it was read by a consumer application. This is the latency metric Kafka users care about, the rest is marketing fluff.

With this type of diskless, leaderless architecture it's inevitable you incur significantly higher latency than what your regular Kafka would (20-30x). In order of significance, these steps take the most latency:

S3 PUTs - 200-2500ms; S3 Standard simply isn't designed for consistent low latency. Using S3 Express is more complex and incurs a ton more costs
Batching - 100-500ms; In order to save on S3 API costs and keep that $0.60/hour run rate, you have to send less PUT requests. The only way to do that is to batch the data. This helps reduce the number of small files too
Metadata Store - 10-150ms; The metadata store can become a hot component as it's literally in every critical path of the system (write, read, compact)

It is frankly-said impossible to get consistently-low, <100ms e2e latency with this architecture.

This is why I believe the future is in the engines that support both types of topics - the classically-replicated-on-disk Kafka topics and the new diskless variant:

An overview (as of April 2026) of what engines support different topic profiles. Coming soon to the open source Apache Kafka too

👋 Parting Words

Thanks to StreamNative for publishing the leaderless log protocol. It does not give you the full diskless Kafka secret sauce, as key things need to be implemented on top of it:

no batch writes/reads
caching for reads
garbage collection of the mixed S3 log segments

my manual GC results (deleted the whole bucket)

But those are implementation details that are solvable - not correctness constraints. The core distributed system protocol is there for any motivated engineer (or AI agent) to see and build on top of.

I'm sure I could iterate on it and do a lot more, but this is where I'm officially closing the token gate and concluding this experiment. If you want to continue, the repo is diskless-kafka-in-python.

And if you found this article informative, share it with your network. 🌞

Thanks for reading. ~Stan

Kotlin - Unit Testing Classes Without Leaking Public API!

Stanislav Kozlovski — Sat, 05 Dec 2020 08:34:23 +0000

Introduction

Kotlin is an amazing new up-and-coming language. It's very actively developed and has a ton of features that make it very appealing.
It's been steadily gaining market share ever since Google added Android Development support for it (back in 2017) and made it the preferred language for such development exactly one year ago (May 2019)

For any Java developer coming into the language, spoiler alert, there is one big surprise that awaits you --- the package-private visibility modifier is missing. It doesn't exist.

Access Modifiers

An access modifier is the way to set the accessibility of a class/method/variable in object-oriented languages. It's instrumental in facilitating proper encapsulation of components.

Java

In Java, we have four access modifiers for any methods inside a class:

void doSomething()- the default modifier, we call this package-private. Any such declarations are only visible within the same package (hence the name, private for the package)
private void doSomething() --- the private modifier, declarations of which are visible only within the same class
protected void doSomething() --- declarations only visible within the package or all subclasses.
public void doSomething() --- declarations are visible everywhere.

This wide choice of options allows us to tightly-control what our class exposes. It also gives us greater control over what constitutes a unit of testable code and what isn't. See the following crude example:

package doer.something;

/**
 * Does something.
 */
public class JSomethingDoer {

  private int motivationLevel;

  /**
   * @param motivationLevel a 0-100 integer, denoting how motivated the #{@link JSomethingDoer} is.
   */
  public JSomethingDoer(int motivationLevel) {
    this.motivationLevel = motivationLevel;
  }

  public void doSomething() {
    doOneLittleThing();
    maybeDoSecondLittleThing();
  }

  void doOneLittleThing() {
    System.out.println("Doing one thing.");
  }

  // package-private for testing
  void maybeDoSecondLittleThing() {
    if (feelingAmbitious()) {
      doSecondLittleThing();
    }
  }

  private void doSecondLittleThing() {
    System.out.println("We're overachievers, doing a second thing!!!");
  }

  private boolean feelingAmbitious() {
    return motivationLevel > 90;
  }
}

We can only directly test three of the five methods above. Because doSecondLittleThing() and feelingAmbitious() are private, our unit tests literally cannot call those methods, hence we cannot test them directly.

The only way to test them is through the package-private method maybeDoSecondLittleThing() , which can be called in our unit tests because they're in the same package as the class they're testing.

package doer.something;

import org.junit.Before;
import org.junit.Test;

public class JSomethingDoerTest {

  private JSomethingDoer doer;

  @Before
  public void setUp() {
    doer = new JSomethingDoer(11);
  }

  @Test
  public void Test_doSomething() {
    doer.doSomething();
    // assert
  }

  @Test
  public void Test_doOneLittleThing() {
    doer.doOneLittleThing();
    // assert
  }

  @Test
  public void Test_maybeDoSecondLittleThing() {
    doer.maybeDoSecondLittleThing();
    // assert
  }
}

The package-private modifier comes very useful in this example, because it allows us to test the maybeDoSecondLittleThing() method directly!
Testing would be more cumbersome if we had to test all code paths of the maybeDoSecondLittleThing() method by calling the public doSomething() method.

The package-private modifier also ensures that we don't leak any such internal methods to packages outside of this one.

Notice the different package doer in the Main class.

Kotlin

Kotlin is a bit weirder in this regard. It also has four modifiers:

private fun doSomething() --- vanilla private modifier
protected fun doSomething() --- the same as Java's protected --- only visible within this class and subclasses.
public fun doSomething() --- visible everywhere

These three are pretty standard. I want to focus on the fourth one:

internal fun doSomething() --- Internal declarations are visible anywhere inside the same module.

This is as good as public inside the same module, in that sense. Many small projects have no need for multiple modules, but they're likely to have multiple packages.

There is basically no way to limit access to a Kotlin method to only be within the same package.

What does this mean? It means that we either need to make our methods private, thus making our testing life harder, or make them public, thus exposing unnecessary API to components in the same module.

Our same example from above, again in the doer.something package:

package doer.something

/**
 * Does something.
 * @param motivationLevel a 0-100 integer, denoting how motivated the #[SomethingDoer] is.
 */
open class SomethingDoer(private val motivationLevel: Int) {
  fun doSomething() {
    doOneLittleThing()
    maybeDoSecondLittleThing()
  }

  protected fun doOneLittleThing() {
    println("Doing one thing.")
  }

  private fun maybeDoSecondLittleThing() {
    if (feelingAmbitious()) {
      doSecondLittleThing()
    }
  }

  private fun doSecondLittleThing() {
    println("We're overachievers, doing a second thing!!!")
  }

  private fun feelingAmbitious(): Boolean {
    return motivationLevel > 90
  }
}

But this time, from the doer package, we can call every method:

How do we solve it?

Solution 1 --- Refactor!

It seems to be commonly believed that we shouldn't test methods that are/should be private. The idea being that:

all private methods are reachable by the public methods
the public methods expose the interface/contract of a class
private methods are implementation details, and we want to test the class' functionality, not implementation

While that makes sense in theory, reality is not as black-and-white.

Regardless, you should treat any need to test non-public methods as a code smell.
Re-evaluate your design and rethink whether it makes sense to refactor the logic around such that you get the same test coverage by testing public interface contracts, rather than implementation details.

In some cases, though, it is cleaner and more maintainable to get full test coverage by testing each tiny private method code thoroughly, rather than testing each conditional branch via public methods or creating unnecessary boilerplate (e.g additional classes).

Solution 2 --- Reflection

An alternative solution is to keep the methods private and use the language's reflection features to call into said private methods.

Add a dependency on the kotlin-reflect library and code away!

  fun callPrivate(objectInstance: Any, methodName: String, vararg args: Any?): Any? {
    val privateMethod: KFunction<*>? =
        objectInstance::class.members.find { t -> return@find t.name == methodName } as KFunction<*>?

    val argList = args.toMutableList()
    (argList as ArrayList).add(0, objectInstance)
    val argArr = argList.toArray()

    if (privateMethod?.javaMethod?.trySetAccessible()!!) {
      privateMethod?.apply {
        return call(*argArr)
      } ?: throw NoSuchMethodException("Method $methodName does not exist in ${objectInstance::class.qualifiedName}")
    } else throw IllegalAccessException("Method $methodName could not be made accessible")

    return null
  }

Woah.

Here, our callPrivate() test helper is the star of the show! It allows us to call into private methods, albeit in a funky way.
We need to specify the method name by a string, hence we lose all type checking guarantees. This results in fragile code that's also hard to read.

We can leverage a compiler plugin, like Manifold, to get type-safe meta-programming. But you can argue that takes it a bit too far, as it introduces a large dependency that involves a lot more meta-programming behind the scenes.

Solution 3 --- Open the class, protect the methods and test a wrapper!

A third, perhaps preferable way to solve this is with a bit of subclassing work.

First, we need to make our class open for extension via the open keyword. Then, we need to make our private methods protected.

open class SomethingDoer(private val motivationLevel: Int) {
  fun doSomething() {
    doOneLittleThing()
    maybeDoSecondLittleThing()
  }

  protected fun doOneLittleThing() {
    println("Doing one thing.")
  }
}

Finally, we can create a subclass inside the test class whose sole goal is to expose the protected method directly.

class SomethingDoerTest {

  class TestSomethingDoer(motivationLevel: Int) : SomethingDoer(motivationLevel) {
    fun testDoOneLittleThing() {
      super.doOneLittleThing()
    }
  }
  private lateinit var doer: SomethingDoer
  private lateinit var doerWrapper: TestSomethingDoer

  @Before
  fun setUp() {
    doer = SomethingDoer(99)
    doerWrapper = TestSomethingDoer(99)
  }

  @Test
  fun doSomething() {
    doer.doSomething()
    // assert
  }

  @Test
  fun Test_doOneLittleThing() {
    doerWrapper.testDoOneLittleThing()
    // assert
  }
}

This is yet another imperfect solution to the problem which results in more boilerplate and a non-totally-private method.

It is better than the reflection approach in that it's type-safe, therefore less prone to breakage and significantly easier to maintain.

Summary

In this short article we went over the access modifiers of both Java and Kotlin and saw where Kotlin comes short. We proposed some solutions to Kotlin's lack of the package-private modifier with relation to testing, all with sufficient code examples!

Speaking of which, the code for this article live on GitHub --- here.

References

This problem has been hit by many, unsurprisingly. It makes sense given that most people come from the same background --- we miss our good old Java features 😭

Kotlin Discussion Forum --- https://discuss.kotlinlang.org/t/kotlin-to-support-package-protected-visibility/1544; https://discuss.kotlinlang.org/t/another-call-for-package-private-visibility/9577
KT Issue on the matter --- https://youtrack.jetbrains.com/issue/KT-29227
Another take on Medium --- https://medium.com/virtuslab/on-the-missing-package-private-or-why-java-is-better-than-kotlin-in-this-regard-4a1c9ecbe40c

Working With Multithreaded Ruby Part II

Stanislav Kozlovski — Mon, 16 Oct 2017 07:19:09 +0000

We're back at it with another edition of Multithreaded Ruby where we'll continue to dive into concurrency using our beloved language!
Today, I'm going to introduce you to a famous multi-process synchronization problem called the Producer-Consumer problem and we're going to look at Ruby's ConditionVariable class.

Back to Deadlock

A paragraph into the new article and we're at deadlocks again? Well yes, they're pretty prevalent and we did not actually touch on a solution to the problem last time.
Let's bring back the deadlock example we used in Part I, modified just a tiny bit.

require 'thwait'

item_accessories = {}
item = {}
item_acc_lock = Mutex.new
item_lock = Mutex.new

a = Thread.new {
  item_acc_lock.synchronize {
    sleep 1  # pretend to work on item_accessories
    item_lock.synchronize {
      # pretend to work on item
      sleep 1
      puts 'Worked on accessories, then on item'
    }
  }
}

b = Thread.new {
  item_lock.synchronize {
    sleep 1  # pretend to work on item
    item_acc_lock.synchronize {
      # pretend to work on item_accessories
      sleep 1
      puts 'Worked on item, then on accessories'
    }
  }
}

ThWait.all_waits(a, b)

> enether$ ruby item_worker.rb
/Users/enether/.rvm/rubies/ruby-2.4.1/lib/ruby/2.4.0/thwait.rb:112:in `pop': No live threads left. Deadlock? (fatal)

No surprise here, thread a obviously takes a hold of item_acc_lock, thread b takes a hold of item_lock and each of them waits for the opposite lock in an endless loop. So how could we avoid this?
What if we had a way to temporarily release one of the locks at a specific point in our program where we could afford doing so? That way, the other thread could take the lock, do its thing and return it back for the original one to finish its work.

Enter ConditionVariable

ConditionVariable is a Ruby class which lets you block a thread until another thread signals it OK to continue. It is a way to say - "I'm waiting for a lock and I can give up mine at this exact time". It is an ideal way to synchronize our a and b threads here:

require 'thwait'

item_accessories = {}
item = {}
item_acc_lock = Mutex.new
item_lock = Mutex.new
cv = ConditionVariable.new

a = Thread.new {
  item_acc_lock.synchronize {
    sleep 1  # pretend to work on item_accessories
    # At this point, we've just finished work on item_accessories and we're at a window where we
    #   might not care if item_accessories changes. So: let somebody else take it and give it back
    cv.wait(item_acc_lock)  # Temporarily sleeps the thread and releases the lock
    puts 'Gained back access to item_acc_lock'  # on this line, item_acc_lock is re-acquired 
    item_lock.synchronize {
      # pretend to work on item
      sleep 1
      puts 'Worked on accessories, then on item'
    }
  }
}

b = Thread.new {
  item_lock.synchronize {
    sleep 1  # pretend to work on item
    item_acc_lock.synchronize {
      # pretend to work on item_accessories
      sleep 1
      puts 'Worked on item, then on accessories'
    }
    cv.signal
    puts "I'm still working, but I'm finished with item_acc_lock"
  }
}

ThWait.all_waits(a, b)

> enether$ ruby synchronized_item_worker.rb
Worked on item, then on accessories
I'm still working, but I'm finished with item_acc_lock
Gained back access to item_acc_lock
Worked on accessories, then on item

What we achieved here is a sort of synchronization: we can now be sure that one b thread will always reach its cv.signal line before an a thread starts working on item_accessories.
Here is a picture visualizing the process:

You might want to open this in another tab - High-Resolution

It is worth noting that the ConditionVariable#signal method will only wake up one thread which is waiting for the variable. This means that if we have two threads waiting on a ConditionVariable and its signal method is called only once, the thread that does not get called will end up waiting forever for the ConditionVariable, resulting in a deadlock

require 'thwait'

lock = Mutex.new
cv = ConditionVariable.new
threads = []
2.times do
  threads << Thread.new {
    lock.synchronize { cv.wait(lock) }
  }
end

threads << Thread.new {
  lock.synchronize { sleep 1 }
  cv.signal
}

ThWait.all_waits(*threads)

> enether$ ruby cv_pitfall.rb
/Users/enether/.rvm/rubies/ruby-2.4.1/lib/ruby/2.4.0/thwait.rb:112:in `pop': No live threads left. Deadlock? (fatal)

In such a scenario, you need to either call signal as many times as there are waits or use another method - ConditionVariable#broadcast, which will wake up every thread that is waiting on the condition variable.

Producer-Consumer

The producer-consumer problem consists of at minimum two threads, one representing a producer and one representing a consumer.

Producer - Sole job is to create an item and put it into the buffer
Consumer - Sole job is to take the item from the buffer and process it

Here is a sample implementation in Ruby, where the tasks array acts as the buffer with a fictional limitation of having at most 2 items in it at once:

require 'thwait'

threads = []
tasks = []
mutex = Mutex.new

# producer
2.times do
  threads << Thread.new do
    loop do
      mutex.synchronize do
        if tasks.length < 2
          tasks << "Task :)"
        end
      end
    end
  end
end


# consumer
5.times do
  threads << Thread.new do
    loop do
      task = nil
      mutex.synchronize do
        if tasks.length != 0
          task = tasks.shift
        end
      end
      unless task.nil?
        100000.times do
          # Simulating task execution's CPU work
          # also doing it outside the mutex so we don't block the tasks array (other producer might want to take a task as well)
        end
      end
    end
  end
end

ThWait.all_waits(threads)

Producer-Consumer problem

As the name implies, something is not quite right with the code above. Finding problems in concurrent code is hard, so I'm going to give you a couple of minutes to figure out what is wrong.

...
...
...

Okay, if you managed to figure it out - great, if not - don't fret, multithreaded programming is unintuitive.
The problem with the code above is that it wastes time. You see, as we can't control when and to which thread the OS switches to, there is the possibility that we leave our consumer thread and enter the producer's when there is no reason to.
Imagine all our consumer threads are currently executing a task and our tasks array is full (has two elements). If the OS decides to do a context switch and gives control to a producer thread it would only waste time. Since the tasks array would be full, the producer would loop only to check that the array is full and not do anything else, only managing to take precious CPU time from our consumer threads.

Let's track exactly how much useless iterations this thing does:

require 'thwait'

threads = []
tasks = []
mutex = Mutex.new

to_exit = false
times_tasks_added = 0
times_time_wasted = 0
executed_tasks = 0

# consumer
2.times do
  threads << Thread.new do
    loop do
      Thread.kill(Thread.current) if to_exit  # a way to stop execution

      mutex.synchronize do
        if tasks.length < 2
          tasks << "Task :)"
          times_tasks_added += 1
        else
          # time here is absolutely wasted
          times_time_wasted += 1
        end
      end
    end
  end
end

# producer
5.times do
  threads << Thread.new do
    loop do
      Thread.kill(Thread.current) if to_exit  # a way to stop execution
      task = nil
      mutex.synchronize do
        if tasks.length != 0
          task = tasks.shift
        end
      end
      unless task.nil?
        100000.times do
          # Simulating CPU work
          # also doing it outside the mutex so we don't block the tasks array (other producer might want to take a task as well)
        end
        executed_tasks += 1
        if executed_tasks >= 100  # don't loop forever
          to_exit = true
        end
      end
    end
  end
end

ThWait.all_waits(threads)

puts "Total tasks added: #{times_tasks_added}"
puts "Total times we branched out into the useless else statement: #{times_time_wasted}"

And here are the results:

> enether$ ruby squander_of_time.rb
Total tasks added: 102
Total times we branched out into the useless else statement: 1633
> enether$ ruby squander_of_time.rb
Total tasks added: 102
Total times we branched out into the useless else statement: 848282
> enether$ ruby squander_of_time.rb
Total tasks added: 102
Total times we branched out into the useless else statement: 356418

We know that thread context-switching is non-deterministic and these results further prove it. Sometimes we do as little as 1633 useless executions of the else branch and some - as much as 848k, 8316 times more than we need!
This is bad because it will cause your program to run slower at some times and behave seemingly normal in others.
It's good to note that I personally found it hard to figure out the problem in this code even though it's specifically made to illustrate it. Imagine how hard it would be to spot such a thing in an established codebase!

This slow-down is not acceptable, so let's fix it. Thinking the problem through, our problem seems to boil down to having a producer thread wake up when it doesn't make sense.
What would be perfect is if we had the ability to somehow control when we resume the producer thread - specifically when a task is removed from the tasks buffer so we're sure there's room to add another one.

ConditionVariable to the rescue!

The fix is simple: We're just going to put a condition variable which is going to give up the mutex lock whenever we detect that we do not need to continue looping in the producer. We're also going to need to tell the producer he can resume when we have an empty spot in the tasks buffer.

require 'thwait'

threads = []
tasks = []
mutex = Mutex.new

to_exit = false
times_tasks_added = 0
times_time_wasted = 0
executed_tasks = 0
cv = ConditionVariable.new

# consumer
2.times do
  threads << Thread.new do
    loop do
      Thread.kill(Thread.current) if to_exit  # a way to stop execution

      mutex.synchronize do
        if tasks.length < 2
          tasks << "Task :)"
          times_tasks_added += 1
        else
          times_time_wasted += 1
          cv.wait(mutex)  # no need to continue looping in such a case, only continue after it makes sense
        end
      end
    end
  end
end

# producer
5.times do
  threads << Thread.new do
    loop do
      Thread.kill(Thread.current) if to_exit  # a way to stop execution
      task = nil
      mutex.synchronize do
        if tasks.length != 0
          task = tasks.shift
          cv.signal  # one new task can now be added
        end
      end
      unless task.nil?
        100000.times do
          # Simulating CPU work
          # also doing it outside the mutex so we don't block te tasks array (other producer might want to take a task as well)
        end
        executed_tasks += 1
        if executed_tasks >= 100  # don't loop forever
          to_exit = true
        end
      end
    end
  end
end

ThWait.all_waits(threads)

puts "Total tasks added: #{times_tasks_added}"
puts "Total times we branched out into the useless else statement: #{times_time_wasted}"

> enether$ ruby saver_of_time.rb
Total tasks added: 101
Total times we branched out into the useless else statement: 50
> enether$ ruby saver_of_time.rb
Total tasks added: 100
Total times we branched out into the useless else statement: 45
> enether$ ruby saver_of_time.rb
Total tasks added: 100
Total times we branched out into the useless else statement: 42

Woo, performance!

In reality, it's worth noting that the previous example and this one actually run in about the same time. 400k useless iterations sound like a lot but our computers are fast enough to not let us notice this inefficiency. Regardless, I hope this example managed to clearly illustrate the problem.

Further optimization

Do we even need to enter the else branch at all? Could we not put cv.wait inside the block which adds the tasks and have it call wait when the buffer is full? We can and that way it should never enter the else block, as it would only be resumed to add a task and sleep if the buffer is full.

# ...
loop do
  Thread.kill(Thread.current) if to_exit  # a way to stop execution

  mutex.synchronize do
    if tasks.length < 2
      tasks << "Task :)"
      times_tasks_added += 1
      if tasks.length >= 2
        cv.wait(mutex)  # no need to continue looping in such a case, only continue after it makes sense
      end
    else
      times_time_wasted += 1
    end
  end
end
# ...

> enether$ ruby no_time_wasted.rb
Total tasks added: 100.
Total times we branched out into the useless else statement: 16
> enether$ ruby no_time_wasted.rb
Total tasks added: 102
Total times we branched out into the useless else statement: 13
> enether$ ruby no_time_wasted.rb
Total tasks added: 101
Total times we branched out into the useless else statement: 372021

What the hell? We still got entered the else block and we even got back to the previous levels of needless execution!

Concurrent programming is hard. We're using two producer threads here and when one fills up the tasks array it frees the mutex. The other producer thread seems to get resumed (remember we free the mutex only when tasks is full) and enters the else branch as the tasks buffer is full.

You might want to open this in another tab - High-Resolution

Okay, well the simplest thing to do is put a wait back where we had one. This should limit the useless calls as much as possible:

# ...
loop do
  Thread.kill(Thread.current) if to_exit  # a way to stop execution

  mutex.synchronize do
    if tasks.length < 2
      tasks << "Task :)"
      times_tasks_added += 1
      if tasks.length >= 2
        cv.wait(mutex)
      end
    else
      times_time_wasted += 1
      cv.wait(mutex)
    end
  end
end
# ...

> enether$ ruby no_time_wasted_fixed.rb
Total tasks added: 101
Total times we branched out into the useless else statement: 5
> enether$ ruby no_time_wasted_fixed.rb
Total tasks added: 101
Total times we branched out into the useless else statement: 8
> enether$ ruby no_time_wasted_fixed.rb
Total tasks added: 101
Total times we branched out into the useless else statement: 3

Well I'm afraid this is as much as we can do with the ConditionVariable. The reason we still enter the else block a couple of times is most likely the one depicted in the image above.
Although there is one other possibility:

Spurious Wakeups

A spurious wakeup is when a ConditionVariable gets woken up without getting signaled to. This might sound stupid but it makes sense, since it seems to boost performance in some cases. According to David R. Butenhof's Programming with POSIX Threads (ISBN 0-201-63392-2): "Spurious wakeups may sound strange, but on some multiprocessor systems, making condition wakeup completely predictable might substantially slow all condition variable operations.".
They also enforce robust multithreaded code, essentially enforcing you to take care of such cases. This is why it is strongly recommended that you always put your ConditionVariables inside a loop which always checks the appropriate condition (as we do with if tasks.length < 2).
Here is an interesting discussion on the topic: comp.programming.threads

I personally could not identify when a producer thread was woken up spuriously or simply got scheduled when a previous producer went to sleep. I did dig through the MRI code to verify that cv.wait is vulnerable to spurious wakeups.
Here is the way it gets called - rb_condvar_wait -> do_sleep -> mutex_sleep -> rb_mutex_sleep -> rb_mutex_sleep_forever -> rb_thread_sleep_deadly_allow_spurious_wakeup -> sleep_forever
It seems to boil down to calling the sleep_forever function which calls native_sleep in a loop. After the code exits from the sleep, Ruby checks if the thread was woken up on purpose (in RUBY_VM_CHECK_INTS_BLOCKING(th)) and schedules it to be interrupted if so. Since ours isn't, it likely enters the if (!spurious_check) block and breaks the loop, effectively stopping the sleep.

Summary

We touched on a couple of important topics in multithreaded programming.
We learned about the precious ConditionVariable class and more specifically how it allows you to pause threads at will and schedule a resume when you decide to.

ConditionVariable#wait(mutex) - puts the current thread to sleep, releases the given mutex for the time being and gets resumed strictly after a signal
ConditionVariable#signal - allows one thread that holds the given condition variable that to resume
ConditionVariable#broadcast - allows all threads that hold the given condition variable to resume

We dabbled into the producer-consumer problem, trying to optimize it on our own and further explored concurrency problems and in the end learned about spurious wakeups.

Working with Multithreaded Ruby Part I

Stanislav Kozlovski — Tue, 26 Sep 2017 06:47:26 +0000

Introduction

Multithreaded Ruby is a niche topic in our community and to no surprise. Most Ruby applications are web servers built on Rails or Sinatra, those are single-threaded frameworks and developers on such projects rarely even need to know about threads, as the framework usually has got your back.

Even if you do not use it, some basic knowledge of multithreading (and its basic concepts) in an interpreted language like Ruby will surely come in handy throughout your career.

I assume you know about the GIL (Global Interpreter Lock). In case you don't know what it is, you can read my article Ruby's GIL in a nutshell

GIL != the end of the world

Even though it limits parallelism, Ruby's GIL does not completely stop it. As we know, it exists to guard the interpreter's internal state. As such, it only applies to Ruby operations. In our normal day-to-day code there are a lot of operations that are not the job of Ruby's interpreter to handle.

A good example is I/O operations. While waiting for an external service to load something, there is no need to hold the GIL, as this external service cannot harm our internal state.
Ruby's PostgreSQL library is written in C and its method call for a DB query releases the GIL. The following example shows that:

require 'thwait'
require 'pg'

start = Time.now

first_sleep = Thread.new do
  puts 'Starting sleep 1'
  conn = PG::Connection.open(dbname: 'test')
  conn.exec('SELECT pg_sleep(1);')
  puts 'Finished sleep 1'
end

second_sleep = Thread.new do
  puts 'Starting sleep 2'
  conn = PG::Connection.open(dbname: 'test2')
  conn.exec('SELECT pg_sleep(1);')
  puts 'Finished sleep 2'
end

random = Thread.new do
  puts 'In a random thread'
end

ThWait.all_waits(first_sleep, second_sleep, random)

puts "Time it took: #{Time.now - start}"

Here we spin up two threads, create a connection to different databases and run a sleep query for a second. Without parallelism, this should take at minimum 2 seconds.

> enether$ ruby async_pg.rb
> Starting sleep 2
> Starting sleep 1
> In a random thread
> Finished sleep 2
> Finished sleep 1
> Time it took: 1.074824

But it runs in 1 second!
This proves that the PostgreSQL query does not hold the GIL and lets the other thread take control. Not only does it not lock the interpreter but it actually runs the query in parallel with the other query, that's the only way in which we could achieve a 1 second execution time to run two sleep queries!

Reminder: The GIL does not protect you

A problem can occur when two or more threads access shared data and try to change it. This is called a race condition.
Because Ruby's thread scheduling algorithm can swap between threads at any time, you don't know the order in which the threads will attempt to access the shared data. Therefore, the result of the change in data is dependent on the algorithm and seemingly out of your control.
It is therefore possible for two threads to modify data in such a sequence where you get an unexpected outcome.

Here is an example of the so called check-and-act race condition, where you check for a variable's value and then act in regards to it.

require 'thwait'

def send_money(amount)
  puts "Sending $#{amount}"
  sleep 1  # Simulate network call sending of money PS: This is I/O, so you know Ruby releases GIL here
end


threads = []
money_is_sent = false

2.times do
  th = Thread.new do
    unless money_is_sent
      send_money 10
      money_is_sent = true
    end
  end
  threads << th
end


ThWait.all_waits(*threads)

We obviously want to send the money only once but running the code shows that this is not the case

> enether$ ruby balling.rb
> Sending $10
> Sending $10

This is what happens here

As you saw, what looks like straightforward code can end up producing a huge problem (losing us money!) when executed concurrently. It is up to you to make your code thread-safe.

How to protect yourself

So how could we avoid such race conditions?
Simple, you can take the same approach as the Ruby Core team and introduce your own lock (kind of like the GIL), which would be a local lock on a block of code.
This is called a Mutex (Mutual Exclusion) and it helps you synchronize access to blocks of code, acting like a gatekeeper.

require 'thwait'

def send_money(amount)
  puts "Sending $#{amount}"
  sleep 1  # Simulate network call sending of money
end

lock = Mutex.new
threads = []
money_is_sent = false

2.times do
  th = Thread.new do
    lock.synchronize {
      unless money_is_sent
        send_money 10
        money_is_sent = true
      end  
    }
  end
  threads << th
end


ThWait.all_waits(*threads)

We define a Mutex and call the synchronize method. When we enter the block in the synchronize method, our mutex gets locked. If another thread tries to access code through lock.synchronize it will see that the lock is locked and pause until it is unlocked.

> enether$ ruby balling_on_a_budget.rb
> Sending $10

Be sure to note that lock.synchronize only prevents a thread from being interrupted by others wanting to execute code wrapped inside the same lock variable!
Creating two different locks will obviously not work.

2.times do
  Thread.new do
    Mutex.new.synchronize {
      unless money_is_sent
        send_money 10
        money_is_sent = true
      end
    }
  end
end

> enether$ ruby lock_city.rb
> Sending $10
> Sending $10

yeah, no way

Mutexes are not perfect

Now that we know about these locks, we need to pay attention to how we use them. They offer protection but there is also a possibility where that can backfire on you if not used correctly.
It is possible to end up in a so-called deadlock (sounds scary, doesn't it?). A deadlock is a situation where one thread that holds mutex A waits for a mutex B to be released but the thread that holds mutex B is waiting for mutex A.

require 'thread'
require 'thwait'

first_lock = Mutex.new
second_lock = Mutex.new

a = Thread.new {
  first_lock.synchronize {
    sleep 1  # essentially forces a context switch
    second_lock.synchronize {
      puts 'Locked #1 then #2'
    }
  }
}

b = Thread.new {
  second_lock.synchronize {
    sleep 1  # essentially forces a context switch
    first_lock.synchronize {
      puts 'Locked #2 then #1'
    }
  }
}

ThWait.all_waits(a, b)

> enether$ ruby dead_lock.rb
> /Users/enether/.rvm/rubies/ruby-2.4.1/lib/ruby/2.4.0/thwait.rb:112:in `pop': No live threads left. Deadlock? (fatal)

They are both holding what the other thread wants and waiting for what the other thread has.
Of course, this is a pretty specific example and there are not many cases in which you might use two mutexes in such a way, but it is essential to know about this pitfall.

Summary

We saw that regardless of the GIL you can still do tasks asynchronously (I/O and native libraries) and confirmed that it won't save you from your thread-unsafe code.
You learned about the most common pitfall - the check-then-act race condition, we introduced a way of handling the problem through our own little GIL-esque lock (Mutex) and we saw that even that can backfire.

I hope I've managed to showcase how tricky multithreaded programming can turn out to be and how it can introduce problems you would not consider programming synchronously.

Crystal - the Ruby you've never heard of

Stanislav Kozlovski — Mon, 18 Sep 2017 12:19:29 +0000

What?

Crystal is a new, elegant, multi-paradigm programming language that is productive and fast. It has ~~Ruby's~~ a Ruby-inspired syntax and compiles to native code. It is actually unreal how similar to Ruby this language looks like.
This language combines efficient code with developer productivity, adds full OOP, a great concurrency model and a compiler that holds your hand.

This article is meant to give you a short overview, a direct performance comparison to Ruby and show some things that set it apart. It is advised you know at least some Ruby before continuing on reading.

Starting with the fun stuff - a performant example

Let's actually get a feel as to how performant Crystal is.
I wrote an AA Tree in both Crystal and Ruby.
Note: Code quality might not be top-notch. Some lines of Crystal code are intentionally written more explicitly

We are going to be running this code to benchmark each implementation:

elements_count = ARGV[0].to_i  # first command line argument
root = AANode.new(value: elements_count, level: 1)
tree = AATree.new(root)

start = Time.now

elements_count.times do |num|
  raise Exception.new("Tree should not contain #{num}") if tree.contains?(num)
  tree.add(num)
  raise Exception.new("Tree should contain #{num}") unless tree.contains?(num)
end

elements_count.times do |num|
  raise Exception.new("Tree should contain #{num}") unless tree.contains?(num)
  tree.remove(num)
  raise Exception.new("Tree should not contain #{num}") if tree.contains?(num)
end

puts "Time it took: #{Time.now - start} seconds."

What this essentially does is it adds numbers to our tree (which sorts them internally) and then removes each one, one by one. We also check if the tree contains the given number twice per addition/deletion.

The code snippet above is actually Crystal code.
Like I said, these languages are identical at first glance. Rewriting the code from Crystal to Ruby took me a total of 50 line changes for a 360 line file. 27 if you were greedy
It is worth noting that those changes are simply removing .as() method calls and type annotations.

Okay, they look identical but how much faster is Crystal?

Let's build the executable and start testing

> enether$ crystal build AA_Tree.cr -o crystal_tree --release

# 100 elements
> enether$ ./crystal_tree 100
Time it took: 0.0006560 seconds.
> enether$ ruby AA_Tree.rb 100
Time it took: 0.00172 seconds.

# 10K elements
> enether$ ./crystal_tree 10000
Time it took: 0.0044000 seconds.
> enether$ ruby AA_Tree.rb 10000
Time it took: 0.288619 seconds.

# 100K elements
> enether$ ./crystal_tree 100000
Time it took: 0.0498230 seconds.
> enether$ ruby AA_Tree.rb 100000
Time it took: 3.414404 seconds.

# 1 million elements
> enether$ ./crystal_tree 1000000
Time it took: 0.5007820 seconds.
> enether$ ruby AA_Tree.rb 1000000
Time it took: 39.370083 seconds.

# 10 million elements
> enether$ ./crystal_tree 100000000
Time it took: 5.6283920 seconds.
> enether$ ruby AA_Tree.rb 100000000
# Still running

As you can see, it runs laps around Ruby and proves to be ~80 times faster if we were to judge by our 1 million elements example.

Quirks and differences to Ruby

Despite the similarities, there are substantial differences to Ruby, here we will highlight the most obvious and interesting ones.

Types, type checking and type unions

The most apparent difference is that Crystal uses and mostly enforces types for variables. It has great type inference - if you do not explicitly define the type of a variable the compiler figures it out itself.
The way this language does typing is a sort of mix between static and dynamic typing. It allows you to change a variable's type

a = "Hello"
puts typeof(a) # => String
a = 42
puts typeof(a) # => Int32

but it also allows you to enforce a variable's type

a : String = "Hello"  # a should be a string and only a string!
a = 42 # error:  type must be String, not (Int32 | String)

Type Unions

Were you wondering what the (Int32 | String) type was in the error message above?
This is a so-called type union, which is a set of multiple types.
If we were to enforce a to be a union of Int32 and String, the compiler would allow us to assign either type to that variable as it knows to expect both.

a : (Int32 | String) = 42
a = "Hello"
# Completely okay

# But if we were to try to assign another type to it
a = true # => type must be (Int32 | String), not (Bool | Int32 | String)

Type Inference and Type Checking

The compiler can figure out the type of a variable himself in most cases. The type inference algorithm is specifically built to work when the type of the variable is obvious to a human reader and does not dig too deep into figuring out the specific type.

In the cases where multiple conditions are plausible, the compiler puts a union type on the variable. Crystal code won't compile if the possible types do not support a given method invoked on them.

if rand() > 0.5
  a = "String"
  puts typeof(a) # => String
else
  a = 42
  puts typeof(a) # => Int32
end
puts typeof(a) # => (String | Int32)
puts a.camelcase  # => undefined method 'camelcase' for Int32 (compile-time type is (Int32 | String))

This is the way the compiler protects you from silly mistakes with mismatched types, something that is really common in dynamic languages. Its like having your very own programming assistant!

The compiler is smart enough to figure out when a variable is obviously from a given type

if rand() > 0.5
  a = "String"
elsif rand() > 0.75
  a = 42
else
  a = nil
end
puts typeof(a) # => (String | Int32 | Nil)
unless a.nil?
  # a is not nil for sure
  puts typeof(a)  # => (String | Int32)
end
if a.is_a?(String)
  puts typeof(a)  # => String
end

There are ways to ensure the compiler that the appropriate type is set.

puts a.as(String).camelcase

This checks that the a variable is a string and if it is not, it throws an error.

Enforcing types

As we said, we have the option to enforce a variable's type or let it be whatever.
This holds true for a method's parameters as well. Let's define two methods:

def generic_receiver(item)
  puts "Received #{item}!"
end

def string_receiver(item : String)
  puts "Received string #{item}!"
end

I assume you can already imagine what'll happen with the following code:

generic_receiver(1)
generic_receiver("Hello")
generic_receiver(1.5)
generic_receiver(true)
string_receiver("Hello")
string_receiver(1)  # error!

It is usually good practice to not enforce a variable, as it leads to more generic code.

Concurrency

Its concurrent model is inspired by that of Go, namely CSP (Communication Sequential Processing).
It uses lightweight threads (called fibers) whose execution is managed by the runtime scheduler, not the operating system. Communication between said threads is done through channels, which can either be unbuffered or buffered.

Pictured: A lot of fibers who communicate between each other through channels

Crystal currently runs in a single thread but their roadmap intends to implement multithreading. This means that it currently has no support for parallelism (except for process forking), but that is subject to change.
Because at this moment there's only a single thread executing your code, accessing and modifying a variable in different fibers will work just fine. However, once multiple threads is introduced in the language, it might break. That's why the recommended mechanism to communicate data is through channels.

Metaprogramming

Crystal has good support for metaprogramming through macros. A macro is something that pastes code into the file during compilation.

Let's define our own version of Ruby's attr_writer

macro attr_writer(name, type)
    def {{name}}=({{name.id}} : {{type}})
        @{{name}} = {{name.id}}
    end
end

Calling attr_writer foo, Int32 will evaluate to

def foo(foo : Int32)
 @foo = foo
end

class Greeter
    attr_writer hello_msg, String

    def hello_msg
        @hello_msg
    end
end


gr = Greeter.new
gr.hello_msg = "Hello World"
puts gr.hello_msg # => Hello World
gr.hello_msg = 11 # => no overload matches 'Greeter#hello_msg=' with type Int32

Crystal macros support iteration and conditionals and can access constants.

MAX_LENGTH = 3
macro define_short_methods(names)
  {% for name, index in names %}
    {% if name.id.size <= MAX_LENGTH %}
        def {{name.id}}
          {{index}}
        end
    {% end %}
  {% end %}
end

define_short_methods [foo, bar, hello]
puts foo # => 0
puts bar # => 1
# puts hello => undefined local variable or method 'hello'

Miscellaneous

Crystal has taken a lot of cool features from other languages and provides various syntax sugar that is oh so sweet!

Initializing class instance variables directly in a method

def initialize(@name, @age, @gender, @nationality)

is equal to

def initialize(name, age, gender, nationality)
  @name = name
  @age = age
  @gender = gender
  @nationality = nationality
end

Implicit object notation

Switch statements support invoking methods on the giving object without repeatedly specifying its name.

case string_of_the_gods
when .size > 10
  puts "Long string"
when .size == 5
  puts "Normal String"
when .size < 5
  puts "Short String"
end

case {1, 1}
when {.even?, .odd?}
  # Matches if value1.even? && value2.odd?
end

External keyword arguments

My personal favorite - Crystal allows you to name a function's parameters one way for the outside world and one way for the method's body

def increment(number, by value)
  number + value
end
increment(10, by: 10)

Compiler

As you saw earlier, this language is compiled to an executable. Regardless, it still has something like a REPL which proves to be similar to our beloved irb - https://github.com/crystal-community/icr
You can also directly run a file without having to compile it and then run it, via the crystal command.

> enether$ crystal AA_Tree.cr 200000
Time it took: 0.536102 seconds.

This runs a little bit slower because we do not make use of the optimizations that the --release build flag brings with itself.

C Bindings

There is a way to write a performant library in Crystal which you can run in your Ruby code. The way you do this is to bind Crystal to C, which allows you to use it from Ruby.
I did not delve too deep into this but apparently it is easy and you can do it without writing a single line of C. That is awesome!

Conclusion

If you write Ruby, picking up Crystal is natural and can quickly find yourself writing performance-critical software in it. I believe it has a lot of potential and can yield a lot of benefits to our community but also to non-ruby programmers, as the syntax is just too easy to pass up. It is a joy to write and it runs blazingly fast, that is an unique combination which very few languages can boast with.
I hope I've sparked your interest by these short examples! I strongly encourage you to take a look at the language for yourself and notify me if I've missed something.

Here are some resources to read further up on:
Google Group
Gitter Chat
IRC
Subreddit
Newsletter

Ruby's GIL in a nutshell

Stanislav Kozlovski — Mon, 11 Sep 2017 07:12:20 +0000

Liquid syntax error: Unknown tag 'endcomment'

Managing RESTful URLs in Django Rest Framework

Stanislav Kozlovski — Mon, 14 Aug 2017 05:37:47 +0000

We've all been taught about RESTful API design. It does not take much to realize that these endpoints

POST /products/1/delete
POST /products/1/update
GET /products/1

are inferior to

DELETE /products/1
PUT /products/1
GET /products/1

You could also imagine that these multiple URLs per object stack up quickly and if we were to introduce more objects like /merchants/, /shops/ we'd quickly be managing a lot of URLs which can get confusing. Nobody wants to read a 100-line urls.py file.

But Django Rest Framework does not support mapping the same URL to different class-views based on the request method. How could we map this one URL with different methods in our urls.py file?

Let's first create the initial project with our bad URLs

Initial Project

We'll have a dead simple project which allows us to interact with product objects. We want to be able to update a product, get information about it and delete it.

models.py

from django.db import models
from rest_framework import serializers


class Product(models.Model):
    name = models.CharField(max_length=500)
    price = models.DecimalField(decimal_places=2, max_digits=5)
    stock = models.IntegerField()


class ProductSerializer(serializers.ModelSerializer):
    class Meta:
        model = Product
        fields = '__all__'

views.py

from rest_framework.generics import DestroyAPIView, UpdateAPIView, RetrieveAPIView

from restful_example.models import Product, ProductSerializer


class ProductDestroyView(DestroyAPIView):
    queryset = Product.objects.all()
    serializer_class = ProductSerializer


class ProductUpdateView(UpdateAPIView):
    queryset = Product.objects.all()
    serializer_class = ProductSerializer


class ProductDetailsView(RetrieveAPIView):
    queryset = Product.objects.all()
    serializer_class = ProductSerializer

and our ugly urls.py

from django.conf.urls import url
from django.contrib import admin
from restful_example.views import ProductDestroyView, ProductUpdateView, ProductDetailsView

urlpatterns = [
    url(r'^admin/', admin.site.urls),
    url(r'^products/(?P<pk>\d+)/delete$', ProductDestroyView.as_view()),
    url(r'^products/(?P<pk>\d+)/update$', ProductUpdateView.as_view()),
    url(r'^products/(?P<pk>\d+)$', ProductDetailsView.as_view()),
]

Wait, don't forget tests!

tests.py

from rest_framework.test import APITestCase

from restful_example.models import Product, ProductSerializer


class ProductTests(APITestCase):
    def test_can_get_product_details(self):
        product = Product.objects.create(name='Apple Watch', price=500, stock=3)
        response = self.client.get(f'/products/{product.id}')
        self.assertEqual(response.status_code, 200)
        self.assertEqual(response.data, ProductSerializer(instance=product).data)

    def test_can_delete_product(self):
        product = Product.objects.create(name='Apple Watch', price=500, stock=3)
        response = self.client.delete(f'/products/{product.id}/delete')
        self.assertEqual(response.status_code, 204)
        self.assertEqual(Product.objects.count(), 0)

    def test_can_update_product(self):
        product = Product.objects.create(name='Apple Watch', price=500, stock=3)
        response = self.client.patch(f'/products/{product.id}/update', data={'name': 'Samsung Watch'})
        product.refresh_from_db()
        self.assertEqual(response.status_code, 200)
        self.assertEqual(product.name, 'Samsung Watch')

Run our tests and you'd see that this works.

Becoming more RESTful

Okay, now it's time to fix our URLs into more sensible ones. As we said, we want all of these views to point to one exact URL and differ only by the method they allow.

The idea here is to have some sort of base view to which the request on the url gets sent to. This view will have the job to figure out which view should handle the request given its method and send it there.

class BaseManageView(APIView):
    """
    The base class for ManageViews
        A ManageView is a view which is used to dispatch the requests to the appropriate views
        This is done so that we can use one URL with different methods (GET, PUT, etc)
    """
    def dispatch(self, request, *args, **kwargs):
        if not hasattr(self, 'VIEWS_BY_METHOD'):
            raise Exception('VIEWS_BY_METHOD static dictionary variable must be defined on a ManageView class!')
        if request.method in self.VIEWS_BY_METHOD:
            return self.VIEWS_BY_METHOD[request.method]()(request, *args, **kwargs)

        return Response(status=405)

This simple class requires us to inherit it and define a class variable named VIEWS_BY_METHOD. A dictionary which will hold our method names and their appropriate handlers.
Using this base class, creating the ManageView class for our Product model is trivial:

class ProductManageView(BaseManageView):
    VIEWS_BY_METHOD = {
        'DELETE': ProductDestroyView.as_view,
        'GET': ProductDetailsView.as_view,
        'PUT': ProductUpdateView.as_view,
        'PATCH': ProductUpdateView.as_view
    }

It is worth mentioning here that any permission classes must be defined in the separate views and will not work if put in the ManageView

We need to quickly edit our urls.py's urlpatterns

urlpatterns = [
    url(r'^admin/', admin.site.urls),
    url(r'^products/(?P<pk>\d+)$', ProductManageView.as_view()),  # this now points to the manage view
]

Let's test this new view as well
tests.py

class ProductManageViewTests(APITestCase):
    def test_method_pairing(self):
        self.assertEqual(len(ProductManageView.VIEWS_BY_METHOD.keys()), 4)  # we only support 4 methods
        self.assertEqual(ProductManageView.VIEWS_BY_METHOD['DELETE'], ProductDestroyView.as_view)
        self.assertEqual(ProductManageView.VIEWS_BY_METHOD['GET'], ProductDetailsView.as_view)
        self.assertEqual(ProductManageView.VIEWS_BY_METHOD['PUT'], ProductUpdateView.as_view)
        self.assertEqual(ProductManageView.VIEWS_BY_METHOD['PATCH'], ProductUpdateView.as_view)

    def test_non_supported_method_returns_405(self):
        product = Product.objects.create(name='Apple Watch', price=500, stock=3)
        response = self.client.post(f'/products/{product.id}')
        self.assertEqual(response.status_code, 405)

We change the previous tests' urls to use the new one and we can see that they pass

Creating test database for alias 'default'...
...
----------------------------------------------------------------------
Ran 3 tests in 0.037s

OK
Destroying test database for alias 'default'...

And voila, we have the same functionality but in one url!

Summary

What we did was create a main view which dispatches requests to the appropriate views given the request method. I believe that this is the right way to handle multiple methods per URL in DRF when you want to have different class-based views handling each method.

This, however, is not the optimal case with our dead simple example. For views which are somewhat to extremely generic (like ours, no custom logic inside) or function views, there are Routers and ViewSets

Bonus - Routers and ViewSets

Since DRF is awesome, they provide us with the ability to define all the basic operations on a model in just a few lines.
With a ViewSet class, we can define our create, read, update and destroy logic without writing any code.

views.py

from rest_framework import viewsets
class ProductViewSet(viewsets.ModelViewSet):
    queryset = Product.objects.all()
    serializer_class = ProductSerializer

and we need to define the URL in our urls.py using a Router class

from rest_framework.routers import DefaultRouter
router = DefaultRouter(trailing_slash=False)
router.register(r'products', restful_views.ProductViewSet)

urlpatterns = [
    url(r'^admin/', admin.site.urls),
]

urlpatterns += router.urls

This viewset, believe it or not, has all the functionality we implemented above plus some additional like creating a Product(POST /products) or getting a list of all Products(GET /products)

ViewSets also allow you to override the views and create custom ones. This allows us to create function views for one URL but not class-based views.
They are absolutely amazing for our example and other simple projects, but sub-par for managing multiple class-based non-generic views with custom logic inside them.

From Zero to Hero (How I became a professional developer in a year)

Stanislav Kozlovski — Tue, 25 Jul 2017 05:39:50 +0000

I had always thought that I would have a hard time finding a job as a programmer. When starting out, I was easily stifled by various problems. Thoughts that this was not the right thing for me were frequent. I kept going though, as I did not have a better alternative.

Graduating from high school, I decided to ditch university in favor of coding bootcamps. This was just at a time when they were gaining popularity here in Bulgaria and they gave me hope I would be able to land a job quicker than going through the 4 years of university.
I had a couple of key reasons for not enrolling in an university

I did not want to waste 4 years for attaining something that could have taken less. Yes, of course those 4 years are not 'wasted' and everything learnt there sooner or later pays off in a direct or indirect(as the case with math) way.
I did not want to be a financial burden on my parents for 4 more years. I knew that I could not sufficiently learn if I had to work simultaneously, so getting a job was out of the question.
I was unsure if this was the right thing for me and I was scared of studying for years just to figure out I didn’t like it once I got a taste of the professional world

The Beginning

So I ventured out into the world of coding bootcamps, at first online, learning the very basics of programming. For this, I want to give a shout out to Software University (www.softuni.bg). Their introductory courses proved invaluable for a beginner like me. Two hours for a lecture on Arrays, live coding and numerous challenges on the material might seem like overkill now, but they proved to be immensely useful at the time. SoftUni has world-class introductory programming lessons and it’s a real shame that there are no english lectures, as the world would greatly benefit from them. Take it from somebody who tried to learn programming through other courses/tutorials, I have not been able to find an equally good beginner programming course online. Out of the others that I have seen, most go through the material too quickly and do not give the needed problems for the student to solve in order to fully grasp the material. Some do not even have homework!

Regardless, a couple of introductory courses are simply not enough. I quickly realized that one bootcamp was not going to cut it and at first decided that building up a portfolio to go alongside those certifications was the thing to do.
I started simple, building a RPG console game in Python. Of course, this was a dead-simple application but I did grow it to around 5k lines of code, which, at the time, were a lot for me.
This still was not enough and I ventured out to other courses (some online, some in-person). My workload grew and I found myself consistently spending 10+ hours to stay on top of everything. It felt good though, as I could actually see the progress.

We were now close to the new year and I had a team project to do on a course. Working with a team was not easy, it was especially hard to divide the work. Knowing me, I did most of it just because I wanted to(you learn by doing, after all) and nobody else had anything against that. This is where we created a really silly Facebook clone for pets (https://github.com/codeBusters-softuni/petbook)

This was cute

At this point, I felt reasonably confident in my skills and thought that landing an internship sometime in the new year was the next best step. It was time to surround myself with professionals if I wanted to continue growing.No offense, but most people in coding bootcamps are not that good. You can't blame the bootcamp though, it's just that this format attracts people who want the 'easy' way to get into an industry with high salaries, but I digress.
I decided that in order to get in a good company, I'd need to know a fair amount of Data Structures and Algorithms. No problem, there were plenty of resources on the topic.
Well, this material turned out to be the toughest yet. Suddenly, I was back to the feeling that maybe I’m not cut out for this. This all culminated when it took me two-three days to implement a Red-Black tree (the classical kind, none of these left-leaning garbage variants). This was around Christmas too and as such I feel that an obligatory xkcd is in place here:

On the upside, finally grasping a structure/algorithm was rewarding enough to keep me going.

Very quickly, me and a friend got addicted to the direct problem solving aspect of algorithms and started competing in online programming competitions. Obviously, we did not score that well, but it was fun and it incentivized further learning. I felt I made significant progress in my algorithmic thinking here and I encourage everybody to give competitive programming a go.
For the following months (December-March) I spent a fair amount of time solving problems, writing out data structures and learning algorithms.
In retrospect, I'm very glad I spent some time on those topics as they built resiliency, logical thinking and confidence in my skills. I actually continue to take online courses on algorithms (I recommend this one) and I will get back into competitive programming some day.

My 'trophies'

Inspired by the platform for algorithmic problems and competitions Hackerrank, I wanted to make something like that but with features I felt were missing. In February I started my third project - Deadline. This was initially planned as a hackathon project and while it won us tickets to the 2017 WebIt festival both me and my partner knew we would continue to develop this (and we still do actually, track its progress here: deadline)

Time to work

We were now around April and it was time to find a job. I felt confident I could land any internship, as I thought the main thing companies look for in an intern is their motivation and potential. I had plenty of motivation and around 8 months of non-stop Github activity to prove it.
From December I had wanted to take a shot at Uber even though it seemed infeasible (a year of bootcamps/courses and getting to work at a top technological company where some of the world’s most elite engineers are? yeah, right).
Well, against all odds, I did get an invite to an interview. I was ecstatic and had fun throughout the whole process. After it, I thought I had a fair chance of getting in, as I did my best to show my motivation, personality and ambitions. The technical questions were not hard at all and I felt I did well on them.
I did not get chosen. Not a big surprise, albeit disappointing nonetheless, I'm glad I strived for it, as it helped me become a better engineer.

As the things with Uber were over, it was now time to continue my search. I wanted to get in a company where the problems were interesting and I would find maximum growth as an engineer. My next choice was VMware, but their recruiting process was insanely slow and I decided I had enough of waiting. I started to apply to other companies and landed a position as a Junior Ruby Developer at SumUp. I had no previous experience with Ruby, but I made sure to quickly learn the basics and at this point I feel more than comfortable writing it.

Here we develop our own product and that is something I value in a company, I did not want to work in an outsourcing company.
I love my current job, the colleagues are great and even though I did not get to work at the place I initially wanted, I'm glad things turned out this way.
I look forward to a long career with them, trying to make the biggest impact I can. They trusted me and I will make sure to pay that off.

Conclusion

In the end, I feel I am better off this way due to my decision to skip university for the time being. I can now afford to enroll in uni due to desire and curiosity in the field, not out of seeming necessity. And if I do enroll, which I very well might, I will have a clearer vision on the material taught there and I will appreciate it more, as I would be able to more easily see its applications.

As a moral of the story, I want to emphasize that you do not have to follow the beaten path to become successful in any endeavor and sometimes that can even be the better choice.
Know that everything is possible with sufficient determination and work.

Hi, I'm Stanislav Kozlovski

Stanislav Kozlovski — Sun, 23 Jul 2017 13:07:39 +0000

I have been coding for 1.5 years.

You can find me on GitHub as Enether

I live in Sofia, Bulgaria.

I work for SumUp

I mostly program in these languages: Ruby, Python, Javascript.

I am currently learning more about everything.

Nice to meet you.