DEV Community: Daniel Orner

Building an Unsplash Slide Show on NeoCities

Daniel Orner — Fri, 07 Jul 2023 18:53:04 +0000

Unsplash is a great website that offers free images in a variety of formats and subjects. When we first went into lockdown in 2020, my home setup was two screens - my laptop, for actual work, and an external screen which is where my video calls took place, so I could still reference my work screen at the same time.

However, while a call wasn't taking place, my top screen just kind of sat there, blank. I wanted something to look at during thinking breaks. Basically I wanted a screen saver, but you can't get those going on just one screen.

Way back in the day, I had a number of personal pages on GeoCities. This revolutionary site allowed you to write your own HTML and have it hosted for free! Turns out, there's a modern version of this, called NeoCities. It allows you to create simple sites with static files that are hosted for free.

I used this post to make use of the source.unsplash.com endpoint to show a random, full-page image on my browser which I made full-screen on the external monitor. (I use Firefox for my browsing, so I just stuck Chrome up there for the slide show). I also made use of finicky so that all Google Meet links went to Chrome, while all other links stayed with Firefox.

Problem! A couple of weeks ago, Unsplash removed the source.unsplash.com endpoint. Instead, you need to use their API directly. Thankfully, the API is still free for low levels of usage, so I was able to modify the site to work with it.

The Design

Once you sign up, you'll get an API key. For these purposes, I didn't care too much about exposing it in a static site since I don't have any payment info on Unsplash and I'm not using it for anything else.

The overall design is as follows:

We want a "target" image div which will hold the image.
We will fetch 30 images (the max we can) from the API in the collection we're interested in.
We will loop through those images and show a new one once per minute.
Once we've exhausted the ones we have, we'll fetch more. This means we should in general only be making 2 API requests per hour.
If I don't like the image showing, I want an easy way to skip to the next one. I want to add a "refresh" button that will do that for me.

I'm going to keep this super simple. The only dependency I have is jQuery which I'm using for animations. You could easily do this with CSS animations, but I'm not as used to those.

The Implementation

First, here's the HTML:

<head>
<style>
* {
  box-sizing: border-box;
}

body {  margin: 0; background-color: black;}

.trigger {  position: fixed; top: 2rem; right: 2rem;}
  a { font-size: 2rem; color: white; transition: color 200ms ease; }
  a:hover { color: rgba(white, 0.7) }

#target {background-size: cover; background-position: center; background-repeat: no-repeat; height: 100%; width: 100%; }

</style>
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/material-design-iconic-font/2.2.0/css/material-design-iconic-font.min.css"></link>
<script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/3.2.1/jquery.min.js"></script>
</head>
<body>
<div class="trigger">
  <a style="cursor: pointer;" id="refresh"><i class="zmdi zmdi-refresh-alt"></i></a>
    </div>

    <div id="target"></div>
</body>

Now for the JavaScript.

// control the auto refresh rate -> 1000 = 1 second
window.refreshRate = 60000;
// you can change the collections ID to switch these out. Or you can use any other query from the API if you want truly random images
window.url = "https://api.unsplash.com/photos/random?w=1600&h=900&collections=879001&count=30&client_id=YOUR_API_KEY&orientation=landscape"
// use this to cache our data
window.jsonData = null;

// fetch the data, put it into the JSON blob, and pre-load the images from the data
async function fetchData() {
  const response = await fetch(window.url); 
  window.jsonData = await response.json();
  window.jsonData.forEach((image) => {
    const img = new Image();
    img.url = image.urls.full;
  });
}

// display the next image
async function nextImage() {
  const image = window.jsonData?.pop()
  if (image) { // we have more in the array
    const imageURL = image.urls.full
    $('#target').fadeOut(function() {
      $('#target').css({'background-image' : `url(${imageURL})`
    }).fadeIn(1500)
      });
  }
  else { // we've exhausted the array, go get more
    await fetchData();
    await nextImage();
  }
}

window.intervalTimer = setInterval(nextImage, refreshRate);
setTimeout(nextImage, 0);

$('#refresh').click(async function() {
  clearInterval(window.intervalTimer);
  await nextImage();
  window.intervalTimer = setInterval(nextImage, refreshRate);
});

// Hide the images on load
$('#target').hide();

Et voila! Hope someone out there finds this helpful!

Ruby through the lens of Go

Daniel Orner — Fri, 16 Apr 2021 18:33:23 +0000

I've been an ardent Rubyist for over eight years now. I've dabbled in other languages, but at this point Ruby is the thing I'm most comfortable in.

Over time, languages and programming styles change and so do the people who use them. While the power and elegance of Ruby still makes it my go-to for writing things, a recent foray into Go is making me rethink some of the things I've been taking for granted.

Why Go?

Go has been used at Flipp for some time now, although not widely in my team. I wanted to use Go to create a command-line executable, something that Ruby unfortunately isn't capable of doing. (There are options, such as ruby-packer, but it seems like a "heavy" solution and doesn't seem to fit the Ruby paradigm.)

Also, I was curious about Go as a whole, so I figured it was a great opportunity to learn more.

Language Philosophies

Both Ruby and Go have extremely strong philosophies underpinning them. The guiding principle of Ruby is programmer happiness. To achieve that aim, Ruby tries to be as flexible as possible, providing multiple ways of achieving the same thing in an effort to speak to each developer's preferred way of thinking. It supports lots of dynamism, allowing developers to "reopen" any class or module, or create whole new ways of interacting with code such as creating new DSLs.

I found it really interesting when reading up on Go that it claims to have a similar philosophy. In their FAQ, it says "[Go was] designed by thinking about what programmers do and how to make programming, at least the kind of programming we do, more effective, which means more fun."

Having said that, it's pretty clear that Go is more of a polemic against systems languages such as C. When they say "more fun", they mainly mean more fun than C.

Go's real philosophy is a kind of extremist minimalism. The language is brutally strict about features that are and aren't allowed, keeping the feature set to an absolute minimum. The reasoning behind this is mainly that the language's raisons d'être are all about performance, speed, and simplicity. New features that might make the code itself slower or more complex are kiboshed.

As an example of the philosophical difference, here's how you would find an element inside an array in Ruby:

arr = [1, 2, 3]
arr.index(2) # 1

And here's how you'd find it in Go:

arr := []int{1, 2, 3}
result := 0
for i, v := range(arr) {
  if (v == 2) {
    result = i
  }
}

Ruby, filtering out even numbers:

arr = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
filtered = arr.select(&:even?)

Go:

arr := []int{1, 2, 3, 4, 5, 6, 7, 8, 9, 10}
filtered := int[]{}
for i, v := range(arr) {
  if (v % 2 == 0) {
    filtered = append(filtered, v)
  }
}

However, language design is only part of the story behind a language. Just as important is the ecosystem: the reusable code modules/packages, the people that manage them, which ones are "popular", what "best practices" are, etc.

One of the things I love the most about Ruby is that it tends to coalesce around one or two really popular libraries. Rails is the big one obviously, but over time you see libraries designed for a particular purpose "winning" over other things. This includes things like linting/code analysis (Rubocop), authentication (Devise), testing (RSpec and Minitest) and more. The emphasis is on making something good great rather than making a lot of different good things.

In Go, it seems like packages themselves are nowhere near as popular as the language, compared to what I see elsewhere. I haven't been in the ecosystem long, but it seems to be rare that articles about Go coalesce on a particular solution similar to Rails or something like lodash or express in the JavaScript world. Instead, the main feeling when asked "How do I do X" in Go seems to be "do it yourself". The kind of thing that in Ruby would be derided as "plumbing" is "real code" in Go; the goal is to make you think about every single line you write.

Go's lack of generics make it difficult to make an all-purpose utility library to make these sorts of things easier, so maybe once they land in a year or two the rawness of the language could be mitigated. But for now, Go and Ruby feel like as far apart as they could be.

What can we learn?

The strictness of Go means that there likely aren't too many things that we could import into it from Ruby and similar languages. And nor should we; one of the refrains of the Go philosophy is that languages should be different rather than similar, and Go is outspoken in its adherence to its own unique identity.

However, I think that some of the surprising advantages I found in Go can be moved over to Ruby without much work other than a slight shift in mindset. (There are other amazing things Go provides that Ruby simply can't do, and in fact might start affecting the beautiful things about it - this post is not about all the nice things in Go but the things that we can do with our Ruby code to gain some of the same advantages.)

Keep code and data separate

Here is how you define a method on a type in Go. (This example intentionally has a code smell, where I'm sending an e-mail from something that shouldn't be able to or care about it.)

type Product struct {
  Id string
}

func (product *Product) sendEmail(customerEmail string) {
  sendProductEmail(product.Id, customerEmail)
}

// ...

product = Product{Id: "some-product-id"}
product.sendEmail("me@example.com")

The usage of the method looks the way we expect it, but wait... something's missing from the function body. Where's the this?

The answer is that there isn't one. Not because the idea doesn't exist (it does - it's called a receiver and Ruby has the same idea) but because the syntax calls out a very important fact, which is that methods and data do not need to live together in an object.

To make this abundantly clear, let's realize that we made a stupid mistake and move the sendEmail function from the product to a top-level function:

func sendEmail(product *Product, customerEmail email) {
  sendProductEmail(product.Id, customerEmail)
}

// ...

product = Product{Id: "some-product-id"}
sendEmail(product, "me@example.com")

What's changed in our function body?

The answer: Absolutely nothing. The definition and usage changed, but the function body is identical. This is a very strong hint that methods work on data. And whether the methods and data "live together" in an object or not, the exact same thing is happening.

Keeping methods out of objects helps to reduce the urge to introduce side effects. And that means you can often model your functions as pure input/output, which has a whole whack of advantages.

As an example, take a Ruby implementation of the above.

class Product
  attr_accessor :id
  def send_email(customer_email)
    send_product_email(@id, customer_email)
  end
end

Let's do the same refactor now:

class Product
  attr_accessor :id
end

def send_email(product, customer_email)
  send_product_email(product.id, customer_email)
end

We've had to change our function body to accommodate this now.

So how can we achieve the same flexibility in Ruby? Simple. Treat data as data and methods as methods. Prefer static methods that work on data. This has the added advantage of enabling you to "pipe" data more easily without having to fall back to solutions I've always found odd like tap and yield_self.

class Product
  attr_accessor :id
  def self.send_email(product, email)
    send_product_email(product.id, email)
  end
end

Product.send_email(my_product, 'me@example.com')

The functions stay in the class definition (Go doesn't have class functions, which IMO makes it a bit harder to organize) but are static and don't use or change any object state.

For example, let's say we've decided to move this function to a new module which deals with all the e-mail sending. The function doesn't change at all any more - in fact, even the pattern of calling it stays the same. The only thing that changes is the class name.

class Product
  attr_accessor :id
end

class EmailService
  def self.send_email(product, email)
    send_product_email(product.id, email)
  end
end

EmailService.send_email(my_product, 'me@example.com')

Obviously this won't work 100% of the time, but I've found this pattern to be much easier to work with.

Use types and not hashes

Go is a statically typed language, in its own maverick way. It doesn't support inheritance, and mainly works through structs and interfaces (which is a great feature that Ruby can approximate, much less elegantly, via "duck-typing" and respond_to? calls).

Ruby is dynamically typed, and in a typical Ruby project you'll see classes, OpenStructs (a class that basically makes a hash "pretend" to be an object by using dot-notation), and other solutions, but in many cases you're using hashes to store your data.

There are some real benefits to static typing, not the least that it makes IDE's much smarter at telling you when you're doing something wrong. Even better, types implicitly document what your data looks like much better than hashes.

Ruby is slowly starting to drift towards better type functionality, with first Sorbet and then RBS. But you don't need any static typing tool in your own code to do things smarter.

It's not like Ruby doesn't already support structs! Check the difference here.

# @param json [Hash]
def create_dashboard(json={})
  DashboardCreator.call(
    name: json[:name],
    graphs: json[:graphs],
    owner: json[:owner]
  )
end

In my opinion, it doesn't matter if DashboardCreator.call is defined with keyword arguments or an options hash. The fact that the input json is a hash means that your code has no idea if, for example, :name is a valid key into your JSON.

Hashes are maps. Maps are key-value pairs. They're meant for lookups. For some reason, Ruby has latched onto this idea that they should be used as options, keywords, or attributes. This is confusing and error-prone.

Now here's the same thing with structs.

# @attr graphs [Array<Graph>]
# @attr name [String]
# @attr owner [String]
Dashboard = Struct.new(:graphs, :name, :owner)

# @param dashboard [Dashboard]
def create_dashboard(dashboard)
  DashboardCreator.call(dashboard)
end

Note how I've packaged up all the data into a single object which is "strongly typed", or as strong as Ruby can get for now. We (and our IDEs) now know absolutely what that object looks like.

Now there might be cases where all we've done here is push up the error cases where our parsed JSON doesn't match what we expect. For example, we might need a method that looks like this:

# @param json [Hash]
# @return [Dashboard]
def parse_dashboard(json)
  raise "No graphs!" if json[:graphs].nil? || 
   !json[:graphs].respond_to?(:size)

  Dashboard.new(
    name: json[:name],
    graphs: json[:graphs],
    owner: json[:owner],
    keyword_init: true
  )
end

The thing is, we've moved the error into a method whose only job is to parse JSON. We can do further validations here. But importantly, we don't encode our expectations about JSON into all the downstream functions. Only one thing cares about JSON itself. Everything else cares about dashboards.

Simplify, simplify

Ruby's dynamism allows for some truly concise and beautiful-looking code. But like always, you need to make sure your toolbox is using the right tools for the right situations.

When you're building a framework, using the more advanced metaprogramming features like define_method and method_missing can work well and provide a useful interface. But using it in your code can cause a world of pain.

Just because you can do something in Ruby doesn't mean you must do it. I've moved over to preferring code generation over code reflection. In other words, using generators to create the code you care about rather than deciding what your code should do at runtime.

For example, let's say you have a web parser that has to fetch a number of item fields by using different XPath or CSS selectors. We want to define a parent class that defines all these fields and child classes that separately specify how to fetch them for different websites. Here are a couple of different ways to achieve this.

First, how we want to use this code:

Item = Struct.new(:name, :description, :price)

class Site1Parser < Parser

  def initialize(item)
    @item = item
  end

  def name(browser)
    browser.xpath("/some/path/to/name")
  end
end

parser = Site1Parser.new
parser.item.name # the value parsed from the site

Way number 1 uses method_missing to set attributes to an item:

class Parser

  attr_accessor :item

  def parse
    browser = Browser.new
    Item.members.each { |f| self.send(f, browser) }
  end

  def method_missing(method, *args, **kwargs, &block)
    if item.respond_to?(method)
      item.send("#{method}=", block.call(args[0]))
      return
    end
    super
  end
end

Option 2 goes a bit more explicit by actually defining methods using define_method:

class Parser
  attr_accessor :item

  def parse
    browser = Browser.new
    Item.members.each { |f| self.send(f, browser) }
  end

  Item.members.each do |field|
    define_method(field) do |browser|
      raise "Not implemented yet!"
    end
  end
end

Leaving alone the possible debugger confusion, when we humans are looking at either of these options, we need a lot of information in addition to the body of the method. What are valid values for method or field? What possible arguments could be passed? What does self mean at this point?

What if instead, we created a generator which created code for you? Something that might look like this?

class Parser

  attr_accessor :item

  def parse
    browser = Browser.new
    self.name(browser)
    self.description(browser)
    self.price(browser)
  end

  def name(browser)
    raise "Not implemented yet!"
  end

  def description(browser)
    raise "Not implemented yet!"
  end

  def price(browser)
    raise "Not implemented yet!"
  end
end

In this case, rather than writing all this code yourself, you rely on the generator to do it for you. Some benefits of this approach:

The code is much easier to understand, because you can actually look at each method and see exactly what it calls, without having to step through it or check to see what Item.members returns.
Your codebase becomes searchable, both by you and by code analysis tools. Your function calls are explicit and you can logically step through them without confusion.
You can more easily override specific methods as necessary when they're already there for you to do so. This is the kind of thing object oriented design is really good at. And you can easily tell what methods you have to override.

Obviously you run into other problems on the framework side (when you add a field to Item, how do you make sure you can update the code without blowing away anything you've changed in it?) which might involve more advanced abstract syntax tree munging. But I'll leave this here as a thought experiment, nevertheless.

Conclusion

Both Ruby and Go are amazing languages in their own way, and both provide numerous benefits in different directions. There are definitely things I wish I could change about Go, but the nice thing about Ruby is that I know I can change some things about it all on my own!

Your Programming Toolbox: Functional and Object-Oriented Paradigms

Daniel Orner — Fri, 27 Nov 2020 14:31:55 +0000

Object-oriented (OO) and functional programming (FP) are the two most popular programming paradigms in use today. Most languages provide at least some features from one or both of them. Although I don't intend to dive into what exactly OO and FP are, as there are many, many articles already written that attempt to do this, I do want to try to weigh in on a corollary argument: "Which one's better?"

There have been countless keystrokes shed in pursuit of this age-old question. For example, we've got Ilya Suzdalnitzky proclaiming the end of OO as we know it, and a number of other articles passionately arguing the opposite.

This combat area has been re-tread so often, it's practically a no-go zone. The most thoughtful articles on this are can be distilled into the following insights:

Any paradigm can be done well, or badly.
Both FP and OO have advantages in certain cases and disadvantages in others.
Both FP and OO have wide-ranging definitions, and depending on what you're doing and in what language, you may think you're doing one or the other but someone else may disagree, or maybe you're doing a mishmash of them both.

As time goes on, it becomes clear that many popular languages are diving deeper into becoming multi-paradigm - they allow you to write in procedural, OO or FP and you can pick and choose your language features.

Because of this, you can take bits and pieces of these different approaches and try to synthesize them into the best of both worlds. I'd like to touch on my thoughts on what the good pieces of the two approaches are, and when to use the rest.

The Human Touch

One of the major differences between OO and FP is in the visceral, intellectual way that people see code. In OO "everything is an object"; in FP "everything is a function". Of course, neither of those statements is entirely true. But at a high level, people who are more mathematically inclined seem to enjoy thinking of code as functions, while those less so find it more intuitive to think of code as objects.

I personally fall into the latter camp - my first task in any programming problem is to think of the "things" and after that, what has to happen with those "things". Because of this, my starting approach is OO, and then I start thinking of how I can use aspects of FP to improve some of the shortcomings of it.

How functional?

In my opinion, the power of FP comes from thinking about data as immutable and functions as pure (or pure-ish). The main benefits are testability and confidence. You can write your tests knowing that given a particular input, you will always get the desired output, since you don't need to worry about the state of the object you're calling it on.

Having said that, I do not advocate a complete FP approach unless your whole team is onboard with that, because then you have to wade into the waters of monads and functors and higher-level functions and functional math (ew, math). I think this article, written about JavaScript, gives a really good overview of how to take the good parts of FP without being slavish.

My take on pure functions is more along the lines of: "Your function should either be pure, or accomplish something in the real world, but not both."

The Nitty Gritty

I'll be using Ruby in my examples, as it's the multi-paradigm language I'm most comfortable with, but it's almost always possible to convert these examples to JavaScript, Java, Python, etc.

One of my earlier revelations about FP and OO is that from a technical perspective, they are practically interchangable. If you think of your object as data, and the functions/methods as acting on that data, then the following two chunks of code are equivalent - except that the first is in OO-land and the second is in FP-land.

Person = Struct.new(:name, :status) do
  def marry!
    @status = "married"
  end
end
person = Person.new("Daniel Orner")
person.marry!
person.status # married

Person = Struct.new(:name, :status)
def marry(person)
  Person.new(person.name, "married")
end

marry(Person.new("Daniel Orner")).status # married

They accomplish the same thing - except that in OO-land, the function lives with the data and changes the data, while in FP-land, the function lives outside the data and returns a new set of data.

Most important things that can be achieved by one paradigm can be achieved by the other as well, as I'll explore below.

Code Organization

One use of classes is a way to organize your methods. If all your methods dealing with a person are in the Person class, it's easy to parse out where to find the thing you're looking for.

FP has a corresponding way to do this as well, although it's more language-specific. In JavaScript, you might put all your functions in a single file. In other languages, you might use packages, namespaces, modules, etc. and ensure that you use "static" or "module-level" functions.

## OO-land
class Person
  attr_accessor :first_name, :last_name, :address
  def move_to(address) # ...
  def marry(person2) # ...
end

## FP-land
class Person
  attr_accessor :first_name, :last_name, :address
end

# person_operations.rb
module PersonOperations
  def self.move_to(person, address) # ...
  def self.marry(person, person2) # ...
end

This leaves the Person class as a value object - it's more of a type than a class - and all operations live outside it.

Encapsulation

With OO, you can have private state which isn't exposed, and you only expose essentially an API for external code to call. How could you do that with FP?

The short answer is, not easily. You can use closures to achieve some level of encapsulation, but it will largely be on a global level rather than encapsulating data in an individual object. This is one of the weaknesses of FP.

However, the bigger question is why you are encapsulating data in the first place. If you're using FP, your functions are pure and your data is immutable - so you can't have anything accidentally or maliciously change your data.

The main reason I see to encapsulate data is as a form of documentation: This class has ten fields, but only five of them are relevant to anything outside the methods in this module that do these things to it. (Honestly, once you've even had this thought, it usually turns me to "do these five fields really belong with this data, or should they be somewhere else?")

Composition

One of the basic tenets of functional programming is using composition to combine functions together. For example, here's a way to take two functions and combine them into one that does both things:

add_one = ->(x) { x + 1 }
double = ->(x) { x * 2 }
double_and_add_one = add_one << double
double_and_add_one.call(5) # 11

In OO-land, composition means modeling things as a "has-a" relationship instead of an "is-a" relationship. For example, instead of treating an Employee as a subclass of Person, you might instead give your employee an attribute of type Person. It's not as intuitive, but it works just as well while avoiding some of the pitfalls of inheritance.

class Person

  attr_accessor :first_name, :last_name

  def initialize(:first_name, :last_name)
    @first_name = first_name
    @last_name = last_name
  end

  def full_name
    "#{@first_name} #{@last_name}"
  end

end

class Employee
  extend Forwardable
  attr_accessor :person

  def initialize(first_name, last_name)
    @person = Person.new(first_name, last_name)
  end

  def_delegators :@person, :full_name

When to use what?

In general, you can use what I feel are the "important parts" of FP easily within an OO framework, but not so much the other way around. For example, I like the idea of value objects. My sweet spot is leaving all functions related to reading or writing attributes in the class, but anything bigger than that outside of it.

But OO has its advantages as well. When should you:

Have methods live with your data
Provide an inheritance structure
Allow overriding of "super" methods

I've found two really good cases for these:

Framework code
Value object hierarchy

In all other cases, I'd put a big question mark on it.

Framework code

When I use the term "framework", I mean you are defining a way to accomplish a particular task without actually doing that task - generally when you have to have many slightly different ways of doing the same thing. Some examples from my past include:

A web scraper that saves data to a particular schema/model
A producer that takes data and maps it to Kafka topic
A CSV parser that needs to be able to handle different formats for different clients

In all these cases, you want to have some "base behavior" defined which you can easily use, but you want to be able to override bits and pieces, and provide helper methods.

Using a pure FP approach in this case is definitely possible, but a lot of work, because the "overriding" piece needs to happen via the use of composition and hooks.

Here's a sample parser (without showing method implementation) that takes in a CSV file and outputs a structured array to save to the database, in OO-land:

class Parser
  def parse(filename) # ...
  def read_file(filename) # ...
  def parse_headers(header_row) # ...
  def parse_row(row) # ...
  def clean_up_data(structured_array) # ...
  def results # ...
end

Using this framework, you can write a parser only by overriding the two methods that every parser needs to write, parse_row and parse_headers. The other functionality "just works" since it uses the parent code.

In FP-world, you'd need to implement it similar to this:

  module Parser
    def self.parse(filename, parse_header, parse_row)
    def self.read_file(filename) # 
    def self.clean_up_data(structured_array) # ...
  end

You'd need to pass in the parse_header and parse_row functions to the parse method to be able to override this behavior.

This example doesn't look too bad, but what if there are additional steps you want to add functionality to? Inheritance, and particularly the super keyword, allows you to easily "wrap" a function around specialized code without having to do anything special on the parent class:

  class MyParser < Parser
    def clean_up_data(row)
      new_row = handle_special_characters(row)
      super(new_row)
  end

In FP-land, you'd need to do something like this:

module Parser
  def self.parse(filename, parse_row, parse_header, clean_up_data)
    ...
    if clean_up_data
      row = clean_up_data(row)
    end
    internal_clean_up_data(row)
end

And if you want to "wrap" the original behavior, you'd need to grab a reference to the function on the module that does it and call it:

  def clean_up_data(row)
    my_row = handle_special_characters(row)
    Parser.method(:clean_up_data).call(my_row)
  end

  Parser.parse("my_file", my_parse_func, my_parse_header_func, clean_up_data)

As your framework grows in functionality and usefulness, you'll find more and more of these overrides are necessary as the use cases expand. If you don't want to build them all into your own code, you need to allow people to make changes as necessary, and OO allows a much cleaner way to do that.

Inheritance in Value Objects

I think there is also space for OO-style inheritance when dealing with your data, as long as you can absolutely model that data as an "is-a" relationship. The main difference is that your classes are things that are, not things that do. You can do this with a Service, Employee, House, etc. - not with a Parser, Manager, or DatabaseLayer.

For example, if a user has a name, e-mail address and login, and so does an employee, it makes sense to have Employee inherit from User since you get its fields for free, and you can safely say that a) every Employee is a user, and b) Employee would never change the behavior defined in User in such a way to violate the Liskov Substitution Principle.

You can also freely use interfaces or abstract classes in languages that support it, and approximate them in languages that don't (e.g. defining a class in Ruby where every method consists of a body that raises a NotImplementedError).

However, I'd consider it a code smell going past that in non-framework code.

Final Thoughts

When I was first introduced to FP a few years ago, I had a hard time wrapping my head around it because so much of the discussion around it felt so academic. As I've come to understand the real-world benefits of using it, I'm more able to reap those benefits without entirely overhauling how I work. I'm still feeling my way towards the golden road of combining the features of these paradigms in the right ways in the right cases.

Hope this is helpful to others out there as well!

Code Reuse in RSpec

Daniel Orner — Fri, 20 Nov 2020 14:41:10 +0000

When writing your tests with RSpec, you have a plethora of ways to help DRY up your code and keep it clean and reusable. So many ways, in fact, that it can be pretty confusing to keep track of when you should do what. Forthwith, a quick reference for you!

Contexts

First, an explanation of what I mean by "context". In RSpec-world, a context basically refers to setup code and methods that are available to the current example. This includes:

before, after, and around hooks
let and let! declarations
subject declarations
helper methods

Nested Groups

The simplest way of reusing code is when your tests all belong to the same file. You can reuse your context simply by nesting your tests within the same group. Remember that you declare an example group with the describe or context method.

Here's an example.

RSpec.describe Taxi do
  let(:driver) { Driver.new(name: 'John' }
  let(:taxi) { Taxi.new }

  before(:each) do
    taxi.driver = driver
  end

  it 'should take a fare' do
    taxi.put_fare(50)
    expect(driver.income).to eq(50)
  end

  it "should add a drive to the driver's history" do
    taxi.start_drive("123 Main St.")
    expect(driver.history.last).to eq("123 Main St.")
  end

  it 'should get an address to drive to' do
    taxi.address = "123 Main St."
    expect(taxi.next_drive.address).to eq("123 Main St.")
  end

end

You'll notice that all three examples in the group can make use of the taxi and driver declarations, since it's declared at the group level.

But note that one of those examples doesn't actually need a reference to the driver, and doesn't need the setup code that assigns it! This is a contrived example, but you can easily spend hundreds of lines setting up data in your tests, and you want that data to be targeted so you're not re-creating data where you don't need it. This is where you can nest your tests a little deeper.

RSpec.describe Taxi do
  let(:taxi) { Taxi.new }

  it 'should get an address to drive to' do
    taxi.address = "123 Main St."
    expect(taxi.next_drive.address).to eq("123 Main St.")
  end

  RSpec.context 'with driver' do
    let(:driver) { Driver.create!(name: 'John' }
    before(:each) do
      taxi.driver = driver
    end

    it 'should take a fare' do
      taxi.put_fare(50)
      expect(driver.income).to eq(50)
    end

  end
end

Here, only the tests that need a driver have access to one, but all tests still have access to the taxi.

Shared Context

This is all well and good when testing a single class or feature. But what about when you're testing a category of related features? This is when shared contexts can be useful.

Shared contexts are a way to define a reusable context that can be used across files, including declarations, hooks and even helper methods. Let's say that we not only care about taxis, but about all kinds of vehicles.

First you define your shared context in a file that's included by all tests. If you don't have many of these contexts, you can put them inside your main spec_helper file. If you have more, you can separate them out into its own file and manually include it from your test files.

# file: /spec/vehicles/shared_setup.rb
RSpec.shared_context("with driver") do
  def drive_vehicle(vehicle)
    vehicle.driver = driver
    vehicle.drive!
  end

  let(:driver) { Driver.new("John") }

  before(:each) do
    driver.wake_up!
  end

end

# file: /spec/vehicles/car_spec.rb
require_relative "./shared_setup.rb"

RSpec.describe(Car) do
  include_context("with driver")

  let(:car) { Car.new(driver) }

  it "should drive" do
    # this is true because wake_up! was called in the before hook
    expect(driver).to be_awake 
    drive_vehicle(car)
    expect(driver.is_driving).to eq(true)
  end

end

Here, you've got access to the driver and your helper methods from your shared context in this file, and you can include that context in any code that needs a driver.

Shared Examples

Shared contexts help to create data or behavior for your tests. Shared examples actually are tests which you can run in multiple places.

Why would you do this? A classic reason is inheritance (or composition, mixins, or any other way you have to reuse your own code). If you have a number of classes or modules that all behave a certain way, you can write a single set of examples that should apply to all of them and ensure that they all pass.

Here's an example:

# /spec/vehicles/shared_setup.rb
RSpec.shared_examples("a vehicle") do

  it 'should be able to move' do
    subject.move("123 Main St.")
    expect(subject.address).to eq("123 Main St.")
  end

end

# /spec/vehicles/car_spec.rb
require_relative "./shared_setup.rb"

RSpec.describe(Car) do

  it_should_behave_like "a vehicle"

  it ... # more tests only for cars
end

When running the suite, all the shared examples will run along with the specific examples. Shared examples can make use of the current context, so for example you can use let or subject declarations to act as parameters, or you can pass parameters directly to shared examples.

When?

Here's a rubric as to when to use these three features:

Factories

This is more specific to ActiveRecord, but factories are a way to DRY up your data creation! Factories have an advantage over fixtures in that you aren't limited to specific, hand-crafted records but can define how to create data.

The most common way to use factories is to include the FactoryBot gem. Once set up, you can use it to manage your data creation.

As an example, let's say you can create taxis that are yellow, or that are New York taxis, or that are out of service. FactoryBot allows you to create traits to reuse this behavior:

# /spec/factories/taxis.rb
FactoryBot.define(:taxi) do
  trait :yellow do
    color { "yellow" }
  end

  trait :new_york do
    city { 'New York' }
  end

  trait :out_of_service do
    state { 'OUT_OF_SERVICE' }
  end

end

# /spec/taxi_spec.rb
RSpec.describe(Taxi) do

  it 'should have the right attributes'
    taxi = FactoryBot.create(:taxi, :new_york, 
      :yellow, :out_of_service)
    expect(taxi.color).to eq('yellow')
    expect(taxi.city).to eq('New York')
    expect(taxi.state).to eq('OUT_OF_SERVICE')
  end

end

When should you create a factory vs. just creating your data manually? Some good rules of thumb are:

When you have tests that don't need to understand the underlying data model but need to create the data
When you've identified a pattern of data creation that's called from multiple places that you can simplify
When you want to encapsulate how your data looks in a simpler, centralized way

RSpec is powerful but can be confusing, so hopefully this explains the different features in an organized way. Happy coding!

Linting only changed files with GitHub Actions

Daniel Orner — Mon, 24 Feb 2020 16:29:23 +0000

In our team at Flipp, like many companies, we have a venerable and enormous Rails monolith which is feeling its age. We've started the long process of breaking it up, but in the meantime there are hundreds of thousands of lines of code which are changed piecemeal across dozens of features.

We introduced Rubocop, the Ruby linter, a few years ago, and very slowly started updating the code as we changed it. We used a pre-commit Git hook to ensure that our developers couldn't even commit code that didn't pass the linter. We used a modified version of dirty_cop - at the time this was just a gist we found online - which only runs the linter on modified files.

As time went on, this solution started to become less and less ideal. Our codebase was so big and so old that just starting the linter could take upwards of 20 seconds. We took on new team members, not all of whom installed the pre-commit hook; other teams began submitting pull requests as well. This just wasn't a great solution to ensure we had a clean codebase with a consistent style.

The Project

Our first instinct was to move the lint check to our CI tool, CircleCI. It's really powerful, but there was one big problem - CircleCI runs on pushes, not pull requests. It would have no way of knowing what files changed in the PR.

Finally, we started investigating GitHub Actions. This seemed like it ticked all the boxes - it's a CI tool that can be set to run on pull requests, meaning it would be possible to detect which files were changed in that PR. Even better, because it runs within the GitHub ecosystem, it would be relatively simple to add comments or even commits to the PR itself fixing or commenting on the issues that were found.

Luckily enough, we found two existing actions that looked right up our alley:

file-changes-action outputs the files changed during a PR.
lint-action runs a linter (it supports many languages, not just Ruby) and does auto-fixing and commenting as well.

On with the show!

The Solution



name: Lint
on: [pull_request]

jobs:
  build:
    name: Lint
    runs-on: ubuntu-latest
    steps:
    - name: Checkout
      uses: actions/checkout@v2
    - name: Setup
      uses: actions/setup-ruby@v1
      with:
        ruby-version: '2.4'
    - name: Cache gems
      uses: actions/cache@v1
      with:
        path: vendor/bundle
        key: ${{ runner.os }}-gems-${{ env.cache-name }}-${{ hashFiles('**/Gemfile.lock') }}
        restore-keys: |
          ${{ runner.os }}-gems-

    - name: Bundle install
      run: |
        bundle config our_gem_repo.com our_username:${{ secrets.GEM_REPO_PASSWORD }}
        bundle config set without 'production staging'
        bundle config path vendor/bundle
        bundle install
        git checkout Gemfile.lock

    - name: Get file changes
      id: get_file_changes
      uses: dorner/file-changes-action@v1.2.0
      with:
        githubToken: ${{ secrets.GITHUB_TOKEN }}
        plaintext: true
    - name: Echo file changes
      run: |
        echo Changed files: ${{ steps.get_file_changes.outputs.files }}
    - name: Run lint
      uses: dorner/lint-action@v1.3.3
      with:
        github_token: ${{ secrets.GITHUB_TOKEN }}
        rubocop_bundler: true
        auto_fix: true
        rubocop_bundler_args: -R --fail-level C ${{ steps.get_file_changes.outputs.files}}

The Issues

When developing this, we ran across a number of issues we were able to solve:

Our gems are hosted on a private repo. To enable access to them, we had to store a secret in the repo with our credentials.
Our Gemfile.lock uses an old version of Bundler (1.17). Newer versions of Bundler automatically update the Gemfile.lock to indicate that it was bundled with that version - which means older Bundlers will subsequently fail. This is fine if you're not doing anything with that file, but the auto_fix option on the linter means it may commit the contents of the workspace back to the repo, along with the modified Gemfile.lock. To solve this, we had to use git checkout to keep the file pristine before continuing.
The file-changes-action outputs the files as a JSON array of strings. This isn't so convenient to pass into a shell command, so I made a fork of it and added the plaintext option. The great thing about GitHub Actions is that literally any repo can be an action - as long as it's tagged, I didn't have to wait for a PR to be accepted. I just switched the source of the action to my fork and went on my merry way.
The lint-action Rubocop linter had two issues that needed to be fixed:
- We use a shared gem on our private repo to share our styles across our projects. The action doesn't work with bundle exec. Since we had to use rubocop in the context of the bundle to use our shared gem, I needed to create a brand new linter called rubocop_bundler just to add the bundle exec before the command.
- The current Rubocop linter adds "." to the end of the command. This is not only unnecessary but forces the linter to run on all files, even if we pass a list of files to it. My fork removed the period and all works well.

The action forks fixed all the problems, so now this workflow file:

Caches the gems, so it almost always runs in less than 1 minute.
Records all warnings and errors as comments in the PR.
Auto-fixes fixable warnings with an additional commit in the PR.
Adds a Check to the Checks pane where we can inspect the offenses in a nice UI.

This was done with Ruby, but you can use the linter for any language you like!

Addendum: I've added PRs to both the abovementioned actions, and have gotten a response, so this functionality should be available in the base repos soon, although it might look a bit different. (In addition, see this issue where it mentions that some of the annotations might be clobbered in some cases - we felt this was acceptable.)

Update March 30: My PRs and issues have been released! The new way to do this (using the official action repos) is as follows:



    - name: Get file changes
      id: get_file_changes
      uses: trilom/file-changes-action@v1.2.3
      with:
        output: ' '
    - name: Echo file changes
      run: |
        echo Changed files: ${{ steps.get_file_changes.outputs.files }}

    - name: Run lint
      uses: samuelmeuli/lint-action@v1.4.0
      with:
        github_token: ${{ secrets.GITHUB_TOKEN }}
        rubocop: true
        auto_fix: true
        rubocop_args: -R --fail-level C --display-only-fail-level-offenses ${{ steps.get_file_changes.outputs.files}}
        rubocop_command_prefix: bundle exec

Introducing Deimos: Using Kafka as the Data Backbone for your Architecture

Daniel Orner — Fri, 22 Nov 2019 18:18:46 +0000

In the previous article, I detailed the microservice pattern of using Kafka as a data backbone for your architecture — Kafka acts as the source of truth and each microservice consumes data from it. This allows each service to be independent and to own only the data it cares about. We also said that there should be some way to marry traditional relational databases like MySQL and Postgres with the Kafka data backbone with a minimum of fuss.

We left off with a couple of problems:

Ensuring that downstream systems are not tied to your internal data schemas;
Fixing the transaction problem, where due to errors data can be written to Kafka or your database but not both.

Ruby on Rails — the Database Experts

Here at Flipp, we use Ruby on Rails heavily for systems that talk to databases. Rails’s ActiveRecord package has been in active development for over 13 years, covers a wide range of features and has been battle-tested in thousands of applications. Rails allows us to talk to our database in a consistent way and abstracts most of the underlying details away for us.

When we started widely using Kafka in our engineering org, we quickly realized we needed a way to:

Save data from Kafka to our database
Write data to Kafka — either updating state or sending events
Encode/decode data using Avro and the Confluent Schema Registry (a free centralized hub for all our Avro schemas)
Standardize logging and tracing for all our consumers

Our first attempt to send data across systems was to use Kafka Connect. We ran into a number of problems with it, some of which were detailed in the previous article. We decided it was more prudent for us to strike our own way to solve this problem.

We started off with a shared library we called FlippRubyKafka in late 2017. Over the last two years, this library has grown, added features and fixed bugs and has powered over a dozen of our core microservices. We are happy to announce that this project, now called Deimos, is now open source!

Deimos Features

Deimos is firmly designed to cut down as much as possible on boilerplate and plumbing code. It is focused entirely on the use case of Kafka as a data backbone with Avro composing the structure of the data.

Deimos is built on a previous, smaller library called Phobos (hence the name), which in turn is built on RubyKafka, a pure Ruby implementation of the Kafka client.

Note that Deimos assumes that you have a copy of the Confluent Schema Registry running. If you’ve already got Kafka and Zookeeper up then all you need to do is deploy the Docker container to the cloud provider of your choice and get a reference to its URL.

Configuring Producers and Consumers

You can use Deimos simply as a way to tie Avro and Kafka together, and it works really well even if you’re not using Rails! Both producers and consumers allow your business logic to deal exclusively with Ruby hashes — they will automatically be encoded to and decoded from Avro via the schema registry before interacting with Kafka.

Because Avro is a central tenet of Deimos, all producers and consumers must specify which Avro schema and namespace they will use while consuming or producing. In addition, they need to specify how they handle keys for these messages:

The default, and recommended, way to handle keys is to Avro-encode them. This allows for any downstream systems that use static types (e.g. those written in Java or Scala, which have a large ecosystem related to Kafka) to easily decode them.
You can also indicate that your topic has no keys (e.g. event topics should not have keys as you want to keep all events, not just the last updated state).
Finally, you can leave your keys as plaintext, although this is not recommended.

When using Avro-encoded keys, you can provide a separate key schema, or you can have Deimos auto-magically create one for you from your value schema, by specifying a field (usually “id”) to extract from it.

Sample producer:

    class MyProducer < Deimos::Producer

      namespace 'com.deimos.my-app-special'
      topic 'MyApp.MyTopic'
      schema 'MySchema' # will search in the configured path
      key_config field: 'key_field'

      def send_some_message(an_object)
          payload = {
            'some-key' => an_object.foo,
            'some-key2' => an_object.bar,
            'key_field' => an_object.my_field
         }

          self.publish(payload)
        end
      end
    end

Sample consumer:

    class MyConsumer < Deimos::Consumer

      schema 'MySchema'
      namespace 'com.my-namespace'
      key_config field: :my_id 

     def consume(payload, metadata)
       MyHandlerClass.do_something_with(payload[:my_field], metadata[:key])
      end
    end

All schemas are automatically registered in the schema registry using the topic name (so if you produce to MyTopic, your schemas are automatically registered to MyTopic-keyand MyTopic-value.

Producing to Kafka

Deimos producers always produce to a given topic. Producers have a number of convenience features:

You will get an automatically generated timestamp (current time) and message_id (uuid) in your payload, if they are present in the schema. This helps immensely when debugging messages across systems.
Types will be coerced before saving, so if your schema expects a string but you give it an int, you don’t need to worry about crashes.
Metrics are sent allowing you to track the messages you send per topic and partition, as well as tracing enabled to determine if encoding your message is slow.

Consuming from Kafka

As with Phobos, Deimos creates a separate thread per consumer, where each consumer listens to a single topic. Metrics and tracing are used to track the number of messages consumed as well as how long they take to process, and there is a special “lag reporter” that sends metrics on the lag for each topic/partition being listened to.

Consumers will “swallow” errors by default (relying on tracing to handle alerting in this case) and move on, to prevent individual bad messages from blocking your consumer forever. You can change this behavior or define specific errors or situations where you would not want the consumer to continue.

Working with Databases

Although just including Avro will unlock a lot of potential, Deimos really starts to shine once you hook it up to a database via ActiveRecord. As with Avro, there are ActiveRecord versions of both producers and consumers:

ActiveRecordConsumers essentially act as a “sink” — dumping a topic into a database table based on some business logic.
ActiveRecordProducers act as a “source” — taking in an ActiveRecord object and automatically turning it into a payload to be encoded and sent to Kafka.

Sample ActiveRecordCosumer:

    class MyConsumer < Deimos::ActiveRecordConsumer

      schema 'MySchema'
      key_config field: 'my_field'
      record_class Widget

      # Optional override to change the attributes of the record before 
      # they are saved.
      def record_attributes(payload)
        super.merge(:some_field => 'some_value')
      end
    end

Sample ActiveRecordProducer:

    class MyProducer < Deimos::ActiveRecordProducer

      topic 'MyApp.MyTopic'
      schema 'MySchema'
      key_config field: 'my_field'
      record_class Widget

      # Optional override to change the default payload calculated from the record.
      def generate_payload(attributes, record)
        super.merge(:assoc_key => record.some_association.assoc_key)
      end
    end

    MyProducer.send_events([Widget.new(foo: 1), Widget.new(foo: 2)])

In addition to the producer, Deimos also provides a mixin you can add to your ActiveRecord objects which will *automatically *send messages whenever your records are created, modified or destroyed (as long as you don’t use mass operations like update_all or import).

    class Widget < ActiveRecord::Base
      include Deimos::KafkaSource

      def self.kafka_producers
        [MyActiveRecordProducer]
      end

    end

Database Backend

So we’ve got Deimos sending your database records out to Kafka and receiving messages and dumping them in. So far so good. But let’s take this one step further. In the previous article we discussed some of the issues with having both databases and Kafka in the same application — the problem of transactions.

You can read that article for more, but in a nutshell, we want a guarantee that every record written to the database has a corresponding Kafka message, and every Kafka message is written to the database.

Deimos achieves this with the database backend.

This feature transparently replaces the inline sending of Kafka messages to instead save them to a table in the database. There is then a separate process that reads the messages off this table and sends them off in batches.

)

Top: Naive Kafka. Bottom: DB Backend pattern.

Enabling this feature is incredibly simple:

Generate the migration to add the table.
Set the publish_backend configuration setting to :db .
Run the DB Producer in some way (forked process, thread, rake task) via Deimos.run_db_backend!.

This is also known as the transactional outbox pattern and achieves full transactional integrity. All messages (whether they represent changes to objects or events) are part of the same transaction and will only be saved if the transaction commits. If it rolls back, all messages will also roll back and not be sent.

Calling code doesn’t even need to know that the DB backend is turned on. You call the publish method on your producer the same way — instead of sending the message to Kafka directly, it will encode it and save it as a message in the database.

This feature is currently powering our largest producer of data and our production metrics show being able to send 4,000–6,000 messages per second with a single thread.

Test Helpers

Deimos comes with a robust suite of RSpec test helpers to allow you to test your business logic without having to actually set up Kafka and the schema registry for local testing.

Just include the TestHelpers module in your specs and automatically stub out all loaded producers and consumers:

    RSpec.configure do |config|
      config.include Deimos::TestHelpers
      config.before(:each) { stub_producers_and_consumers! }
    end

Consumers can be tested by passing a hash and the consumer class into a test method:

    expect(Widget.count).to eq(0)
    test_consume_message(WidgetConsumer, {id: 5, name: "Darth"}, call_original: true)
    expect(Widget.count).to eq(1)

Producers can be tested by just sending messages normally and using the have_sent matcher:

    expect('widget-topic').not_to have_sent(anything)
    Widget.update(:my_attr, 1)
    expect('widget-topic').to have_sent(hash_including(:my_attr => 1))

What’s Next

We’re just getting started with Deimos. We’d love to hear your use cases and feature requests, and we’ve got a bunch of our own to start with and plenty of improving to do. We welcome your contributions and we’re excited to see how far this will go!

Using Kafka as a Data Backbone: Part 1

Daniel Orner — Fri, 22 Nov 2019 18:10:47 +0000

Apache Kafka, a distributed, asynchronous streaming platform, has exploded in popularity over the last few years. It boasts a number of advantages, including fault-tolerance, availability, reliability and scalability, and is being used by hundreds of companies ranging from tiny startups to enormous companies like PayPal and Twitter.

In this article, I’ll describe how Kafka can be used to act as the data backbone for your microservice architecture, and provide a host of advantages while being able to solve many of the inherent disadvantages that come with that pattern. I’ll also cruelly hint at an awesome open-source toolkit we built that solves a big part of this but which will have an entire Part 2 dedicated to it. 😈

Event-Driven Microservices

First off — the use case we are describing here is for an ecosystem largely consisting of event-driven microservices. It’s beyond the scope of this article to dive deep into this pattern, but in a nutshell, we want to:

Decrease coupling between features and services.
Allow different teams to work independently on unrelated features.
Allow services to scale independently.
Allow services to be deployed independently.
Enforce a strong contract via technology (instead of convention) so we can be largely confident in contract testing instead of integration testing.

There are a number of challenges with this paradigm, though — in this article I’ll describe one way to use Apache Kafka to solve some of those challenges.

The Data Backbone

In order for our services to be able to do their jobs, they will have to do one of two things:

Have all the data it needs to do its job, OR
Be able to ask some other service in real-time for the data it needs to do its job.

Option 2 is the one mostly used by HTTP REST services. Unfortunately, this violates some of our goals above, notably decreased coupling and independence, since service B needs to know how to reach service A, and if service A goes down, service B now needs to handle that case.

Instead, our pattern is to copy all necessary data from service A into service B. This way, each service has immediate use of all the data it needs to do its job.

Just put all your data on the backbone!

Ignoring the increased use of disk space (which is really cheap nowadays), there are a couple of problems with this approach:

How does service B get service A’s data? Having it talk directly to service A’s database is a no-no, since that violates independence yet again.
How can we ensure real-time (or close to real-time) updates of that data?
How can we make sure we don’t miss any updates?
How do we communicate the structure of the data (and changes to that structure) without being strongly coupled to the original service?

Turns out, Apache Kafka and Apache Avro have some cool features that will help us achieve our goals and neatly bypass most if not all of our challenges.

Upsert Messages

Every message in Kafka has a key associated with it. This key by convention does not indicate the ID of the message, but of the entity **represented by that message. This allows Kafka to contain “upsert” messages, where each message represents the **current state of an entity.

Kafka itself uses this idea in its compaction algorithms — if compaction is turned on, it will automatically delete all messages but the last one which share the same key. This allows new consumers to simply skip all previous updates of a particular entity and jump straight to its current state. A deletion can be indicated by sending the entity’s key with a null message.

For example, if you send a message on a Characters topic that says name: "Luke" with a key of 105, then a second message that says name: "Darth" with a key of 105, Kafka will eventually delete the first message and only leave the last, most updated one, giving Mr. 105 the name “Darth”. If you then send a null message with a key of 105, the last message will eventually be replaced with null, indicating Mr. 105 has been deleted (this is known as a “tombstone” message).

Key compaction — old data is removed and only latest state remains.

It’s a small feature, but it has huge ramifications. It means you can use Kafka as **the data backbone of your system — as a distributed, subscribable source of truth for any type of data. **You can send the current state of any object and downstream systems will know to create, update or delete it as necessary (I explore this fully below).

This means you can use Kafka to tie together any number of smaller services, each with their own database, as long as those services send the updates of their data to Kafka individually (and Kafka is set to keep those messages around as appropriate).

Structuring Your Data

This only solves part of the problem, though. We want to have these messages represent our data for anyone who wants to read it. We don’t want to get into API hell where every downstream system needs to know about the system that generated that data so it can understand its structure. We want true independence.

What we really want is to impose some kind of schema on our messages so that not only are we sending updated state, we are also enforcing a contract between producers and consumers so that state is reliable and understandable. Also important is that the contract needs to be flexible so that we can update the producer in specific ways without immediately causing all our downstream producers to crash.

Enter Apache Avro. This specification not only defines a schema language, but also a binary encoding of that language so that downstream systems can deterministically decode the message to the schema.

Even better, it introduces the concept of “compatible” schemas, meaning that the schemas on the upstream and downstream systems don’t have to match exactly as long as they are compatible. Upstream systems are free to add fields and make changes, as long as those changes are done in a compatible way, and downstream systems don’t need to move to the new schema unless they need the new changes.

Example of a compatible change — adding a new field with a default value.

The combination of Kafka and Avro provides a strongly structured, distributed data backbone for a set of microservices.

Duplicating Your Data

But if data is “owned” by Service A, but is needed by Service B, how does it get there? The answer is duplication of data. Service B contains some kind of materialization (in a SQL/NoSQL database, in memory, etc.) of the contents of a Kafka topic. This materialization is by definition ephemeral. At any time, a service should be able to blow away its materialization and reconstruct it from the Kafka topic. The topic is the source of truth of data — all materializations are a view into it.

This allows each service to truly be independent. Because data gets into a service asynchronously, no service has a direct link to any other service. If your Orders service has to know about Users, and your Users service is currently down, the Orders service doesn’t care. Its data might be a bit out of date, but it reads it from its own materialization of that data. In fact, it doesn’t even know that a Users service exists —all it knows is that there is a Users topic.

Consuming services get information from their data stores, not directly from the producing service.

And because every message in the topic is encoded with the same Avro schema, it has a guarantee of what the contents of that topic look like — and because it provides its own reader schema, another guarantee that it’ll never crash when reading a message (assuming we don’t break the requirement that all messages in a topic are encoded with the same schema).

Now that’s all well and good. But how do we implement this “source of truth” in Kafka? What does it look like in the real world? More specifically, how does the data get into Kafka in a safe way?

Kafka and State — Uneasy Allies

The “ideal” state is one where all data is written to the topic first, and then read back into the service’s materialization of that topic. This means that even for producing services, there is no primary database or other state — the topic is everything.

“Ideal” workflow — write to topic, then read it back to the data store.

In practice, this is hard to swallow. Confluent, who is the leading company in the Kafka business, provides some helpful tooling for JVM languages around this pattern (e.g. KTables and KSQL), but it all masks the point that Kafka is always asynchronous — and sometimes synchronous things need to happen. Particularly, anything that’s user-facing will often need to render the most up-to-date state back to the user. There are a number of tactics to allow the pattern to still work in this case (add a throbber / Ajax request, use some kind of cache, etc.) but they all introduce a whole whack of complexity which honestly isn’t really needed.

What’s more, using Kafka as your single state is simply not always the best option. Relational databases are decades old, and there are hundreds if not thousands of tools and frameworks built around them. They are well understood, extremely performant for most use cases, and incredibly good at what they do. Most web frameworks assume some kind of transaction-enabled, SQL-based database to do what they need to do. We should be able to leverage this standardized technology and the new hotness of Kafka at the same time.

Databases and Kafka

There are two current leaders in the field of marrying databases and Kafka: Confluent’s Kafka Connect (which polls tables in the database and turns them into Avro-encoded messages), and Debezium (which tails the binary log of your database). Both are used pretty extensively, but they each have downsides. Kafka Connect has no way of sending deletions, since it polls the database, and deleted rows aren’t in the database any more. Debezium is heavily tied to your internal database schema (so is Kafka Connect by default, but you can write custom connectors to get around that).

So unfortunately, neither of these solutions gives you the whole picture without a bunch of pain points.

In an ideal world, your database schema is perfect! Everyone will want every column on your table and nothing in there is confusing in any way.

In the real world, you need your database schema to be internal and shielded from your consumers. You’re going to have implementation details that just aren’t relevant to anyone else. You may also want to combine multiple tables into a single event without having to write a separate joiner service downstream or being forced to write views for every table. You want to publish external-facing messages which use your data but not your schema.

That’s problem 1.

The Transaction Gotcha

We’ve been talking about sending your database state around — but let’s not forget that Kafka can and should still be used to send events. Most services that have state will still want to send events down the Kafka pipe. Keeping this in mind introduces one more sneaky issue: Transactions.

When writing data to a database, the write can fail (due to deadlocks, timeouts, etc.). Writes can also fail to Kafka (downtime, timeouts, broker issues). If one succeeds and not the other, you’ve now broken your guarantees of “single source of truth”.

Let’s say you’re writing your Character to a table, and sending a “Character Requested” event to a separate Kafka topic (perhaps with user and timestamp information). If you do this all in one transaction, the Character write can succeed but the sending the event can fail (meaning you know about the new character but the event didn’t go out), or the event can succeed but the database write can fail (meaning the event went out but it’s referencing a character that was never written).

Write works to Kafka but not the database — Luke is in the topic but not the service’s own DB.

Write works to the DB but not Kafka — downstream systems will not get the new write.

That’s problem 2.

So how do we do it? How do we marry our database and Kafka in a way that keeps our data eventually consistent, allows us to use all our lovely relational database tools for our web apps, provides the advantages of an Event-Driven Microservices architecture, and even lets us break up long-running monoliths?

Go forward and read Part 2!