DEV Community: Mihai Bojin

Build products, skip the non-functional requirements

Mihai Bojin — Sun, 17 Apr 2022 00:00:00 +0000

"Photo by Júnior Ferreira on Unsplash"

🔔 This article was originally posted on my site, MihaiBojin.com. 🔔

I asked myself, am I overengineering my pet projects? Are you!?

I like solving problems! I like finding problems more...

Call me negative! That's how I work; like anything, it's both good and bad, depending on perspective.

Almost every time I find a problem, I also get ideas. Ideas on how that particular experience could be better! And my hands start itching. I've had plenty of these moments during my career—plenty of ideas seemingly simple to implement and turn into a successful product.

But in 17 years, I barely shipped something (outside of my day job, that is).

I often ask myself, WHY!?

Has it got to do with time? Sure, I have a full-time job that keeps me busy - I always had. And I also have small kids who take up a lot of my time and energy.

But I have some spare time; spare time can be a self-imposed constraint. It took me a while to realize the power of constraints - but I am here now!

Ok, so given some limited amount of time that I can put towards personal projects, what do I do?

Well, the thing is that I'm also indecisive. I have too many ideas, and it's not reasonable that I can do everything. I started many projects but finished none. This challenge took longer to grasp and digest, but I'm also here now!

Good! So I need to apply focus to limited availability.

This leads me to the topic at hand.

Am I building products or engineering for fun!?

Building products means moving fast and lean, releasing new software, testing it against your user base, and then rinsing and repeating.

(Cue your favorite buzzword here: Lean, Agile, SDLC, etc.)

So I know what to do, but I just don't do it!

I recently started thinking of a fun data project I'd like to build. It requires a few data collection jobs running in perpetuity.

I started thinking about where I would run these and how I could optimize my costs. Cloud providers are a no go since the costs scale up fast, and I also want to avoid vendor lock-in.

You get the most bang for the buck by going low-level. Rent a bare-metal server, deploy a few VMs (i.e., with the KVM hypervisor), and operate your software.

Easy, right?

But this model is also very old-school, not to mention unreliable. What if your VMs crash?

It's better to use a scheduler such as Kubernetes. Installing it, however, is non-trivial; you need backups, automation, you need to factor in resiliency, control plane upgrades, etc. The list goes on and on!

It sounds like a fun project, and once I finished automating all of the above, I could safely deploy and operate my data collector jobs.

That I would have had not written a single line of code for...

If that's not crazy, I don't know what is!

This has been my existential crisis all along. I use my short and valuable time to solve the wrong problem and optimize because that's what I'd do professionally at scale, and that's what I am most comfortable with.

I'm starting to get it now. As the famous saying goes: "You have to spend money to make money!"

I have to find a way to start these projects more expensive than I'm comfortable with because the cost of missed opportunities is way higher than a few hundred coins a month!

Taking this approach is a forcing function of its own. I can easily afford to pay 50 quid a month for a server I don't use and will unlikely feel any pressure to deploy something on it instead of watching the next "promising" Netflix show...

This is a loser's game.

I can do better.

TBC!

If you liked this article and want to read more like it, please subscribe to my newsletter; I send one out every few weeks!

Aiming for object-oriented design elegance

Mihai Bojin — Tue, 15 Mar 2022 00:00:00 +0000

🔔 This article was originally posted on my site, MihaiBojin.com. 🔔

One of my goals for this project was to create a library that is elegant and easy to use while feeling Java idiomatic.

Let's talk SOLID. They're a set of five principles aimed at making object-oriented design implementations flexible and maintainable. I am designing a library that I hope will be a joy to use and will make developers want to adopt it, for which reason I interpreted these principles in the best possible form I could think of.

My initial thought was to offer a base Prop interface, abstracting away lower-level implementation details that are not relevant to users of this class.

However, I settled on using a few abstract classes, for several reasons, inspired by a few of the SOLID principles:

all Prop objects need a few common traits
a class should have a single responsibility; since I was building multiple themes, sticking them all into a single class didn't feel elegant
it should be easy to build on top of each layer
not all methods need to be exposed in the final public contract; unfortunately, Java interfaces do not support non-public methods

Let's break it down; here is the high-level end-result class design:

@FunctionalInterface
public interface Subscribable<T> {
  void subscribe(Consumer<T> onUpdate, Consumer<Throwable> onError);
}

public abstract class SubscribableProp<T> implements Subscribable<T> {
  /* Processes a value update event. */
  protected void onValueUpdate(@Nullable T value, long epoch) {}
  /* Processes an exception encountered during an update. */
  protected void onUpdateError(Throwable error, long epoch) {}
}

public abstract class Prop<T> extends SubscribableProp<T> implements Supplier<T> {
    /* Identifies the Prop */
    public abstract String key();

    /* Returns the Prop's value */
    public abstract T get();
}

public abstract class BoundableProp<T> extends Prop<T> {
  /* Allows the Registry to update a Prop's value */
  protected abstract boolean setValue(@Nullable String value);
}

public class Registry {
  /* Binds a Prop object to the Registry object, allowing it to process update events and set the Prop's value */
  public <T, PropT extends BoundableProp<T>> PropT bind(PropT prop) {}
}

Subscribable denotes that a Prop can be subscribed to. The result of a prop update is either success or an error.

SubscribableProp is a partial implementation that hosts the logic necessary to process updates/errors and notify clients safely.

Prop is the absolute minimum public contract that a consumer/client should care about. It defines an identifier (key) and a way to get the prop's value.

Finally, BoundableProp encompasses all of the above and also includes a mechanism that allows the Registry to update prop values when the underlying sources are updated.

However, in practice, relying on a key and value alone, is not enough of a reason to adopt this library.

For that reason, the CustomProp class provides an almost complete implementation, par the corresponding Converter.decode() method, which requires a knowledge of the Prop's type.

public abstract class CustomProp<T> extends BoundableProp<T> implements Converter<T> {
    /* Identifies the Prop */
    public String key() {};
    /* Returns the Prop's value */
    public String get() {};
    /* Describes the prop */
    public String description() {};
    /* true, if the prop is required */
    public boolean isRequired() {};
    /* true, if the prop is a secret */
    public boolean isSecret() {};
}

@FunctionalInterface
public interface Converter<T> {
  /* Decodes a String into the desired type; must be implemented */
  T decode(@Nullable String value);

  /* Encodes the value into a String, defaulting to using Object.toString() */
  default String encode(@Nullable T value) {
    return value == null ? null : value.toString();
  }
}

One way to extend CustomProp is to provide an implementation for Converter.decode, thus completing the class, e.g.:

public class LongProp extends CustomProp<Long> {
  public Long decode(String value) {
    Number number = safeParseNumber(value);
    try {
      return NumberFormat.getInstance().parse(value).longValue();
    } catch (ParseException e) {
      log.log(SEVERE, e);
      return null;
    }
  }    
}

However, we can do a bit better. Since one can assume that most props will be of common Java datatypes, I have provided a series of default converters that can be composed into a final implementation. The above code can be rewritten as follows:

public class LongProp extends CustomProp<Long> implements LongConverter {  
}

public interface LongConverter extends Converter<Long> {
   @Override
   public Long decode(String value) {
    Number number = safeParseNumber(value);
    try {
      return NumberFormat.getInstance().parse(value).longValue();
    } catch (ParseException e) {
      log.log(SEVERE, e);
      return null;
    }
  }  
}

How would we use this in production? Here's a small complete excerpt:

    Source source = new PropertyFile(PATH_TO_PROP_FILE);
    Registry registry = new RegistryBuilder(source).build();

    Prop prop = registry.bind(new LongProp("a.key"));

    prop.get(); // will return the value corresponding to a.key
    prop.subscribe(updatedValue -> {/* process updates */}, 
                   error -> {/* process any errors */});

Hopefully, this article serves as a good high-level introduction to the contract one can expect from the props library.

In future series I'd like to explore the props library's API a bit more and show a few real-world examples of how it could be used to simplify application settings/property management in Java projects.

As always, any feedback is welcome; feel free to ping me on Twitter.

Thanks!

If you liked this article and want to read more like it, please subscribe to my newsletter; I send one out every few weeks!

Subscribing to concurrent property updates

Mihai Bojin — Tue, 26 Oct 2021 00:00:00 +0000

Concurrency is hard. I know this is a truism, but it had to be said!

I spent some time in the past week implementing a mechanism for synchronizing multiple Props and returning them to the caller, all at once.

On the surface, this felt like an easy task.

Each Prop<T> is backed by an AtomicReference<T>, which enables the registry to safely (and concurrently) update the Prop's value.

One of the features I wanted to build in this library was to allow users to subscribe to updates. Whenever the registry receives a new value for a Prop, it should update the bound Prop and also notify any subscribers of the new value.

Since atomic references back the Props, I assumed the implementation would be straightforward:

create two Props and bind them to a registry
pass the two Props to a "Pair" implementation
for each of those Props, subscribe to any updates
the subscriber would retrieve both values and send an event containing both props

This implementation would work well when all the calls are synchronous. However, that is not an option. To increase this library's performance, I chose to offload the processing part of sending updates to Java's ForkJoinPool.

Asynchronously processing these events means there are no guarantees regarding the processing order.

Imagine the following scenario:

Events:
- set value A
- notify subscribers of value A
- set value B
- notify subscribers of value B

ForkJoinPool:
- notifies subscribers of value B
- notifies subscribers of value A

Outcome:
- the Prop's value is set to B
- but the subscribers were last sent A

We need a mechanism for ensuring that both the Prop's value and all the subscribers receive the same value.

One option for achieving this goal would be to synchronize the two actions and only allow for a single value to be updated and one notification to be sent. This would solve the problem but would be incredibly slow. All future value updates would need to wait until the subscribers are notified. If a slow subscriber is registered, the Prop's value would not be updateable, and any code accessing the Prop would receive the old value - not great.

A better choice is to mark down the epoch at which the value was received and track the last processed epoch when notifying subscribers. When a value is updated, we'd also increment the epoch. Any ForkJoin tasks looking to send updates to subscribers will be able to check the last processed epoch and discard any expired updates.

Ensuring the implementation worked as expected took some effort and required a bit of trial and error. Here are a few lessons I learned along the way:

Write tests to reproduce your scenario and expected results, then run them repeatedly (think Junit + repeat until failure, tens of thousands of times)
Sometimes, you need extra info to debug problems; you'd be tempted to use System.out.print(), but that can be misleading; it adds a few millis of lag which can totally derail your repro when the scenario you're trying to debug executes in nanoseconds.
System.out.print() lies: the output stream is buffered and can be printed in the wrong order; calling System.out.flush() helps, but is not perfect and still suffers from the orders of magnitude problem explained above
Two independent atomic operations do not make for one atomic operation; this is counterintuitive at first but makes total sense the more you think of it; two threads executing the same two operations could order them in 4 different ways; find a way to update both values in a single atomic operation
Syncing multiple props means the updates need to be ordered; I found one solution: queue the update operations and pass them to AtomicReference.updateAndGet() as UnaryOperator<Pair<T, U>. This method uses weakCompareAndSetVolatile memory effects, specified in the JLS as:

... defines the happens-before relation on memory operations such as reads and writes of shared variables. The results of a write by one thread are guaranteed to be visible to a read by another thread only if the write operation happens-before the read operation.

Since we don't care about sending each and every value, we can simply trigger an event operation when an underlying Prop is updated and only pull the latest value at the time of processing. This implementation will at least ensure some order to the logged operations, but it can't guarantee how many updates the subscriber sees (it could be one, or could be each and every one). That's acceptable for our use case.

Another limitation that I had to build in was that the library needs to ensure an update is consumed before sending another update. Imagine a case where two ordered operations update multiple consumers. Due to nondeterministic thread execution, a subset of the consumers may be concurrently receiving a value while others may be receiving a different value.

This could have been avoided by having a dedicated processing thread for each Prop polling a blocking queue and then sending any updates. Apart from being slower due to locking, creating a Thread has a cost (memory usage but also in terms of the maximum number of props the system can handle at once). The ForkJoinPool does not create new threads for each task and seems like the right tool for the job since, in most cases, we want a set-and-forget type situation; we want to get an update event as fast as possible to any subscribing consumers.

Takeaways

don't assume ordering
test your code until breaking point
avoid implementing concurrency primitives from scratch unless there's a real need

If you liked this article and want to read more like it, please subscribe to my newsletter; I send one out every few weeks!

Microbenchmarking Java code with JMH

Mihai Bojin — Thu, 23 Sep 2021 00:00:00 +0000

🔔 This article was originally posted on my site, MihaiBojin.com. 🔔

As a general best practice, it's a good idea to benchmark your code.

JMH, or Java Microbenchmark Harness, is a tool that can be used to analyze the performance of JVM languages.

Since I wanted to profile the performance of the Props library, I integrated JMH into the codebase.

The following is a simple step-by-step tutorial about integrating the JMH Gradle plugin in a Java codebase.

Gradle configuration

First, add the JMH Gradle plugin in your build.gradle.kts file:

plugins {
    id("me.champeau.jmh").version("0.6.6")
}

Doing so will add a few tasks to your Gradle project:

gradle jmh: Runs all benchmarks
gradle jmhJar: Generates a portable JAR that you can run on a different machine

The second target is helpful for running the benchmarks on a dedicated machine (that is not your developer laptop), resulting in predictable, comparable, and reproducible results.

Writing a JMH benchmark

The plugin expects all the benchmark code to exist in src/jmh/java and src/jmh/resources. This avoids having to create a separate project and importing all the code while at the same time avoiding shipping the benchmark code with the main library in src/main/java.

Let's create the first benchmark. Save this file as src/jmh/java/Benchmark.java in your module.

@Fork(value = 1, warmups = 1)
@Warmup(iterations = 1)
@Measurement(iterations = 1)
@OutputTimeUnit(TimeUnit.SECONDS)
public class Benchmark {
  @Benchmark
  public static void oneBenchmark() {
    // do something
  }
}

The code above is just scaffolding. Let's look at each annotation.

@Fork: configures how many times the current benchmark is forked. If value=0, the benchmark will be run in the same JVM. The warmups parameter defines how many times the benchmark is forked (but the results discarded).

The main benefit of warming up is to load all classes and cache them. Unfortunately, since the JVM uses lazy loading and Just In Time compiling, the first iteration of our benchmark would incur the cost of all these actions and skew the results.

@Warmup determines how many warmups are performed and discarded per fork.

@Measurement allows us to specify how many iterations to execute per benchmark.

And finally, @OutputTimeUnit allows us to specify the unit reported in the results.

There are more annotations and parameters, but I won't get into the weeds of it just yet.

"Consuming" results

There is a small caveat when writing benchmarking code in that the JVM is smart enough to optimize code that is not actually used.

For example, in the following code, the result of tested.get() is never used (consumed) so the running JVM may decide to simply skip the call altogether, making the benchmark invalid.

public class Benchmark {
  @Benchmark
  public static void oneBenchmark() {
    // assume an object under test
    tested.get();
  }
}

JMH introduces the concept of a Blackhole. The code above can be rewritten, ensuring the results are always used and the code being benchmarked is executed:

public class Benchmark {
  @Benchmark
  public static void oneBenchmark(Blackhole blackhole) {f
    // the code under test is always executed
    blackhole.consume(tested.get());
  }
}

You can now run all the benchmarks with the gradle jmh command.

Et voila! A super simple intro to JMH in a Gradle project!

An update on efficient multi-layered key ownership

Mihai Bojin — Tue, 21 Sep 2021 00:00:00 +0000

🔔 This article was originally posted on my site, MihaiBojin.com. 🔔

In the previous article, I introduced the concept of key ownership.

Props is a library that allows users to load settings from multiple sources, layering them in a predetermined order.

Given two sources (i.e., layers) added to the Props registry in order, if both define a key (X), the second one wins (we say owns) the key and will provide its value, which we call an effective value.

My original design was to keep a map of keys to values for each Layer and further have the Registry keep a synchronized map of keys to layers, enabling it to determine which Layer owns a Key and then query it to retrieve its effective value.

As I was implementing this algorithm, I realized it was allowing a race condition that would require a severe decrease in throughput to fix.

Imagine the following scenario:

given that both Layers 1 and 2 define values for X
when the client requests X
then the registry determines Layer 2 owns X
however, before retrieving a value, Layer 2's source updates itself and unsets X
in this scenario, the client will receive a wrong response (X=null) instead of the correct value from Layer 1

This situation is similar to a dirty read in a transactional database system.

To prevent this edge case, we'd need a global, per-key lock to ensure that the Layer that owns X cannot unset it during a get(X) call.

Since this would play out over multiple objects (Registry and Layer) and data structures (Map<Key, Layer>, Map<Key,Value>), we would need another data structure to indicate that Key is currently being retrieved.

When any layer is updated, we'd check this map and ensure we only update X when it is not being read. In high QPS applications, this would quickly become a bottleneck!

There are ways to optimize this flow, but ultimately I realized that this design was flawed.

What's most interesting to me is that I had this epiphany while writing this very article, showing the value of building in public

When you have to organize your thoughts well enough to write them down, magic happens!

This situation prompted me to rethink the implementation.

Instead of mapping keys to layers, I could instead map keys to effective values.

This would allow the registry to keep a single synchronized Map<Key,EffectiveValue>, guaranteeing atomic operations and acting similarly to the read committed isolation level in a DBMS. This feels like a better implementation!

Let's examine some code:

// first we need an object to hold a value/layer pair
class ValueLayer {
    final String value;
    final Layer layer;

    // compares this object with deconstructed properties (value, layer) to avoid creating a new instance
    boolean equalTo(@Nullable String value, Layer layer) {
        return Objects.equals(value, this.value) && Objects.equals(layer, this.layer);
    }
}

// then we can implement reads/writes
public class Registry {
  protected final ConcurrentHashMap<String, ValueLayer> effectiveValues = new ConcurrentHashMap<>();

  // reads are straighforward, since the map is synchronized
  public ValueLayer get(String key) {
    return this.effectiveValues.get(key);
  }

  // writes are a bit more complex, since we need to determine how the value changes
  public ValueLayer put(String key, @Nullable String value, Layer layer) {
    return this.effectiveValues.compute(
        key,
        (k, current) -> {
          if (current == null) {
            // if we had no previous value for this key

            // and if the new value is null
            if (value == null) {
              // do not map the key
              return null;
            }

            // otherwise map the effective value and its owning layer
            try {
              return new ValueLayer(value, layer);
            } finally {
              // and send an update to any objects following this key
              this.sendUpdate(key, value, layer);
            }
          }

          // if nothing has changed
          if (current.equalTo(value, layer)) {
            // simply return the current value
            return current;
          }

          // if the layer's priority is lower than the current owner
          int oldPriority = current.layer.priority();
          int priority = layer.priority();
          if (priority < oldPriority) {
            // do nothing as this layer does not hold ownership over the key
            return current;
          }

          // if the value has not changed
          if (Objects.equals(current.value, value)) {
            // we need to modify the layer/value mapping
            // but not send an update to any objects following this key
            return new ValueLayer(value, layer);
          }

          ValueLayer ret;

          // but if the value has changed

          if (value == null && priority == oldPriority) {
            // it was either unset from the owning layer
            // in which case we need to find a lower priority layer that defines this key
            ret = findPrevious(key, current);

          } else if (value == null) {
            // it was unset in a higher priority layer
            // in which case, this is a no-op
            return current;

          } else {
            // or it was legitimately updated
            // and we need to modify the layer/value mapping
            ret = new ValueLayer(value, layer);
          }

          try {
            return ret;
          } finally {
            // and send an update to any objects following this key
            this.sendUpdate(key, ret != null ? ret.value : null, layer);
          }
        });
  }

  // sends updates to any objects following the key
  public abstract void sendUpdate(String key, String value, Layer layer);

  // finds the first lower priority layer that defines the key
  // or returns null if one does not exist
  private static ValueLayer findPrevious(String key, ValueLayer current);
}

After I fully implement this solution, I will do two things:

write a few unit tests to verify that the behavior of the system will not change after further refactorings.
write a few benchmarks and compare this new design with the previous version, for which I'll useJava Microbenchmark Harness (jmh).

Until next time, stay tuned and if you want to follow this series by email don't forget to subscribe!

If you liked this article and want to read more like it, please subscribe to my newsletter; I send one out every few weeks!

Hosting my site on Vercel

Mihai Bojin — Thu, 09 Sep 2021 00:00:00 +0000

🔔 This article was originally posted on my site, MihaiBojin.com. 🔔

Today I switched my site's hosting from Gatsby Cloud to Vercel.

After writing my latest post on application design, I ran into an issue I couldn't quite solve.

So I decided to take this opportunity and try out Vercel!

Prerequisites

Before we go any further, take a moment to check your package.json file. It should contain the engine section, enabling Vercel to use the same version as on your local development environment.

{
  "engines": {
    "node": ">=14.17 <15"
  },
}

Hosting on Vercel

Porting my site over was super easy:

First, I went to https://vercel.com/new, where I clicked Add GitHub Org or Account, authenticated with my GitHub credentials, and imported my site's repository.

Vercel automatically detected I was using GatsbyJS and preconfigured the project for me.

All I had to do was to click Deploy! Vercel promptly built my site and started serving it on a subdomain. Done (or not quite)!

Custom domain

Since I wanted to use my own domain, I had to do a bit of extra config.

Vercel's domains page is a good starting point, but you can also achieve the same from your project's dashboard.

I manage all my domains using Cloudflare and maintain DNS records using Terraform, so I had to update my zone config and reapply them.

Here is the Terraform HCL I used, which can be adapted and applied with terraform plan && terraform apply.

resource "cloudflare_record" "com_mihaibojin_apex" {
  zone_id = var.zone_id_com_mihaibojin
  name = "mihaibojin.com"
  value = "76.76.21.21"
  type = "A"
}

resource "cloudflare_record" "com_mihaibojin_vercel_cname" {
  zone_id = var.zone_id_com_mihaibojin
  name = "www"
  value = "cname.vercel-dns.com"
  type = "CNAME"
}

That's all, folks

And that was it! After a few minutes of simple configurations, my site is now hosted on Vercel.

What's even cooler is that its performance (judging by the Lighthouse report) seems to be a bit better!

If you liked this article and want to read more like it, please subscribe to my newsletter; I send one out every few weeks!

System Design goal: efficient property reads from multiple sources

Mihai Bojin — Mon, 06 Sep 2021 00:00:00 +0000

🔔 This article was originally posted on my site, MihaiBojin.com. Since it contains a few diagrams, it cannot be correctly displayed on DevTo. I highly recommend reading it on my site! 🔔

A problem of ownership

I set out to build the props library to standardize retrieving (key,value) pairs from multiple sources while also allowing a simple mechanism to establish key precedence when more than one source is aware of the same key.

Let's consider the following scenario, where multiple source_s define _property X:

Source (Layer)	Value
Prop X is defined by source 1	1
Prop X is defined by source 3	2
Prop X is defined by source 4	3
Effective value	3

Sources can be property files, system properties, or they can be coming from a key-value store or database.

Since each (key,value) pair can change independently on each source, we need to establish a key's precedence, basically answering the "Who provides the value for this key?" question.

We do so by introducing the concept of layers.

Each added layer can retrieve data from a source. As we add more layers, each takes precedence over the previous layer, for any keys it defines.

We call this concept ownership.

Sources vs. layers

Let's think for a moment about sources and layers.

A naïve approach would be to model a 1:1 relationship between Sources and Layers, basically creating a single entity to represent both, then passing a list of these entities to a registry.

We see that both sources point to the same file on disk. However, we have two objects that independently read the same data and create a potential for contention and consistency problems (e.g., SourceLayer 1 reads data, the file is updated on disk, SourceLayer2 reads different data).

Consistency is not always possible and cannot always be a property of a system, but in this case, we can ensure that a source has the same view of its data in a running JVM process.

Rather than have source_s and _layer_s be the same (1:1), let's instead have a single _source representing a dataset and multiple layers (1:N) which receive updates when the dataset changes. An added benefit of this design is that our system only needs to do one read operation per dataset instead of one per layer. Since I/O is generally more "expensive" than in-memory operations, this is a win!

Here are the components of our system:

Source: loads the data from its origin (e.g., filesystem), reloads it using user-defined logic (e.g., an interval), and notifies registered layers of any reload events
Layer: holds copies of the data for each registry
Registry: queries data from each of its layers

Retrieving the effective value

But what if a layer adds a new key or removes an existing one!?

Between two reads of X, the owner could change from Layer 3 to Layer 2.

A simple algorithm to solve this problem is always to query all layers, e.g.:

ask Layer 1-3: Do you define X?
Layers 2 and 3 respond yes
Layer 3 has precedence and owns the key
X is unset from Layer 3
we repeat the first three steps and establish Layer 2 as the owner of X

This choice is not ideal. First, we'd have to repeat this on every read , which would result in O(K) time-complexity (where K is the number of layers in the registry) for every single read property. This doesn't sound performant!

A side objective here is to deal with a high frequency of ownership changes; for example, a source dataset may repeatedly set and unset one or more keys in a short interval.

Type casting values

Before we talk about ownership, let's quickly segue into data types. The values will always be serialized as strings as we're reading them from property files, system properties, and the environment. (The library will support loading values from other data stores, e.g., databases, but the problem of casting strings from property files remains)

Here's an example of a standard Java property file:

string.prop=some value
int.prop=42
float.prop=1.23

Even though we have integer and float values, the relevant Java APIs will still return a Map<String,String>, leaving the job of casting these values to the appropriate types to the developer.

In practice, while developers "come and go" in projects, this results in many ways to, not-so-optimally, load strings and repeatedly convert them to the desired type.

When I set out to build the props library, I wanted to give developers "one" way of loading the values corresponding to each key without worrying about types, simple validations, or caching. My goal is to implement a no-hassle, efficient library for dealing with application properties!

How do we establish key ownership?

Ok, back to ownership!

The registry needs to keep track of a large enough number of keys (let's say in the 10,000s range), and retrieve values from the appropriate layer.

Developers will be able to rely on a registry to retrieve values cast to their desired type.

Let's examine the internals of a get(key, type) call.

A client queries the registry (1), which in turn checks all its layers (2), awaiting responses (3), then determining the owner and returning the effective value to the client (4).

In practical applications, we can assume a registry will have a few layers (K).

If our application handles N properties (keys), then reading all the properties at least once would be an O(N*K) operation. Given that ultimately a key is owned by a single layer, the best we can hope for is O(N) for N keys, or O(1), constant time, per key.

This time-complexity is not only more efficient but also totally achievable!

Internally, the registry needs to keep track of which layers own each key.

Since layers can be updated concurrently, we need a thread-safe data structure.

Since we need to "map" (pun intended) keys to layers, we need to use a Map (duh!)

The Java SDK provides a very efficient, thread-safe map: ConcurrentHashMap.

Let's see how this could look in code:

public class Registry {
    ConcurrentHashMap<String, Layer> owners;

    // retrieves the value for the specified key
    public <T> T get(String key, Class<T> clz) {
        // finds the owner
        Layer owner = owners.get(key);
        // retrieves the value
        String effectiveValue = owner.get(key);
        // casts it
        return clz.cast(effectiveValue);
    }
}

public interface Layer{
    // retrieves the value for the specified key, from this layer alone
    String get(String key);
}

The last piece of the puzzle is updating ownership. Below, we examine the new key case.

public class Source{
    List<Layer> layers;

    // reloads data periodically, at the specified interval
    void reload(Duration interval) {
        // the trigger will be periodically scheduled at an interval
        var trigger = () -> {
            // data is reloaded from source
            Map<String, String> data = ...;

            // and then sent to all registered layers for this source
            for (Layer layer : layers) {
                layer.onReload(data);
            }

            // we may also want a sync stage here
            // to make the new dataset available to all sources at the same time
            // but it's beyond the scope, for the moment
        }
    }
}

public class Layer{
    // processes the reloaded data
    public void onReload(Map<String, String> data) {
        // for each key in the dataset
        for (var entry : data.entrySet()) {
            // updates the value
            updateValue(entry.getKey(), entry.getValue());
            // and notifies the registry that a new owner might be available
            registry().bindKey(entry.getKey(), this);
        }
    }

    // decides the priority of this layer, in the current registry
    int priority();

    // reference to the registry that owns this layer
    abstract Registry registry();

    // stores the (key,value) pair
    abstract void updateValue(String key, String value);
}

public class Registry {
    ConcurrentHashMap<String, Layer> owners;

    // registers ownership of a layer over a key
    void bindKey(String key, Layer layer) {
        // finds the current owner
        Layer owner = owners.get(key);

        // determines if ownership change is required
        if (owner.priority() < layer.priority()) {
            // change the current owner
            owners.put(key, layer);

            // from this point, all "get"s of "key" will be routed to "layer"
        }
    }
}

The scenario above details how the system will handle new keys appearing in a layer.

A full implementation also needs to consider deleted keys and updated keys. For those cases, we'd want to avoid calling registry().bindKey(...) since it would result in a no-op anyway.

Overall, I am willing to bet that ownership changes will be less frequent than value changes in real-world scenarios. So, the upfront cost of adding and deleting keys, represented by registering and unregistering keys from a layer, will be worth it to achieve constant time (O(1)) reads per key.

Conclusion

At this point, I am genuinely excited to implement this code!

I'm sure I will learn more as I go along and may change my mind on some of these choices, but this is the plan for now!

Until next time...

If you liked this article and want to read more like it, please subscribe to my newsletter; I send one out every few weeks!

Create a new open-source Java project using the Gradle build tool

Mihai Bojin — Thu, 02 Sep 2021 00:00:00 +0000

"Photo by Danist Soh on Unsplash"

🔔 This article was originally posted on my site, MihaiBojin.com. 🔔

When building a house, you should start with a strong foundation. Similarly, creating a new open-source project requires a certain level of organization!

Roughly, here is a base minimum that makes for a decent project:

A version control system: having it is a must-have, especially if you're planning to publish an open-source project!
A README: help your visitors understand what the project is about and how they can find out more
A LICENSE: choosing one early informs potential users if they can avail of the project
A CONTRIBUTING guide: this is not precisely day one stuff, but starting one and updating it as your project develops, encourages outside contributions
A build tool: regardless of the chosen programming language, you will need a way to manage dependencies and build or redistribute your project as binaries

Any decent build tool should be able to support:

subprojects or modules
code formatting
static code analysis
global versioning scheme
generating sources and documentation
running tests
enforcing a coding standard

Let's look at all of these in detail below.

Version Control System

Most projects today use git hosted on GitHub. Other options exist, but they are beyond the scope of this article.

To create a new repository, take the following actions:

Create a new organization: https://github.com/account/organizations/new?coupon=&plan=team_free
Create a new repository
Choose a license (I chose MIT because it's permissive; doing so will create a LICENSE file in your repository)
Here's the result: https://github.com/props-sh/props

Basic repository information

At the very least, provide essential information, in a README, that helps your users understand what the project is about, how it can help them if they can use it, and how they can contribute to it.

A LICENSE file informs potential users about the conditions under which they can use your project. This article doesn't aim to inform about all differences between various types of licenses, but you can read more about it here.

Finally, a CONTRIBUTING guide will help potential developers open PRs and collaborate in your project.

Build tool

And now, let's get to the 'meaty' part.

Since I'm writing a Java project, this is very opinionated. For the first iteration of this project, I used Bazel but, since I'm starting from scratch, I thought I'd have some fun and use Gradle.

I started by installing SDKman! and then Gradle: sdk install gradle.

Since I'd never used it before, I started by reading the getting started guide. Another helpful page is how to configure java projects.

I use IntelliJ as an IDE, and it has full support for Gradle using Kotlin DSL.

Next, let's configure a sound basis for a Java library made up of multiple submodules.

Root directory configuration

I've set up this project as a multi-project build.

I started by creating a settings.gradle.kts file, which defines a name, and the pluginManagement/repositories section, which represents a set of artifact repositories to use in all (sub) projects.

rootProject.name = "props"

pluginManagement {
    repositories {
        mavenCentral()
        gradlePluginPortal()
    }
}

Define a global version for all dependencies

I created a build.gradle.kts file and defined a group and version.

group = "sh.props"
version = project.version

The version is implicitly loaded from the project and can be specified in a gradle.properties file:

version=0.2.0-SNAPSHOT

or by passing it via the command-line, e.g., gradle -Pversion=0.3.0 ...

Create a subproject

IntelliJ can easily create subprojects:

right-click the project and select New Module
or press Cmd-N
select Gradle Java module
fill in the details and you're good to go!

Alternatively, create a new directory and inside of it:

create a build.gradle.kts file
create a Maven-like directory structure: src/main/java, src/main/resources, src/test/java, src/test/resources

Include this subproject in the settings.gradle.kts file:

include("java-props-core")

Set a java toolchain version

Edit the subproject's build.gradle.kts file and define the following block to set the Java version to 11.

plugins {
    `java-library`
}

java {
    toolchain {
        languageVersion.set(JavaLanguageVersion.of(JavaVersion.VERSION_11.toString()))
    }
}

// additionally also allow incremental builds, speeding up build speed
tasks.compileJava {
    options.isIncremental = true
}

Generate source JARs and Javadocs

Since props is a library, I wanted to generate source and documentation (Javadoc) JARs.

Luckily, this is super easy to do in Gradle. Again, edit the build.gradle.kts file:

java {
    withJavadocJar()
    withSourcesJar()
}

tasks.create<Zip>("docZip") {
    archiveFileName.set("doc.zip")
    from("doc")
}

Running tests

I chose the JUnit 5 (Jupiter) framework, for which we need to set up a few things:

First, define a new task that will run the tests when called

tasks.test {
    // Use JUnit Platform for unit tests.
    useJUnitPlatform()

    // limit heap usage to 1G (can we tweaked later)
    // https://docs.gradle.org/7.2/userguide/java_testing.html#sec:test_execution
    maxHeapSize = "1G"
}

As well as the necessary dependencies:

dependencies {
    testImplementation(libs.junit.jupiter.api)
    testRuntimeOnly(libs.junit.jupiter.engine)
}

The libs.junit.jupiter.api notation represents type-safe project dependencies in Gradle. These are defined in the root project settings; see below.

Define a version catalog

Version catalogs are an incubating feature and must be explicitly enabled.

Edit the settings.gradle.kts file and add the following section:

// enable the incubating feature
enableFeaturePreview("VERSION_CATALOGS")

dependencyResolutionManagement {
    versionCatalogs {
        create("libs") {
            // define the version once
            version("junit", "5.7.2")

            // then create aliases to each component, referencing the version above
            alias("junit-jupiter-api").to("org.junit.jupiter", "junit-jupiter-api").versionRef("junit")
            alias("junit-jupiter-engine").to("org.junit.jupiter", "junit-jupiter-engine").versionRef("junit")
        }
    }
}

Once the above is done, any tests deployed in **/src/test/java can be run by executing gradle test.

Code formatting

As a codebase grows, it's essential for the code to 'look' the same, as it helps give developers a consistent experience.

The chosen format (style) is less important than doing this from early on. After all, you can always change the style and reformat the codebase in one go later on!

This is, in my opinion, best done by a tool that indiscriminately auto-formats your code (as you write it)!

Gradle has a great plugin for this: spotless.

Define the plugin's version in your settings.gradle.kts file:

pluginManagement {
    plugins {
        id("com.diffplug.spotless").version("5.14.3")
    }
}

And add the following configuration to your build.gradle.kts file:

spotless {
    // generic formatting for miscellaneous files
    format("misc") {
        target("*.gradle", "*.md", ".gitignore")

        trimTrailingWhitespace()
        indentWithSpaces()
        endWithNewline()
    }

    // chose the Google java formatter, version 1.9
    java {
        googleJavaFormat("1.9").aosp()

        // and apply a license header
        licenseHeaderFile(rootProject.file("props.license.kt"))
    }
}

As you can see in the comments, it formats Java files as well as other extensions.

For Java, it uses the google java formatter and also applies a license header to every file. The props.license.kt contains a copy of the LICENSE file with a few tweaks:

the text is a comment block: (/* ... */)
and the year is programmatically filled in: ($YEAR)

/*
MIT License

Copyright (c) $YEAR Mihai Bojin

Permission is hereby granted...
*/

Check formatting with gradle spotlessJavaCheck or apply it with gradle spotlessJavaApply.

Coding standard

Checking that a project's coding standard meets minimal quality criteria results in better code!

For this I use checkstyle, which is also very easy to set up with Gradle:

Add the following to settings.gradle.kts :

dependencyResolutionManagement {
    versionCatalogs {
        create("libs") {
            // make a global version available
            version("checkstyle", "9.0")
        }
    }
}

Then edit the subproject's build.gradle.kts file:

plugins {
    checkstyle
}

checkstyle {
    // will use the version declared in the catalog
    toolVersion = libs.versions.checkstyle.get()
}

Run the check with gradle checkstyleMain.

Smarter static code analysis

Finally, the last component I'll be talking about in this article is errorprone, a library built by Google that performs smarter static code analysis. Apart from being a great standalone mistake finder, it also integrates with other projects such as NullAway, a tool that helps developers find and fix NullPointerExceptions in their Java code.

Start by adding the plugin to your settings.gradle.kts file:

pluginManagement {
    plugins {
        id("net.ltgt.errorprone").version("2.0.2")
    }
}

And then add the following configuration to your subproject's build.gradle.kts file:

import net.ltgt.gradle.errorprone.errorprone
plugins {
    id("net.ltgt.errorprone")
}

dependencies {
    errorprone(libs.errorprone.core)
    errorprone(libs.nullaway)
}

// hook up the checker to the compilation target
tasks.withType<JavaCompile>().configureEach {
    options.errorprone.disableWarningsInGeneratedCode.set(true)
}

Conclusion

Hopefully, this tutorial gave you an idea of the first steps you should take towards starting an open-source Java library on GitHub.

So far, I only spent a few hours learning how to do all of these from scratch and had a lot of fun doing it!

Since I haven't ported over any code from the v1 repo, I have probably yet to smooth out all the rough edges. If you end up using it, ping me on Twitter and tell me how you got on! ( I'd appreciate it! )

You can find all the code referenced above on GitHub in the props project!

Stay tuned for more articles in this series, and don't forget to subscribe to my newsletter! I'll be sending regular updates as I write more code.

Thank you and until next time!

If you liked this article and want to read more like it, please subscribe to my newsletter; I send one out every few weeks!

Designing a library for reading layered application settings in Java

Mihai Bojin — Sun, 29 Aug 2021 00:00:00 +0000

"Photo by Sigmund on Unsplash"

🔔 This article was originally posted on my site, MihaiBojin.com. 🔔

For the past year, I've been on and off thinking of a common problem: handling properties in Java. While the problem seems simple, dealing with multiple sources, optionality, secrets, etc., is not trivial.

I started by looking for existing implementations, but I couldn't find many.

The closest I found was Netflix's Archaius, but it didn't do everything I needed and had features I didn't need. However, what sealed the deal was that the library seemed abandoned, with the last minor release having happened in 2019.

I decided to build my own (#NIH, I know 😝).

The first version

My first attempt resulted in the props library. For this, I had a few goals.

I wanted the ability to load properties from multiple sources : Java property files, System properties, OS environment variables, etc.

Since these sources are text-based, the values are first loaded as strings. However, strings can be deserialized into many types, which is what the application would expect. Hence, we want such a library to type cast values into the appropriate (standard or user-defined) types. We could do that every time a value is read. However, doing so on every read would result in many unnecessary type conversions. A better approach is to cast once and memoize the typed value until and if it changes.

Apart from its value, a prop has other attributes (metadata). The metadata includes:

if a prop is required or optional
if it has a default value
if it represents a secret (in which case extra care needs to be taken)
a description of what the property is used for, aiding with documentation

A key could be defined in more than one source, so the library needs to determine which source owns which key.

Then, the values could change at the source; to tackle that, we'd need a mechanism to check for updates and refresh any updated keys periodically.

We also want to account for extendability. For example, a client of this library might need to define new sources (e.g., read properties from a MongoDB database or an HTTP endpoint). To that end, we provide an interface that users can implement to add extra functionality.

Another use case is managing secrets. While it is best to use production-ready solutions (e.g., a KMIP-based server), some applications require keys to be defined in property files. In that case, it is best to encrypt them at rest. Encrypting generates binary output; a common way to represent binary data as text (e.g., in a property file) is to base64-encode it. Our library would need to be able to deserialize a base64-encoded, encrypted value as a string seamlessly.

Since I wanted to use this library in Java projects, I set up a CD pipeline to release it to Maven central.

I have built Props v1 with the above in mind.

The architectural problem

However, I was too focused on building a proof of concept and didn't think too much about performance. I also built this first version with a static use-case in mind (the client is responsible for refreshing a prop's value.)

There are, however, cases where a client would also benefit from being notified when values change. Upon realizing that, I added support for notifying subscribers of updates. I also started looking at where the bottlenecks were. While the library is by no means slow, a few original decisions make it difficult to improve.

This is a common problem in software engineering. For established libraries with significant user adoption, it is costly to rearchitect from scratch. Usually, the code would be updated slowly, taking great care to avoid breaking changes.

Since this is a personal project with no current production usage - I have a bit of flexibility.

I have decided to rewrite the code with performance in mind. However, I also want to focus on asynchronous primitives, which are a better choice for today's software.

Imagine the following scenarios:

A. A property is loaded once from a configuration file and used to configure the maximum number of requests allowed by the application in a given interval. When the application needs to be reconfigured, the file is updated, and the application restarted. Restarting is not instantaneous, and it can result in downtime for users. To avoid it, multiple instances need to be run and managed by an external orchestration service (e.g., Kubernetes).

B. The same property is aware of its origin and can notify the implementing code of any updates. The application can then reconfigure itself without fully restarting.

There is, of course, no 100% right solution. If a platform already manages the application, and multiple instances are deployed and running, and if restarting the service is a fast enough endeavor, Solution A may be perfectly viable. In most cases, however, at least having the option of using Solution B is a positive net benefit: it's there, but you don't have to use it. Not having it, though, could be limiting. This logic could be extended to replacing secrets, enabling a new feature for an increasing subset of users, etc.

How would a better implementation look like?

Glossary of terms

Let's define a few common terms:

source : where a key=value pair originates from
prop : an object that can return a type cast value for its registered key
registry : an object that keeps track of all sources and props
layers : a mechanism that helps to establish the priority in which sources own a particular key
effective value : the final value of a key, taken from the source that owns it

Design considerations

Let's outline a few design considerations.

Sources must be layered in the sense that they should establish clear precedence over which source ultimately owns which key.

When a value is updated on a source, and that source owns the prop, the prop needs to be (eventually) updated with the new value. "Eventually" means that a prop may not see all the state transitions (writes) between reads. It's important to note that I am not designing an event streaming solution. Therefore, some updates may never reach the client.

We need to choose a 1:1 or 1:many mapping between keys and prop objects in the registry. On the one hand, allowing more than one prop for any given key could be flexible. For example:

var prop1 = new StringProp("my.key");
var prop2 = new StringProp("my.key");

Multiple callers could bind and use new prop objects without any limitations. However, I believe this choice would enable users to write inefficient code.

The next question is, should we allow props of different types to be bound to the same key?

Consider the following code:

var prop1 = new DurationProp("my.key");
var prop2 = new LongProp("my.key");
var prop3 = new StringProp("my.key");

You could argue that all three forms are needed, but the same could be achieved in the following way:

var prop = new DurationProp("my.key");
Duration result = prop.get();
Long seconds = result.toSeconds();
String stringValue = result.toString();

Asynchronous updates would need to be propagated fast. There is no point in implementing such a library only to support a few updates per second or only a few tens of keys defined in the system.

Let's focus on performance by having the library restrict one prop object per key to keep the number of prop objects small and the implementation lean.

Since sources can specify custom update intervals, we can assume values could change, and keys could appear or disappear , on multiple sources, in quick succession.

Let's look at a few scenarios for a given key defined on two sources:

Action	Ownership	Value
Prop is defined by source 1	Source 1, val=1	1
Prop is defined by source 2	Source 2, val=2	2
Prop is updated by source 1	Source 2, val=3	2
Prop is updated by source 2	Source 2, val=4	4
Prop is deleted from source 1	Source 2	3
Prop is defined by source 1	Source 2, val=5	4
Prop is deleted from source 2	Source 1	5
Prop is defined by source 2	Source 2, val=6	6
Prop is deleted from source 2	Source 1	5
Prop is deleted from source 1	N/A	null

Our implementation needs to handle all cases correctly. These could happen in fast succession for a large number of keys, so it's essential to handle these state transitions as efficiently as possible.

Some of them are easier than others; for example, updating a key's value without changing source ownership is straightforward. However, changing the source that owns a key is more involved since we need first to determine the correct source that should provide the effective value.

There is a real possibility of update storms ; for example, a source could repeatedly own and disown a key over a short interval. This is probably more of an edge case than a legit need. In any case, the implementation should be able to:

effectively deal with ownership transitions in quick succession
flag when this behavior occurs so that a human operator can investigate the correctness

It's important to note that this case is different than everchanging values (e.g., a boolean flag that keeps flipping from true to false and back, or a counter that is incremented thousands of times per second.)

Another question is if we should eagerly or lazily update each key?

Even though several sources could potentially define thousands of key-value pairs, that doesn't necessarily imply that those values are always read or that they are refreshed at the same interval.

Eagerly establishing source-ownership and updating the effective value of every key would probably be costly (performance-wise).

On the other hand, lazily determining which source owns each key at reading time, retrieving the effective value, and finally, type casting it would result in slower reads.

The implementation will need an efficient way to keep track of key ownership.

For example, when a value is required, it would be best if the registry could go directly to the source that owns that key, instead of having first to find (iterate over all sources, in O(N) time) the appropriate source. Respectively, when a source's values are updated, it also needs to efficiently publish updates to all prop objects it owns while skipping ones it does not.

Conclusion

I could write more, but for now, this article would have hopefully set the stage for what this library will do.

Rather than outlining every possible feature ahead of time, I prefer developing a series of articles as I get more and more into the code. Who knows, I may even change my mind about some of these things along the way!

I plan on (re)writing this Java library from scratch and documenting it all, including topics such as:

setting up a GitHub repository
configuring a build system
configuring a CI/CD pipeline
writing performant, well-tested code
pushing releases to a public Maven repo (e.g., Maven Central)
formatting the code to a common standard
linting the code
benchmarking and documenting a performance baseline
and others

Stay tuned for more, and don't forget to subscribe to my newsletter if you want to receive all updates about this series by email!

Until next time...

Mihai

Crossposting articles from Gatsby to Medium, Dev.to, and Hashnode

Mihai Bojin — Sat, 21 Aug 2021 00:00:00 +0000

As sole authors, especially when starting up, we don't have established audiences. Without a critical mass of people to reach, the content we put so much love and care into ends up being ignored and not helping anyone. This is tough to deal with from a psychological level, but also what many people call ' the grind.'

With all that, it's essential to take any opportunity to reach a broader audience. One solution is to crosspost our content on various distribution platforms.

Since I write for software developers, my go-to places are Medium.com, Dev.To, and Hashnode.com.

Generally, this is a two-step process:

create an RSS feed containing your most recent articles
(a) have the destination monitor the feed and pull new articles or (b) use automation tools to push new content using an API

Let's dig in!

Publishing an RSS feed from GatsbyJS

Start by installing one of Gatsby's plugins:

npm install gatsby-plugin-feed

Once installed, add it to your gatsby-config.js file:

module.exports = {
  plugins: [
    {
      resolve: `gatsby-plugin-feed`,
      options: {
        query: ``,
        setup(options) {
          return {};
        },
        feeds: [
          {
            title: "RSS feed",
            output: '/rss.xml',
            query: ``,
            serialize: ({ query: {} }) => {},
          },
        ],
      },
    },
  ],
};

The configuration above is not enough to generate a usable feed.

Let me break it down, option-by-option.

Note: when I first implemented the RSS feed on my site, I struggled to find examples that explained how I could achieve everything I wanted. I hope this article will help others avoid my experience.

Retrieve site metadata

First, each feed will need to access the site's metadata. The format of your site's metadata can be different, but generally, the GraphQL query to select it will look as follows. If it doesn't work, you can always start your GatsbyJS server in development mode (gatsby develop) and debug the query using the GraphiQL explorer.

module.exports = {
  plugins: [
    {
      resolve: `gatsby-plugin-feed`,
      options: {
        query: `
          {
            site {
              siteMetadata {
                title
                description
                author {
                  name
                }
                siteUrl
              }
            }
          }        
        `,
      },
    },
  ],
};

Setup

The gatsby-plugin-feed relies on a few pre-defined (but not well-documented) field names to build the resulting RSS.

Below, you can see a snippet with line-by-line explanations:

module.exports = {
  plugins: [
    {
      resolve: `gatsby-plugin-feed`,
      options: {
        setup(options) {
          return Object.assign({}, options.query.site.siteMetadata, {
            // make the markdown available to each feed
            allMarkdownRemark: options.query.allMarkdownRemark,
            // note the <generator> field (optional)
            generator: process.env.SITE_NAME,
            // publish the site author's name (optional)
            author: options.query.site.siteMetadata.author.name,
            // publish the site's base URL in the RSS feed (optional)
            site_url: options.query.site.siteMetadata.siteUrl,
            custom_namespaces: {
              // support additional RSS/XML namespaces (see the feed generation section below)
              cc:
                'http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html',
              dc: 'http://purl.org/dc/elements/1.1/',
              media: 'http://search.yahoo.com/mrss/',
            },
          });
        },
      },
    },
  ],
};

Select articles to include in the RSS feed

This and the next sections are highly specific to your site and are based on how you set up your posts' frontmatter. You will likely need to customize these, but they should hopefully serve as an example of how to generate a feed will all the necessary article data.

The provided GraphQL query filters articles with includeInRSS: true in their frontmatter and sorts the results by publishing date (frontmatter___date), most recent first.

Since articles on my site support a featured (cover) image (featuredImage {...}), we want to select and include it in the feed, along with the alt and title fields.

We also select several other fields from frontmatter and some made available by allMarkdownRemark.

module.exports = {
  plugins: [
    {
      resolve: `gatsby-plugin-feed`,
      options: {
        feeds: [
          {
            query: `
              {
                allMarkdownRemark(
                  filter: { frontmatter: { includeInRSS: { eq: true } } }
                  sort: { order: DESC, fields: [frontmatter___date] },
                ) {
                  nodes {
                    excerpt
                    html
                    rawMarkdownBody
                    fields {
                      slug
                    }
                    frontmatter {
                      title
                      description
                      date
                      featuredImage {
                        image {
                          childImageSharp {
                            gatsbyImageData(layout: CONSTRAINED)
                          }
                        }
                        imageAlt
                        imageTitleHtml
                      }
                      category {
                        title
                      }
                      tags
                    }
                  }
                }
              }            
            `,
          },
        ],
      },
    },
  ],
};

Serializing the necessary data

The last step in the generation process is to merge all the available data and generate complete feed entries. This is achieved during serialization:

module.exports = {
  plugins: [
    {
      resolve: `gatsby-plugin-feed`,
      options: {
        feeds: [
          {
            serialize: ({ query: { site, allMarkdownRemark } }) => {
              // iterate and process all nodes (articles)
              return allMarkdownRemark.nodes.map((node) => {
                // store a few shorthands that we'll need multiple times
                const siteUrl = site.siteMetadata.siteUrl;
                const authorName = site.siteMetadata.author.name;

                // populate the canonical URL
                const articleUrl = `${siteUrl}${node.fields.slug}`;

                // retrieve the URL (src=...) of the article's cover image
                const featuredImage =
                  siteUrl +
                  node.frontmatter.featuredImage?.image
                    .childImageSharp.gatsbyImageData.images.fallback
                    .src;

                // augment each node's frontmatter with extra information
                return Object.assign({}, node.frontmatter, {
                  // if a description isn't provided,
                  // use the auto-generated excerpt
                  description:
                    node.frontmatter.description || node.excerpt,
                  // article link, used to populate canonical URLs
                  link: articleUrl,
                  // trick: you also need to specify the 'url' attribute so that the feed's
                  // guid is labeled as a permanent link, e.g.: <guid isPermaLink="true">
                  url: articleUrl,
                  // specify the cover image
                  enclosure: {
                    url: featuredImage,
                  },
                  // process local tags and make them usable on Twitter
                  // note: we're publishing tags as categories, as per the RSS2 spec
                  // see: https://validator.w3.org/feed/docs/rss2.html#ltcategorygtSubelementOfLtitemgt
                  categories: node.frontmatter.tags
                    .map((tag) => makeTwitterTag(tag))
                    // only include the 5 top-most tags (most platforms support 5 or less)
                    .slice(0, 5),
                  custom_elements: [
                    // advertise the article author's name
                    { author: site.siteMetadata.author.name },
                    // supply an image to be used as a thumbnail in your RSS (optional)
                    {
                      'media:thumbnail': {
                        _attr: { url: featuredImage },
                      },
                    },
                    // specify your content's license
                    {
                      'cc:license':
                        'https://creativecommons.org/licenses/by-nc-sa/4.0/',
                    },
                    // advertise the site's primary author
                    {
                      'dc:creator': renderHtmlLink({
                        href: siteUrl,
                        title: process.env.SITE_NAME,
                        text: authorName,
                      }),
                    },
                    // the main article body
                    {
                      'content:encoded':
                        // prepend the feature image as HTML
                        generateFeaturedImageHtml({
                          src: featuredImage,
                          imageAlt:
                            node.frontmatter.featuredImage?.imageAlt,
                          imageTitleHtml:
                            node.frontmatter.featuredImage?.imageTitleHtml
                        }) +
                        // append the content, fixing any relative links
                        fixRelativeLinks({
                          html: node.html,
                          siteUrl: site.siteMetadata.siteUrl,
                        }),
                    },
                  ],
                });
              });
            },
          },
        ],
      },
    },
  ],
};

There are a few caveats here:

I'm using a few custom functions I wrote, here is the source code:

// Generates HTML for the featured image, to prepend it to the node's HTML
// so that sites like Medium/Dev.to can include the image by default
function generateFeaturedImageHtml({
  src,
  imageAlt,
  imageTitleHtml,
}) {
  const caption = imageTitleHtml
    ? `<figcaption>"${imageTitleHtml}"</figcaption>`
    : '';
  return `<figure><img src="${src}" alt="${imageAlt}" />${caption}</figure>`;
}

// Takes a tag that may contain multiple words and
// returns a concatenated tag, with every first letter capitalized
function makeTwitterTag(tag) {
  const slug = tag
    .replaceAll(/[^\w]+/g, ' ')
    .split(/[]+/)
    .map((word) => upperFirst(word))
    .join('');
  if (slug.length === 0) {
    throw new Error(
      `Invalid tag, cannot create empty slug from: ${tag}`,
    );
  }
  return slug;
}

// Prepends the siteURL on any relative links
function fixRelativeLinks({ html, siteUrl }) {
  // fix static links
  html = html.replace(
    /(?<=\"|\s)\/static\//g,
    `${siteUrl}\/static\/`,
  );

  // fix relative links
  html = html.replace(/(?<=href=\")\//g, `${siteUrl}\/`);

  return html;
}

Most of the extra tags are optional but can be helpful if your RSS gets redistributed further, in which case you'd want users to have as much metadata about you and your site as possible!

We prepend the cover image using generateFeaturedImageHtml, because Medium's API does not support the cover image as a field. However, it parses the first image of the submitted raw content and stores it as a cover. This qualifies as a 'trick' 😊.

Since GatsbyJS uses React routes, all internal links will end up as relative links in the RSS. We fix this by prepending the site URL on any such links (this includes images), using the fixRelativeLinks function. (Shoutout to Mark Shust, who documented how to fix this, first!)

When specifying tags in your frontmatter, define the most relevant ones up-top. For example, Medium only supports five tags, and DevTo supports four, which is why we truncate the number of tags(categories) included in the RSS.

Uploading articles to various distribution platforms

Dev.to

Dev.to supports publishing to DEV Community from RSS. This is very easy to set up once you've done all the above. They will parse your RSS feed and create draft articles from new content. You can then review and publish these in the DEV community!

Medium.com

Medium does not support reading RSS feeds, so in this case, we have to rely on Zapier to crosspost.

I won't repeat the same content here; please see the tutorial linked above for details on how to configure Zapier, and have it crosspost your content, for free!

Hashnode.com

The more recent platform I'm crossposting to is Hashnode.com.

Since I've been using Zapier for Medium for a while, I thought it would be fun to build a Zapier integration. I started with their UI and later exported and completed the code, so that everyone can use it!

If you're a developer, feel free to clone or fork my Zapier integration, deploy it in your account, and use it to crosspost on Zapier.com.

Unfortunately, Zapier requires any officially published integrations to be supported by the destination platform (in this case, Hashnode). I've reached out to ask them to take over this code and make it available for their community, but so far, I haven't heard back.

Conclusion

I hope this helps GatsbyJS users properly define RSS feeds and successfully crosspost their articles to reach a wider audience. If you follow this tutorial and struggle, please reach out via DM!

Thanks!

If you liked this article and want to read more like it, please subscribe to my newsletter; I send one out every few weeks!

Configuring social sharing cards in GatsbyJS

Mihai Bojin — Wed, 11 Aug 2021 22:00:00 +0000

"Photo by Nathan Dumlao on Unsplash"

🔔 This article was originally posted on my site, MihaiBojin.com. 🔔

When a user shares a web link, most applications parse a standard set of meta tags and use the provided information to render a 'card', including an image, a title, and a summary.

A few competing standards exist; the most relevant are Twitter Cards and Facebook's Open Graph Protocol.

When the necessary meta tags are missing, applications such as Twitter, Facebook, Whatsapp, Signal, Telegram, etc., will be unable to render the card. In such cases, they will either display the link as-is or try to infer some parts of the card metadata; in the latter case, your mileage may vary.

Here's an example of a site without social sharing cards metadata:

And here's how my site's homepage is rendered:

Since I wanted my site to display correctly in as many apps as possible, I provided both Open Graph and Twitter Card tags!

I use Gatsby for my site, and unfortunately, it does not have native support for social cards. Also, when I implemented this feature, I was pretty new to GatsbyJS and had to figure out how to load files and process images through GraphQL (childImageSharp, gatsbyImageData, GatsbyImage, StaticImage).

It was rough, to say the least (unclear documentation, competing advice for Gatsby v2 and v3, and my general lack of knowledge).

Fortunately, I prevailed! Here's how:

Load any image via GraphQL

  const { myImage } = useStaticQuery(graphql`
    query {
      myImage: file(relativePath: { eq: "name-of-file.jpg" }) {
        childImageSharp {
          gatsbyImageData(layout: CONSTRAINED)
        }
      }
    }
  `);

This works with images placed under src/images/.

Get a link to a processed image

import { getImage, getSrc } from 'gatsby-plugin-image';

// ...

const img = getImage(image);
const imgSrc = SITE_URL + getSrc(img);

Create an SEO component

Ok, I'll admit, Gatsby does have good SEO support, including some general advice on social cards.

This formed the basis of my configuration, although I had to heavily customize it!

I configured the SEO component to accept a number of inputs:

function SEO({ title, description, lastUpdated, tags, image, imageAlt, absoluteURL }) {
  const img = getImage(image);
  const imgSrc = SITE_URL + getSrc(img);

  let meta = [];

  //...
  return (
    <Helmet
      htmlAttributes={{prefix: 'og: http://ogp.me/ns#'}}
      title={title}
      meta={meta}
      //...
    />
  );
}

Seo.propTypes = {
  title: PropTypes.string.isRequired,
  description: PropTypes.string.isRequired,
  lastUpdated: PropTypes.string.isRequired,
  tags: PropTypes.arrayOf(PropTypes.string),
  image: PropTypes.object.isRequired,
  imageAlt: PropTypes.string.isRequired,
  absoluteURL: PropTypes.string.isRequired,
};
export default SEO

And generated the required meta tags:

Generate Twitter Card meta tags

meta.push({
  name: "twitter:card",
  content: "summary_large_image",
});
meta.push({
  name: `twitter:title`,
  content: title,
});
meta.push({
  name: `twitter:description`,
  content: pageDescription,
});
meta.push({
  property: `twitter:image`,
  content: imgSrc,
});
meta.push({
  name: `twitter:image:alt`,
  content: imageAlt,
});

Generate Open Graph meta tags


meta.push({
  property: `og:type`,
  content: `website`,
});
meta.push({
  property: `og:url`,
  content: absoluteURL,
});
meta.push({
  property: `og:title`,
  content: title,
});
meta.push({
  property: `og:description`,
  content: pageDescription,
});
meta.push({
  property: `og:image`,
  content: imgSrc,
});
meta.push({
  name: `og:image:alt`,
  content: imageAlt,
});
meta.push({
  property: `og:updated_time`,
  content: lastUpdated,
});
meta.push({
  property: "og:image:width",
  content: parseInt(img.width),
});
meta.push({
  property: "og:image:height",
  content: parseInt(img.height),
});

Bringing it all together

And that was that! All that was left was for me to send all the required variables via my page layouts, e.g.

<Seo title="My article" description="Short summary of my article" etc="...">

But is it working?

Testing social sharing card rendering

To test your feature, use the following tools:

Bonus: social sharing buttons

No site would be complete without a feature to allow users to share content on social media easily!

I found an excellent tutorial for integrating social sharing buttons.

It goes a little bit like this:

Create a component and import media from react-share

import {
  TwitterShareButton,
  TwitterIcon,
} from 'react-share';

const ShareButtons = ({
  title,
  url,
  tags,
  twitterHandle,
}) => {

  // remove any non-alphanumeric characters
  const hashtags = tags.map((t) => t.replace(/[^a-zA-Z0-9]+/g, ''));

  return (
    <>
      <TwitterShareButton
        title={title}
        url={url}
        via={twitterHandle}
        hashtags={hashtags}
      >
        <TwitterIcon size={40} round={true} />
      </TwitterShareButton>
    </>
  );
};

(Of course, the above is greatly simplified and only displays Twitter. For my site, I defined more.)

Include the component on your pages

<ShareButtons
  url={SITE_URL + location.pathname}
  title={...}
  description={...}
  tags={[...]}
/>

And here is the result:

Conclusion

I hope you found this short tutorial helpful! As I'm writing it now, it feels pretty straightforward, but at the time, I struggled for hours trying to piece together confusing or incomplete information from various articles and blogs.

Hopefully, this can help you avoid my experience and integrate your social sharing features much faster!

In an ideal world, Gatsby would have come with better, out-of-the-box, social sharing features!

Maybe one day...

If you liked this article and want to read more like it, please subscribe to my newsletter; I send one out every few weeks!

Debugging social sharing cards

Mihai Bojin — Wed, 11 Aug 2021 00:00:00 +0000

"Photo by Sigmund on Unsplash"

🔔 This article was originally posted on my site, MihaiBojin.com. 🔔

I am currently working on the Tools section of my site. While implementing it, I had to reconfigure Tailwind to include CSS classes for additional colors.

Just as I posted this on Twitter, I noticed the Twitter card no longer worked.

Since these used to work, I started investigating.

The first step was to check the Twitter card validator.

The output was not at all helpful. So, I started scouring the web and found various advice on robots.txt, rules for setting up meta tags, and general troubleshooting advice.

Nothing made sense, though, as the problems described in these resources did not affect my site.

However, I found the Facebook debug tool during my searches and promptly used it to test my site.

This tool was better than Twitter's one and showed me more information:

missing og:image
and other properties: og:url, og:type, og:title, etc.

At this point, I had a WTF moment since, as you can see below, my site included valid meta tags...

One thing jumped at me - the HTTP response code was 206. This was unexpected, but in retrospect, quite logical. Twitter and Facebook deal with a large majority of the open web. Allowing their crawlers to process an unlimited input size means opening them up to abuse. Status code 206 means Partial Content, a.k.a that the requested data range was received (but not ALL of the page's content!)

Immediately, I looked at where the meta tags were placed on my site's HTML and discovered that the global CSS preceded my meta tags... all 2 MB of it!

My next thought was to configure where react-helmet places its tags, but no luck 😔.

I thought Gatsby would surely have this feature, but no go!

I did, however, find the gatsby-plugin-split-css plugin that brought down the CSS to 600K, which is low enough that I no longer run into this issue.

SCORE!

I am not entirely happy with this fix - something is off, almost like a code smell. I'm no frontend expert, but I'm starting to suspect why CSS-in-JS and Emotion are a thing...

Long term, I want to try a few things:

get rid of global CSS styles, or at least find a way to include them as a separate file
reorder the head tags
move Tailwind's <style> at the very end of <head>

But for now, this will have to do.

Until next time!

Mihai

If you liked this article and want to read more like it, please subscribe to my newsletter; I send one out every few weeks!

DEV Community: Mihai Bojin

Build products, skip the non-functional requirements

Aiming for object-oriented design elegance

Subscribing to concurrent property updates

Takeaways

Microbenchmarking Java code with JMH

Gradle configuration

Writing a JMH benchmark

"Consuming" results

Further reading

An update on efficient multi-layered key ownership

Hosting my site on Vercel

Prerequisites

Hosting on Vercel

Custom domain

That's all, folks

System Design goal: efficient property reads from multiple sources

A problem of ownership

Sources vs. layers

Retrieving the effective value

Type casting values

How do we establish key ownership?

Conclusion

Create a new open-source Java project using the Gradle build tool

Version Control System

Basic repository information

Build tool

Root directory configuration

Define a global version for all dependencies

Create a subproject

Set a java toolchain version

Generate source JARs and Javadocs

Running tests

Define a version catalog

Code formatting

Coding standard

Smarter static code analysis

Conclusion

Designing a library for reading layered application settings in Java

The first version

The architectural problem

How would a better implementation look like?

Glossary of terms

Design considerations

Conclusion

Crossposting articles from Gatsby to Medium, Dev.to, and Hashnode

Publishing an RSS feed from GatsbyJS

Retrieve site metadata

Setup

Select articles to include in the RSS feed

Serializing the necessary data

Uploading articles to various distribution platforms

Dev.to

Medium.com

Hashnode.com

Conclusion

Configuring social sharing cards in GatsbyJS

Load any image via GraphQL

Get a link to a processed image

Create an SEO component

Generate Twitter Card meta tags

Generate Open Graph meta tags

Bringing it all together

Testing social sharing card rendering

Bonus: social sharing buttons

Create a component and import media from react-share

Include the component on your pages

Conclusion

Debugging social sharing cards