DEV Community: Mustafa Turan

Zero Dependency Open Source

Mustafa Turan — Thu, 02 May 2019 07:29:01 +0000

Writing a zero-dependency open-source library could be the way to bring an immortal contribution to the developer communities. Adding dependencies to your open-source will bring many update/upgrade work while your busy life goes on. And you/community may give-up one day to update those dependencies because of several reasons. Even the zero-dependency library might need language version updates. So, the real intention is keeping as minimum dependency as possible.

Instead of putting dependencies, put interfaces to the library and let the library users to use their own choice of libraries by implementing your interface. And share sample implementations with a dependent library as an example instead of directly adding implementation. (What is being defined is also a part of hexagonal-architecture pattern for software design which makes the library optioned without implementation choices.)

4 steps to provide zero-dependency library:

Implement your library functionality
When you need an external dependency(external library from another author), create an interface instead of adding the dependency
Document a sample implementation with/without dependency (but do not implement)
Let the library user decide what to use & leave the interface implementation to the library user.

This could be the way to bring an immortal contribution to the developer communities. Since there will be no dependency, the library can be used for new projects/libraries with different implementation details. Even after years, new implementations would be possible without updating the library...

Before taking action, it is highly recommended also reading DHH's Rails is Omakase post. More ideas are highly welcome. Let's discuss.

Decoupled Package Communication in Go

Mustafa Turan — Sun, 28 Apr 2019 23:14:24 +0000

Go is very powerful language for software development with its simplicity, concurrency, first-class functions and tools.

In Go, usually, a package is responsible only for one thing. Applications grows with several packages with their own responsibilities. When the number of packages in the project and responsibilities of the project increase, it is possible to mess up on communication with other packages. (In here, the communication means calling functions on different packages/modules on the flow with/without waiting result. Like incrementing counter metric when an order received/full-filled)

One of the known ways of decoupling on communication is the event driven software designs. Usually, using an external event/message-bus server is the way distributing the responsibility to internal/external services. But this also brings another external component to your stack. And each stack comes with its own problems.

For small services, packages/libraries, and embedded systems adding an external bus could be unnecessary addition. To overcome this problem, adding an internal message-bus package to your package may help. And when things getting bigger, it is possible to communicate with this internal message-bus to external message-bus services.

Assume that, a project needs to do several things on user registration like; incrementing counters, sending email, logging, and/or sending webhook to external service; instead of directly calling each of the functionalities directly, a message-bus can help to execute multiple functionalities asynchronously without knowing the caller.

Decoupled packages help us to write cleaner code, focus on only one thing at a time. With bus Go package, you can get benefit of real message bus system and write decoupled packages easily. The bus package allows any handler to listen any events, without knowing who generates the event. Thus, packages can communicate without depending on each other. Moreover, it is very easy to substitute a consumer function. As long as the new function understands the Event struct that are being sent and received, the other functions will never know.

Get Started With `Bus` Go Package in 4 Steps

Configure

The bus package requires a unique id generator to assign ids to events. You can write your own function to generate unique ids or use a package that provides unique id generation functionality. Here is a sample configuration using monoton id generator:

import (
    "github.com/mustafaturan/bus"
    "github.com/mustafaturan/monoton"
    "github.com/mustafaturan/monoton/sequencer"
)

func init() {
    // configure id generator (it doesn't have to be monoton)
    node        := uint(1)
    initialTime := uint(0)
    monoton.Configure(sequencer.NewMillisecond(), node, initialTime)

    // configure bus
    if err := bus.Configure(bus.Config{Next: monoton.Next}); err != nil {
        panic("whoops")
    }
    // ...
}

Register Event Topics

To emit events to the topics, topic names need to be registered first:

func init() {
    // ...
    // register topics
    bus.RegisterTopics("order.received", "order.fulfilled")
    // ...
}

Register Event Handlers

To receive topic events you need to register handlers; A handler basically requires two values which are a Handle function and topic Matcher regex pattern.

handler := bus.Handler{
    Handle: func(e *Event){ 
        // fmt.Printf("Event: %+v %+v\n", e, e.Topic)
        // do something
        // NOTE: Highly recommended to process the event in an async way
    },
    Matcher: ".*", // regex pattern that matches all topics
}
bus.RegisterHandler("a unique key for the handler", &handler)

Emitting Events

The bus package provides a generic event struct to standardize what an event is for all of its handlers. Before delivering an event, it is good to know what an event is for the bus package. If this is your fist moments with the library, it is highly recommended to check the data structure of the Event struct.

txID  := "some-transaction-id-if-exists" // if it is blank, bus will generate one
topic := "order.received"                // event topic name (must be registered before)
order := make(map[string]string)         // interface{} data for event

order["orderID"]     = "123456"
order["orderAmount"] = "112.20"
order["currency"]    = "USD"

bus.Emit(topic, order, txID) // emit the event for the topic

Congratulations!!! You emitted your first event with the bus package. Now, the all matched handlers will receive the same event. Please try yourself and share your experience as comment.

Also a sample project using bus package can be found on bus-sample-project Github.

mustafaturan / bus

🔊Minimalist message bus implementation for internal communication with zero-allocation magic on Emit

🔊 Bus

Bus is a minimalist event/message bus implementation for internal communication It is heavily inspired from my event_bus package for Elixir language.

API

The method names and arities/args are stable now. No change should be expected on the package for the version 3.x.x except any bug fixes.

Installation

Via go packages: go get github.com/mustafaturan/bus/v3

Usage

Configure

The package requires a unique id generator to assign ids to events. You can write your own function to generate unique ids or use a package that provides unique id generation functionality.

The bus package respect to software design choice of the packages/projects. It supports both singleton and dependency injection to init a bus instance.

Hint: Check the demo project for the singleton configuration.

Here is a sample initilization using monoton id generator:

import (
    "github.com/mustafaturan/bus/v3"
    "github.com/mustafaturan/monoton/v2"
    "github.com/mustafaturan/monoton/v2/sequencer"
)
func NewBus() *bus.Bus {
    // configure id generator (it doesn't have

…

View on GitHub

Persisting event data to Postgres using GenStage and EventBus

Mustafa Turan — Tue, 31 Jul 2018 08:10:52 +0000

One of the ways to consume EventBus events is implementing GenStage consumers. GenStage handles backpressure easily with configurable workers. event_bus_postgres library uses GenStage to persist event_bus events to postgres DB with batch insert.

How it works

+-----+
|     |                                         GEN STAGE
|     |        EVENTBUS      +------------------------------------------+
|     |        CONSUMER      |                   +---+                  |
|     |        +-----+       |                   |   |                  |
|     |        |     |       |                   |   |          +---+   |
|     |        |  E  |       |                   |   |          |   |   |
|     |        |  v  |       |                   |   |          |   |   |
|     |        |  e  |       |                   |   |          |   |   |
|  E  |        |  n  |       |                   | E |          |   |   |
|  l  |        |  t  |       |  +-------+        | v |          |   |   |
|  i  | topic  |  B  |  topic   |       |        | e |          |   |
|  x  |   +    |  u  |    +     |   Q   |        | n |          | B |       +--+
|  i  |event_id|  s  | event_id |   u   |   ask  | t |    ask   | u |       |  |
|  r  |------->|  .  |--------->|   e   |<-------|   | <--------| c | BATCH |  |
|     |        |  P  |          |   u   |------->| M | -------->| k |------>|DB|
|  E  |        |  o  |          |   e   |   pull | a |    pull  | e | INSERT|  |
|  v  |        |  s  |          |       |        | p |          | t |       |  |
|  e  |        |  t  |       |  +-------+        | p |          |   |   |   +--+
|  n  |        |  g  |       |  GENSTAGE         | e |          |   |   |
|  t  |        |  r  |       |  PRODUCER         | r |          |   |   |
|  B  |        |  e  |       |                   |   |          |   |   |
|  u  |        |  s  |       |                   |   |          |   |   |
|  s  |        +-----+       |                   |   |          |   |   |
|     |<-----------------------------------------|   |          +---+   |
|     |                      |    fetch_event/1  |   |         CONSUMER |
|     |                      |                   |   |                  |
+-----+                      |                   +---+                  |
                             |                  CONSUMER                |
                             |                  PRODUCER                |
                             +------------------------------------------+

Components

EventBus
Message bus for Elixir; it publishes event_id and topic data to topic subscribers.

EventBus.Postgres
Message bus event consumer; it pushes event_id and topic to the EventBus.Postgres.Queue

Queue
GenStage producer; it is a simple queue implementaion

EventMapper
GenStage producer-consumer; it pulls/dequeues from EventBus.Postgres.Queue, and fetch original event from EventBus and then convert data into Ecto model.

Bucket
GenStage consumer; it pulls/dequeues from EventBus.Postgres.EventMapper and batch insert data to Postgres DB.

Source code: https://github.com/otobus/event_bus_postgres

Decoupled Modules with Elixir EventBus

Mustafa Turan — Mon, 23 Jul 2018 07:48:41 +0000

Elixir EventBus is a library that allows different modules to communicate with each other without knowing about each other. A module/function can create an Event struct, and deliver to the EventBus without knowing which modules will consume.

Modules can also listen to events on the EventBus, without knowing who sent the events. Thus, modules can communicate without depending on each other. Moreover, it is very easy to substitute a consumer module. As long as the new module understands the Event struct that is being sent and received, the other modules will never know.

Get started with decoupled modules in 4 steps

Decoupled modules help us to write clear code, focus on only one thing at a time. With EventBus library, you can get benefit of real event bus system and write decoupled modules easily.

Registering event topics

On your app init, or anytime before delivering the events we need to register our topics. For example, when we receive a payment, we will create checkout_completed event.

EventBus.register_topic(:checkout_completed)
EventBus.register_topic(:checkout_failed)

Delivering events

EventBus.EventSource module provides notify helper to deliver events without modifying your current code.

defmodule Order do
  ...
  use EventBus.EventSource
  ...
  def checkout(params) do
    event_params = %{topic: :checkout_completed, error_topic: :checkout_failed}
    EventSource.notify(event_params) do
      ... # process the payment as usual in here and if errors then return {:error, _} tuple
    end
  end
end

Subscribing consumers to topics

Assume that we have several operations waiting on success and failure conditions. Let's subscribe the consumers to the event topics.

EventBus.subscribe({StockUpdateService, ["^checkout_completed$", "^new_stock_arrived$", ...]})
EventBus.subscribe({CargoService, ["^checkout_completed$, "^cargo_dispatched$", "^new_stock_arrived$", ...]})
EventBus.subscribe({EmailService, ["^user_registered$", "^checkout_completed$", ...]})
EventBus.subscribe({EventBus.Logger, [".*"]})

Consuming events

EventBus consumers are just modules so they can be implemented by any kind of consumer strategy including GenStage(event_bus_postgres as sample), pooling, DynamicSupervisors, global consumers, spawn et al. So, depending on your use case, implement your consumers with the best suitable strategy. (But any kind of blocker strategies are not recommended!)

defmodule CargoService do
  use GenServer
  ...
  def process({topic, id}) do
    GenServer.cast(__MODULE__, event_shadow)
  end

  def handle_cast({topic, id}, state) do
    payment_data = EventBus.fetch_event_data({topic, id})
    # do sth with payment_data
    ...
    # mark event as completed for this consumer
    EventBus.mark_as_completed({CargoService, topic, id})
    {:noreply, state}
  end
end

Here is another consumer that needs all event structure and consumes with spawning:

defmodule StockUpdateService do
  ...
  def process({topic, id}) do
    spawn(fn ->     
      event = EventBus.fetch_event({topic, id})
      # do sth with event.data and/or any other event attributes
      ...
      # mark event as completed for this consumer
      EventBus.mark_as_completed({StockUpdateService, topic, id})
    end)
  end
end

About the EventBus library

EventBus library is a traceable, extendible, fast, and memory friendly pure Elixir implementation without any external dependency in the pocket.

Traceable

EventBus library comes with good optional attributes to provide traceability. When a consumer receive the topic and event_id, it can fetch structure with fetch_event/1 function. And if you use EventBus.EventSource helper, the optional fields are set automatically for you. Here is the structure of an Event model:

%EventBus.Model.Event{
  id: String.t | integer(), # required
  transaction_id: String.t | integer(), # optional
  topic: atom(), # required
  data: any() # required,
  initialized_at: integer(), # optional
  occurred_at: integer(), # optional
  source: String.t(), # optional
  ttl: integer() # optional
}

Extendible

EventBus library allows subscribing events with regex patterns, which allows consumers to subscribe new topics automatically. With this feature, it allows extending the system asynchronously with generic consumers like event_bus_logger, generic Postgresql event store, event_bus_metrics UI, and so on.

Fast by design

EventBus library uses builtin memory store ETS effectively to get benefits of concurrent reads and writes. It doesn't block majority of the read operations, and allows concurrent reads to fetch Event data.

It applies queueing theory to handle inputs. And almost all implementation data accesses have O(1) complexity.

Memory friendly

EventBus library delivers only topic and event_id to the consumers and keeps the original event data in the topic's Event table. So, only when consumers are able to process the event, then they fetch the event data.

Distributed?

Even though the main intention to create this library is to use for internal module communications, it is possible to deliver events across nodes. One of the ways to delivery is sending message to connected other nodes as usual Elixir/Erlang way:

for node <- Node.list() do
  Node.spawn(node, EventBus, :register_topic, [:checkout_completed])
  Node.spawn(node, EventBus, :notify, [%{EventBus.Model.Event{..}}])
end

Note: EventBus library is not in the same class distributed solutions like Apache Kafka, RabbitMQ, or Redis Streams.

Original story on Medium: https://medium.com/elixirlabs/decoupled-modules-with-elixir-eventbus-a709b1479411

The source code of EventBus library: https://github.com/otobus/event_bus

When to give a star to an open source library?

Mustafa Turan — Tue, 17 Jul 2018 04:23:00 +0000

Giving a star to an open source library means to a lot to the developers and consumers of the library. Because this will give the developers a positive impact to make the library better. It is definitely a way to tell gently to the developers, there are people out there who use that library.

As a consumer of the library

If you are using an open source library, go ahead and find its Github/Gitlab (or any version control source) and give it a star. If you need to use a high-quality generic library, you should help to bring new people to the library. Starring will bring quality and aliveness to the library. One of the people that you bring, might be the next contributor to the library.

To support the work or idea

You find an open source library in the following interests:

Like the idea
Plan to use it in your current or next project
Like the software design of the library
A topic of interests like distributed systems, data structures, useful links

Don't forget that any positive starring activity will bring more people to like the library with the social impact. This positive impact will also bring more contributors to the library. And will prevent the library to get out-dated.

These are only my ideas while starring a library, please do not hesitate to share yours. I will be happy to include in the story including your name.

DEV Community: Mustafa Turan

Zero Dependency Open Source

Decoupled Package Communication in Go

Get Started With Bus Go Package in 4 Steps

Configure

Register Event Topics

Register Event Handlers

Emitting Events

mustafaturan / bus

🔊Minimalist message bus implementation for internal communication with zero-allocation magic on Emit

🔊 Bus

API

Installation

Usage

Configure

Persisting event data to Postgres using GenStage and EventBus

How it works

Components

Decoupled Modules with Elixir EventBus

Get started with decoupled modules in 4 steps

Registering event topics

Delivering events

Subscribing consumers to topics

Consuming events

About the EventBus library

Traceable

Extendible

Fast by design

Memory friendly

Distributed?

When to give a star to an open source library?

As a consumer of the library

To support the work or idea

Get Started With `Bus` Go Package in 4 Steps