DEV Community: Tamimi

Golang and Event-Driven Architecture

Tamimi — Wed, 06 Apr 2022 18:15:58 +0000

On March 31 2022, I gave a talk at Conf42:Golang about using Go in an event-driven architecture entitled "Gopher in an Event Driven Playground". You can check out the talk here or read along to know more about event-driven systems and how different messaging protocols are used in Go!

What's all the hype about?

For all of us Go enthusiasts out there, we truly understand the beauty of using Go in applications and microservices because of its lightweight, high performance, and elegant syntax, to name just a few (let the debate start! 🔥).

So imagine using your favourite programming language (yes, that is Go) with your favourite architecture (yes, that event-driven architecture) - where do you start? Carry on and read further to explore.

Hold on, what is an event-driven architecture (EDA)?

Glad you asked! There are loads of resources online that talk about EDA. At its core, an event-driven architecture involves asynchronous communication between applications via publishing and subscribing to events over an event broker using a messaging protocol.

Some examples of messaging protocols include open standard and open source protocols such as MQTT, AMQP, and JMS.

To delve more into event-driven architecture and explore different use-cases, check out this page on What is Event-Driven Architecture.

I also wrote a blog post talking about how I built an event-driven NodeJS app on real-time COVID-19 data streams, highlighting the advantages of event-driven architecture, if you want to explore it further:

How I built an event-driven NodeJS app on realtime COVID-19 data streams

Tamimi for Solace Developers ・ Apr 23 '20

#covid19 #javascript #opensource #node

fmt.Println("Show me the CODE!")

Alright, we did a high level overview on what EDA is but let's get to business with some hands-on code. Let me show you how to use:

The Paho MQTT library for Golang
The PubSub+ Messaging API for Go

Note that all the source code can be found in this open source Github repo:

TamimiGitHub / conf42-golang-eda

Conf42 Golang 2022: Gopher in an Event-Driven Playground

Talk

Talk found on YouTube here https://www.youtube.com/watch?v=XBkBpFYAHiw&feature=youtu.be

Source Code

This repo contains source code to showcase how to use the

Paho MQTT library to publish and subscribe to
Solace PubSub+ Messaging API for Go to publish messages

How to run

go run solace_publisher.go
go run paho_mqtt.go

Note: Update the hostname, username, password, and vpn name to your broker's details

Extra Resources

Solace PubSub+ Messaging API for Go samples

View on GitHub

Hands-on 👩‍💻🧑‍💻

Let's start with creating a new directory and initializing go mod. In a new terminal window, execute the following commands:



mkdir funWithEDA && cd "$_"
go mod init GoEDA

MQTT

As per definition: "MQTT is an OASIS standard messaging protocol for the Internet of Things (IoT). It is designed as an extremely lightweight publish/subscribe messaging transport that is ideal for connecting remote devices with a small code footprint and minimal network bandwidth"

The Eclipse Paho project provides open source, mainly client side, implementations of MQTT in a variety of programming languages. For today's fun coding session, we will be using the Eclipse Paho MQTT Go client.

Install the Paho MQTT Go library



go get github.com/eclipse/paho.mqtt.golang

Create a new file and open it with your favourite IDE. I named my file go_mqtt.go.

Initialize the file and import the necessary libraries



package main

import (
    "fmt"
    "time"

    mqtt "github.com/eclipse/paho.mqtt.golang"
)

In your main() function, start with

Creating a new MQTT client and configure it
Add callback functions for
1. Received Messages
2. Successful broker connection
3. Lost broker connection
Subscribe to a topic
Publish on a topic



func main() {
    var broker = "public.messaging.solace.cloud"
    var port = 1883
    opts := mqtt.NewClientOptions()
    opts.AddBroker(fmt.Sprintf("tcp://%s:%d", broker, port))
    opts.SetUsername("conf42")
    opts.SetPassword("public")
    opts.SetDefaultPublishHandler(messageHandler)
    opts.OnConnect = connectHandler
    opts.OnConnectionLost = connectLostHandler
    client := mqtt.NewClient(opts)
    if token := client.Connect(); token.Wait() && token.Error() != nil {
        panic(token.Error())
    }

    sub(client)
    publish(client)

    client.Disconnect(250)
}

We will need to define the callback functions as follows



var messageHandler mqtt.MessageHandler = func(client mqtt.Client, msg mqtt.Message) {
    fmt.Printf("Received message: %s from topic: %s\n", msg.Payload(), msg.Topic())
}

var connectHandler mqtt.OnConnectHandler = func(client mqtt.Client) {
    options := client.OptionsReader()
    fmt.Println("Connected to: ", options.Servers())
}

var connectLostHandler mqtt.ConnectionLostHandler = func(client mqtt.Client, err error) {
    fmt.Printf("Connect lost: %v", err)
}

It is important to note that these functions will be triggered "on" particular actions. So for example, the messageHandler function will be triggered whenever the MQTT client receives a message via mqtt.MessageHandler.

And finally, define your publish and subscribe functions as follows:



func publish(client mqtt.Client) {
    num := 10
    for i := 0; i < num; i++ {
        text := fmt.Sprintf("Message %d", i)
        token := client.Publish("conf42/go", 0, false, text)
        token.Wait()
        time.Sleep(time.Second)
    }
}

func sub(client mqtt.Client) {
    topic := "conf42/#"
    token := client.Subscribe(topic, 1, nil)
    token.Wait()
    fmt.Printf("Subscribed to topic: %s\n", topic)
}

And that's it! Run the application and observe the results



go run go_mqtt.go

Solace PubSub+ Messaging API for Go

Now that you are an expert on messaging concepts with Go, let's take it up a notch and delve into a more advanced messaging API! We'll be using the Solace PubSub+ Messaging API for Go.

Install the Solace Native Go API



go get solace.dev/go/messaging

Create a new file and open it with your favourite IDE. I named my file solace_publisher.go.

Import the necessary packages



package main

import (
    "fmt"
    "os"
    "os/signal"
    "strconv"
    "time"

    "solace.dev/go/messaging"
    "solace.dev/go/messaging/pkg/solace/config"
    "solace.dev/go/messaging/pkg/solace/resource"
)

Define the configuration parameters to connect to the Solace PubSub+ Broker



// Configuration parameters
    brokerConfig := config.ServicePropertyMap{
        config.TransportLayerPropertyHost:                "tcp://public.messaging.solace.cloud",
        config.ServicePropertyVPNName:                    "public",
        config.AuthenticationPropertySchemeBasicUserName: "conf42",
        config.AuthenticationPropertySchemeBasicPassword: "public",
    }

Initialize a messaging service and connect to it



messagingService, err := messaging.NewMessagingServiceBuilder().FromConfigurationProvider(brokerConfig).Build()

if err != nil {
    panic(err)
}

// Connect to the messaging serice
if err := messagingService.Connect(); err != nil {
    panic(err)
}

Build a Direct Message Publisher and start it



//  Build a Direct Message Publisher
directPublisher, builderErr := messagingService.CreateDirectMessagePublisherBuilder().Build()
if builderErr != nil {
    panic(builderErr)
}

// Start the publisher
startErr := directPublisher.Start()
if startErr != nil {
    panic(startErr)
}

Publish messages in a loop



msgSeqNum := 0

//  Prepare outbound message payload and body
messageBody := "Hello from Conf42"
messageBuilder := messagingService.MessageBuilder().
    WithProperty("application", "samples").
    WithProperty("language", "go")

// Run forever until an interrupt signal is received
go func() {
    for directPublisher.IsReady() {
        msgSeqNum++
        message, err := messageBuilder.BuildWithStringPayload(messageBody + " --> " + strconv.Itoa(msgSeqNum))
        if err != nil {
            panic(err)
        }
        topic := resource.TopicOf("conf42/solace/go/" + strconv.Itoa(msgSeqNum))

        // Publish on dynamic topic with dynamic body
        publishErr := directPublisher.Publish(message, topic)
        if publishErr != nil {
            panic(publishErr)
        }

        fmt.Println("Published message on topic: ", topic.GetName())
        time.Sleep(1 * time.Second)
    }
}()

This is the final application.



package main

import (
    "fmt"
    "os"
    "os/signal"
    "strconv"
    "time"

    "solace.dev/go/messaging"
    "solace.dev/go/messaging/pkg/solace/config"
    "solace.dev/go/messaging/pkg/solace/resource"
)

func main() {
    // Configuration parameters
    brokerConfig := config.ServicePropertyMap{
        config.TransportLayerPropertyHost:                "tcp://public.messaging.solace.cloud",
        config.ServicePropertyVPNName:                    "public",
        config.AuthenticationPropertySchemeBasicUserName: "conf42",
        config.AuthenticationPropertySchemeBasicPassword: "public",
    }

    messagingService, err := messaging.NewMessagingServiceBuilder().FromConfigurationProvider(brokerConfig).Build()

    if err != nil {
        panic(err)
    }

    // Connect to the messaging serice
    if err := messagingService.Connect(); err != nil {
        panic(err)
    }

    //  Build a Direct Message Publisher
    directPublisher, builderErr := messagingService.CreateDirectMessagePublisherBuilder().Build()
    if builderErr != nil {
        panic(builderErr)
    }

    // Start the publisher
    startErr := directPublisher.Start()
    if startErr != nil {
        panic(startErr)
    }

    msgSeqNum := 0

    //  Prepare outbound message payload and body
    messageBody := "Hello from Conf42"
    messageBuilder := messagingService.MessageBuilder().
        WithProperty("application", "samples").
        WithProperty("language", "go")

    // Run forever until an interrupt signal is received
    go func() {
        for directPublisher.IsReady() {
            msgSeqNum++
            message, err := messageBuilder.BuildWithStringPayload(messageBody + " --> " + strconv.Itoa(msgSeqNum))
            if err != nil {
                panic(err)
            }
            topic := resource.TopicOf("conf42/solace/go/" + strconv.Itoa(msgSeqNum))

            // Publish on dynamic topic with dynamic body
            publishErr := directPublisher.Publish(message, topic)
            if publishErr != nil {
                panic(publishErr)
            }

            fmt.Println("Published message on topic: ", topic.GetName())
            time.Sleep(1 * time.Second)
        }
    }()

    // Handle OS interrupts
    c := make(chan os.Signal, 1)
    signal.Notify(c, os.Interrupt)

    // Block until an OS interrupt signal is received.
    <-c

    // Terminate the Direct Publisher
    directPublisher.Terminate(1 * time.Second)
    fmt.Println("\nDirect Publisher Terminated? ", directPublisher.IsTerminated())
    // Disconnect the Message Service
    messagingService.Disconnect()
    fmt.Println("Messaging Service Disconnected? ", !messagingService.IsConnected())

}

And that's it! Run the publisher as follows



go run solace_publisher.go

Note: You can use the Solace PubSub+ TryMe tab to connect to the broker and subscribe to any topic you want. Subscribe to topic conf42/solace/>

Bonus! You can run a subscriber application in another terminal, subscribe to conf42/solace/>, and observe the results. You can find more about this on the SolaceSample github org.

Challenge! Run a publisher and a subscriber in the same application! 🤯

SolaceSamples / solace-samples-go

Getting Started Samples for the Solace Go API.

Solace PubSub+ Messaging API for Go

This repository contains sample code to showcase how the Solace PubSub+ Go API could be used. You can find:

/patterns --> runnable code showcasing different message exchange patters with the PubSub+ Go API.
/howtos --> code snippets showcasing how to use different features of the API. All howtos are named how_to_*.go with some sampler files under sub-folders.

Environment Setup

Install the latest supported version of Go from https://go.dev/doc/install. Currently, the samples are run and tested against Go v1.17.
Install the Solace PubSub+ Messaging API for Go into the root of this directory. This is done by either
1. run go get solace.dev/go/messaging
2. Downloading the API archive from the Solace Community
3. Clone the source code into the root of this repo

Run Patterns

[Skip if existing] Initialize the directory with Go modules go mod init SolaceSamples.com/PubSub+Go.
[For local development] Modify the go.mod file…

View on GitHub

Note on protocol interoperability

Using the Solace PubSub+ Event Broker, you can leverage the protocol translation and interoperability features. Even though the Go API was used to connect to the broker, you can use other Solace APIs and/or open standard protocols to connect to the broker and still have the microservices send and receive messages to each other.

Why use Solace native APIs??

You might be wondering, "Why should I use Solace native APIs as opposed to an open-standard open-source messaging protocol like MQTT?" Glad you asked! Check out this blog that talks about the advanced features you can get access to through using Solace's API?

Use Solace Messaging APIs for Advanced Event Mesh Features and the Best Performance | Solace

Using Solace’s native messaging APIs gives you access to the full range of PubSub+ Event Broker features and the best performance.

solace.com

This is all cool. How can I know more 👀

In summary, here are some quick links

A Solace Community post listing a bunch of resources
Solace PubSub+ Messaging API for Go Samples
Blog post on the motivation behind a native Solace Go API
Conf 42 talk on YouTube

If you want to see my colleagues and I during a live streaming session talking about EDA, the Solace PubSub+ Messaging API for Go, and coding (on LIVE television!), check out this event 👀

What about you?

I am curious to hear from the go community!

What is it about Golang that you enjoy?
Have you ever used Golang in an event-driven application?
What messaging protocols and/or open standards have you used?
What message brokers do you use?

Open to all sort of discussions, comments, and questions!

And P.S. You can check out the other talks from Conf42 at conf42.com/golang2022

3 steps to configure Vanilla Forum with Orbit.love

Tamimi — Tue, 14 Dec 2021 18:27:31 +0000

What's orbit
- Integrations
- Where's my integration?
Goal
1. Extract VF Members
- Generate VF Token
- Script
- Issues and Improvements
2. Import Members to Orbit
3. Configure Zapier
- VF New User Webhook
- Zapier Webhook Actions
- Orbit Step
Tips and Tricks

What's Orbit?

Orbit is a platform that helps with understanding your online community. If you are a developer advocate or a community manager then you can consider this platform as the mission control for your community that consolidates and curates all your members and their activities from different places into one place. It's pretty nifty! At Solace we use Orbit to better understand our community members and where they gravitate to the most.

The Orbit Model Framework helps with segmenting community members into different categories depending on their activities and interactions with the community. This framework is based on associating a weighted score for different kind of activities on different platforms. It's important to understand this model for whats coming next 👀

Integrations

The nice thing about Orbit is the ability to connect multiple different integration from their natively supported integrations. Meaning that any activity that happens on the platform, for example a tweet or a PR on a github repo, can be ported into your Orbit workspace, associate with the user, and given a particular weight for scoring purposes.

Where's my integration at?

If your members are hanging out in a platform that is not natively supported by Orbit as an integration, worry not! Since Orbit was built with developers in mind, they naturally have APIs that facilitates the interaction with the platform via importing users and adding activities.

We use Vanilla Forum (VF) as the community forum solution at Solace. Orbit, however, doesn't have a native integration with VF. What's the solution? lets jump into the API!

Goal

The problem we wanted to solve is the following:

Extract all the already existing community members from Vanilla Forum
Import the members into Orbit
Integrate any new user activity from VF to Orbit: New Users, Comments, New Discussions.

Luckily, VF has an API we can leverage to extract information from the forum. They also have support for Webhooks to facilitate integrations between sites and services whenever any activity happens in the forum.

Let's cut the chase and get to the core of it!

1. Extract existing members. Done once!

Generate your VF Token

To use the VF APIs, you will first need to generate an Access Token. To do so , follow the steps on Authentication With Personal Access Tokens

Write your script

I wrote a simple nodejs application to extract all the members from VF via their APIs into a CSV file. You can use your programming language of choice.

async function getUsers(page) {
  const baseURL = `https://solace.community/api/v2/users?expand=extended&page=${page}`;
  const header_config = {
    method: "GET",
    headers: {
      Authorization: `Bearer <Insert Token Here>`,
    },
  };

  let res = await fetch(baseURL, header_config).catch((err) => {
    throw new Error(`Error fetching content from ${baseURL}. ${err}`);
  });

  if (!res.ok) {
    throw new Error(`Response status from ${baseURL}: ${res.status}`);
  }

  let body = await res.json();

  if (body.length === 0) {
    throw new Error(`No content in page ${page}`);
  }
  // CSV list 
  let content = [];

  body.map((user) => {
    let isEmployee = user.roles[1] ? user.roles[1].name === "Employee" : false;
    content.push(`${user.name}, ${user.email},,,${user.extended.LinkedInprofileoptional},,${user.extended.CompanyName}, ${user.photoUrl}, "VF", ${isEmployee},${user.extended.JobTitleoptional}\n`);
  });

 content.forEach((l) => {
    fs.appendFile("users.csv", l, (err) => {
      if (err) {
        console.error(err);
        return;
      }
    });
  });
}

let content = ['name','email','github','twitter','linkedin','discorse','company','avatar_url','tags','teammate','title','\n'];

fs.appendFile("users.csv", content)

for (let i = 0; i < 39; i++) {
  getUsers(i);
}

Notes

The script expects an empty users.csv file to exist
The VF API returns the users in several pages. I manually traversed the pages since I knew how many pages I am dealing with for the initial import. You can check out VF's Pagination section for more information
The generated CSV file follows the Orbit CSV import guidelines. Use this Template if you want to add other tags associated with the members
The script adds a VF tag to the users to make sure we can differentiate our forum users from other users. You can add whatever tags your team decides on adding. This is a comma separated list

Issues and Improvements

The Orbit import format does not allow for specifying the joined_at parameter, so all your imported members will have the joined date as the day you import the members to your workspace
Pagination could be improved in the script above

Now that you have your CSV file ready, lets import the members to Orbit!

2. Import your CSV file to orbit

Navigate to the setting section in your workplace
Import your CSV File
Wait for the import to finish

We're ready to configure Activities! 👇

3. Connect Webhooks to Orbit Activities

Vanilla Forum has support for Webhooks to integrate with other websites and services. You can use integration services like Zapier or SendGrid to leverage the VF Webhooks. In this tutorial I will be using Zapier and assume you already have an account.

Vanilla Forum new user webhook

Create a new zap and search for the Vanilla Forum app
Configure the connection parameters
Choose the event you want to trigger the zap with. I chose new user added
Test the trigger and observe the output

Note

It's important to note that some metadata are not included in the body of the webhook such as the company name and Linkedin profile. Depending on what other parameters you have configured in your VF sign up form, you might want to include other metadata associated with every user.

🚀 Zapier Webhook Actions for the rescue!

Zapier Webhook Actions

We will need to add another step after the Vanilla Forum step. This will leverage VF's REST APIs to query further parameters for the new user.

Click on the '+' sign to add another step
Search for "Webhooks By Zapier"
Under the Action event, choose GET
Under the Setup Action, fill in the URL https://<your_domain>/api/v2/users/{userID}?expand=extended
- Note that the userID is obtained from the previous step. The expand=extended parameter is needed to get extra metadata for the newly added user. Check Expanding User Profile Extender Fields
Add an Authorization Header under the Headers section with a value of Bearer <Insert VF Token>
Click continue and Test the integration

Orbit Step: Add new user

Now that we have all the metadata associated with creating a new user, we are ready to add the user to Orbit

Click on the '+' sign to add a new step
Search for orbit
Choose "Create a new Member" from the actions events
Fill in the information either statically (by typing it in) or dynamically (from the output of the previous step)
Dont forget to add a tag for every activity! For Example VF for a new user, VF-Discussion for a new discussion, and VF-Comment for a new comment
Click continue and test action
Turn on the Zap

And you're done! Now everytime a new user joins your community forum, a webhook is triggered to Zapier, and the Zapier configuration will add that user to Orbit leveraging Orbit's APIs and the details in your step.

You can follow the same steps for new discussions and new comments webhooks. Make sure you add the right tags for every activity!

Tips and Tricks

Adding all the tags and information associated with every activity is extremely crucial. This comes to play and becomes helpful when you want to filter members or activity based on a particular tag
You can modify the weight for every activity. This can be fine tuned and decided by you and your team.

How to query specific line(s) from a file using GitHub APIs

Tamimi — Mon, 10 May 2021 19:21:07 +0000

Permalinks

One of the neat features about GitHub is the generation of permalinks urls that highlights a range of line numbers. This is particularly useful when you want to highlight a chunk of code or a line to show it to someone else.

To do so, just navigate to the file you want on github and click on one line, hit the Shift key and click on the end line for the code block. This generates a URL with the highlighted block content that you can then use for sharing.

Edit: feedback based on comment
Clicking on the start line number will show the ellipses button ... hold down shift and click on the end line number. Then click on the overflow button and select Copy permalink

Tip:
You can just add #L{start_line}-L{end_line} to the URL as follows
https://github.com/{org_name}/{repo_name}/{path_to_file}#L{start_line}-L{end_line}

Note: You can perhaps do the same thing with other version control platforms like bitbucket or gitlabs but I have not checked.

Show me the API!

I was recently working on a project that dynamically generates code snippet from files hosted on github. Naturally, I referred to GitHub's APIs to query file content. Simple right? Well let's dig deeper into it.

Navigating through Github's APIs, I was disappointed to find out there is no API that query a specific line number(s) from a file. The Search API only allows to query the full content of the file. No problem, let's dig into coding logic for post processing.

Show me the code!

To bypass this limitation, I was able to fetch the content of the file using the SearchAPI as follows

async function fetchContent(orgName, repoName, file, ref) {
  const baseURL = `https://api.github.com/repos/${orgName}/${repoName}/contents/${file}?ref=${ref}`
  let res = await fetch(baseURL).catch(err => {
    throw new Error(`Error fetching content from ${baseURL}. ${err}`)
  })

  if (!res.ok){
    throw new Error(`Response status from ${baseURL}: ${res.status}`)
  }

  let body = await res.text()
  // Content body from github is base64 encoded
  return Base64.decode(JSON.parse(body).content)
}

let content = await fetchContent(orgName, repoName, file, ref)

This brings in the whole file as text with new line delimiters presented as \n. You see where I'm going with this 👀

From here, you can split() the text content using \n as the separator and store it in an array

content = content.split('\n')

And then simply generate the snippet using a getSlice(content,range) method

let snippet = getSlice(content,range)

Where getSlice is defined as follows

// Splits the content given the range a-b or just line number a
const getSlice = (content, range) => {
  if (range.includes('-')){
    var a = range.split("-")[0] - 1
    var b = range.split("-")[1]
    return content.slice(a,b).join('\r\n')
  } else if (parseInt(range)) {
    return content[parseInt(range)-1]
  }
}

Note: range is passed as a string with a - delimiter between the start_line and end_line

And thats it!

What about you?

Did you find this useful? I am curious to know if anyone has another approach for this since GitHub does not have an API to do so (yet!)

5 Event-Driven Architectures Myth Busters!

Tamimi — Fri, 07 May 2021 00:51:14 +0000

Event-Driven Architecture: What's the sitch?

Alright lets face it, there are alot of content talking about how REST APIs are not the best architectural approach for designing a microservices architecture. I wrote a blog post about My journey to learning EDA that highlights what event-driven architecture is. Whether you are new to EDA, or have some background with it via dabbling with gRPC, kafka, rabbitMQ, Solace or whatever messaging API, I am here to share with you 5 claims about EDA that I will be BUSTING or CONFIRMING!

Note: The original and a bit longer log was originally written on: Event-Driven Architecture Myth Busting — Part 1: Five Common EDA Claims

1. I will have to re-design my REST-heavy architecture from scratch to adopt EDA

🤯 MYTH BUSTED!

Advanced event brokers allow for protocol translation within the broker. What does this mean you might ask? Well, it is very common in any software architecture design approach to have a polyglot of protocols and APIs in an application. Whether you are using REST, or different messaging protocols (MQTT, AMQP, Solace, Kafka...etc) you would want your different microservices communicate with each other.

Therefore, we can't claim that a complete overhaul of a REST-only architecture is a must to implement EDA.

2. APIs are only asynchronous in EDA

🤯 MYTH BUSTED!

Implementing EDA in an already existing architecture involves the process of event-enabling the underlying technology. Event-driven architecture does not replace synchronous call-and-response REST altogether, but complements it.

Having a harmonious interactions between synchronous and asynchronous APIs is inevitable in a digital transformation journey to adopt EDA.

3. Designing EDA involves lots of moving parts

✅ CONFIRMED!

Yes this is true. Designing and implementing event-driven system requires a comprehensive strategy, and multiple tools. You are dealing with alot of things here: topology visualization, application documentation, curating and cataloging the different topics, defining payload schemas for every published events, defining the interaction between applications, pin-pointing the cascading effects of every change, and ON TOP OF THAT managing access governance!

Phew thats alot! What's the solution? We'll I am presenting you with two solutions:

1️⃣ Use the right tools! Microsoft Excel to track event topics, Wiki pages like confluence to document the payload schemas, Git to version control the different schema versions for every topic, Visio diagrams to visualize the constantly changing event-driven architecture topology and finally a communication tool to communicate the changes. And oh wait, one last thing, a project manager to manage all this 😅

2️⃣ Use an Event Portal! Similar to how REST APIs have an API portal to document and manage the API, Asynchronous Event APIs have an Event Portal that facilitate the design, documentation and governance of events and event-driven applications allowing easier collaboration between different stakeholders.

4. An event broker is the only thing you need to implement EDA

🤯 MYTH BUSTED!

It's important to know that when working on implementing an event-driven architecture the approach should be looked at as a holistic platform composed of a list of tools instead of just one tool (i.e. Event Broker). This is a common misconception that EDA is associated with an event broker only.

Enabling event-driven architecture requires strategies and tools for documentation; governance to help determine the downstream impact of upcoming changes to apps; events and schemas; and topology visualization of many different microservices, along with the relationship of events connecting applications and application domains.

And that is why this statement is a myth!

5. Using Asynchronous API patterns means processing is done “later”

🤯 MYTH BUSTED!

I would definitely recommend looking up message exchange patterns (MEP) in event-driven architecture. When doing so, you will see that request-reply is one of the MEPs in EDA which involves an event consumer requesting information from another provider resulting in a bi-directional exchange of information. This behaviour is a proof that asynchronous APIs does not always mean process later.

SHOW ME MORE!

Did you enjoy reading this? Do you want more details about every claim? Say no more! Check out my original blog post here: Event-Driven Architecture Myth Busting — Part 1: Five Common EDA Claims

Do you have OTHER claims that you would like me to bust or confirm?! Comment down below and I would be more than happy to hear your input and thoughts on this.

P3ace Out ✌️

P.S Follow me on The Twitter for more content @tweettamimi

Event-Driven Architecture Myth Busting — Part 1: Five Common EDA Claims

Tamimi — Thu, 29 Apr 2021 13:50:09 +0000

Here is a claim that I am willing to drop: event-driven architecture (EDA) is filled with myths. Yep, I said it — and that’s a fact! Now that the elephant is out of the room, let me explain. In this blog post, I will be listing a couple of hand-picked claims about event-driven architecture and, you guessed it, I will either bust them or confirm them as fact. Let’s go ahead and delve further into this exploration with our first five EDA claims!

1. I will have to re-design my REST-heavy architecture from scratch to adopt EDA

Poll Results:

A digital transformation journey strategy explicitly takes that into account! It simply answers the question of “how can I liberate my already existing architecture with its legacy tools, tech, and protocols to modernize my infrastructure without throwing away what I already have”. Choosing an event broker to support an event-driven architecture implementation that natively supports protocol translation facilitates the adoption in existing architectures. If the event broker of choice natively supports sending/receiving REST messages, you’ll be able to integrate existing RESTful APIs with messaging APIs and protocols like JMS, MQTT, or AMQP without any extra external adapters, connectors, or bridges.

This is an extremely valuable asset to an event broker as it provides the organization with an architecturally simple way to distribute business events (order placed, payment initiated, room booked, etc.) to core APIs/services. And since digital transformation journeys to adopting event-driven architecture is an iterative process, this pattern of protocol translation support is valuable in the long-term as it provides a gradual path for adoption of an event-driven architecture. Check out these resources to see how you can become an event-driven enterprise in 3 steps or implement event-driven architecture in 6 steps.

Therefore, claiming that a complete redesign of a REST-only architecture is inevitable when adopting event-driven architecture is a complete… BUST!

2. APIs are only asynchronous in EDA

Poll Results:

A digital transformation journey to adopt event-driven architecture involves the process of event-enabling an already existing architecture. This could include a harmonious interaction between microservices and applications that use synchronous REST-based APIs and other components of the system that incorporate asynchronous message-oriented APIs.

A modern event broker should support both synchronous and asynchronous message exchange patterns. We’ve always heard the famous “it depends” answer when asking any architectural question about the choice of technology stack in an architecture. And it’s true – choosing a synchronous or asynchronous strategy to implement a particular feature or solve a specific problem does depend on the use-case. That’s why it’s so important to acknowledge that event-driven architecture does not replace synchronous call-and-response REST altogether, but complements it.

Hence, the myth that APIs are only asynchronous in event-driven architecture is… BUSTED!

3. Designing EDA involves lots of moving parts

Designing and implementing event-driven system requires a comprehensive strategy, and multiple tools. These tools are used to track all the events in the system, the different payload schemas, the documentation of the events, the versioning of the topics, and – on top of that – visualizing the overall event-driven architecture topology between the different application domains in your system.

Organizations use Microsoft Excel to track event topics, schema registries to track the payload schema, Visio diagrams to visualize the constantly changing event-driven architecture topology, wiki pages to document every moving part, and code repositories to store the application code.

Complete event management toolsets like event portals facilitate the design, documentation and governance of events and event-driven applications allowing easier collaboration between different stakeholders. Similar to an API portal that makes REST-focused application development easier, an event portal is a one-stop-shop for all things EDA, starting from designing topics, defining payload schemas, visualizing the topology, defining the contracts between applications and application domains within the organization, all the way to code generation from leveraging asynchronous API specifications like AsyncAPI.

Hence, the myth that designing event-driven architecture involves many moving parts is… CONFIRMED!

4. An event broker is the only thing you need to implement EDA

When deciding whether to adopt an EDA strategy, architects and developers should look at this decision through a platform lens instead of just evaluating event brokers. Event-driven architecture should be implemented with a toolset that enables data transformation, connectivity to a wide range of event sources, and tracing events through an event mesh that cannot be independently provided by an event broker.

All this cannot be successfully implemented with a standalone event broker only, so the myth that an event broker are the only thing you need to implement event-driven architecture is… BUSTED!

5. Using Asynchronous API patterns means processing is done “later”

When implementing event-driven architecture, a developer can leverage several message exchange patterns, including “fire-and-forget” and request-reply. The fire-and-forget pattern is useful for keeping logs or events for later processing, such as analytics. On the other hand, the request-reply pattern is known for being inherently synchronous, meaning that further processing is blocked until the initial request is completed. I would even argue that implementing a request-reply behavior using messaging APIs is more powerful than REST APIs (obviously this is a biased opinion given my focus on messaging).

The request-reply pattern can be used in the case of an event consumer needing to request information from another provider, requiring a bi-directional exchange of information. Implementing this pattern involves:

A requestor publishing a message on topic A.
The message is then received by the provider that subscribed to topic A.
A response is then published by the provider on topic B.
The response is then received by the original requestor that subscribed on topic B.

Due to the asynchronous nature of the implementation, the requestor can engage in more than one asynchronous request-response conversation at one time.

So, the myth that using an asynchronous API means that processing is only done later is… BUSTED!

Stay tuned for more myth busting (or confirming) in Part 2!

The post Event-Driven Architecture Myth Busting — Part 1: Five Common EDA Claims appeared first on Solace.

My journey to learning Event-Driven development

Tamimi — Sun, 24 Jan 2021 19:48:28 +0000

Myth Busters: How to interact with online Web Services?

Legend has it that you can only interact with web services using REST APIs. The same legend also has it that HTTP is the only way of communication on the interweb. And I am here to bust this myth. Thank you in advance for coming to my TED talk.

Elevator pitch

REST over HTTP impose an implied tight coupling between services due to the synchronous point-point nature of the API and protocol respectively. A system based on sending and subscribing to events advocates for a more real-time behaviour and a decoupled architecture hence the name "Event-Driven Architecture" (EDA). The decoupling is achieved by an applications sending events on a topic to a message broker without knowing how the event is controlled or is consumed by.

HTTP, REST and EDA walk into a bar

Okay, let me try to break down these concepts in a very simple manner

{
  0 : {
    "concept" : "HTTP",
    "definition : "Synchronous point-to-point communication protocol between a client and a server to exchange resources"
  },
  1 : {
    "concept" : "REST",
    "definition : "Mechanism leveraging HTTP to send, receive and modify resource one request at a time (request-reply)"
  },
  2 : {
    "concept" : "EDA",
    "definition : "Event Driven Architecture based on the idea of sending events to a message broker on a topic and subscribing to them for consumption"
  }
}

I have a previous blog post here talking about How I built an event-driven NodeJS app on realtime COVID-19 data streams. You can dabble with it if you'd like 🔥

What is my Journey to learning event-driven development

I wrote a lengthier blog post that goes into my journey to learning event driven development in details

In this blog post I talk about

The history of HTTP and the rise of REST APIs
Why HTTP and REST are popular
How our education system is TIGHTLY coupled to industry relevance. In other words, most course outlines are based on "Will this content get a job and dolla dolla billz to our students? If yes then include it in the curriculum!"
Foundational software and technology concepts and how they map to Event-Driven development concepts
REST vs. Event-Driven
Silver bullet when choosing REST vs EDA

You can read more about the post here https://solace.com/blog/developer-guide-event-driven-development/

And with that, I will leave you with

What is your Event-Driven Journey? Have you used event-driven architectures before and how were you introduced to it? Is this your first time hearing about event-driven development and what are you thoughts about it?

A Guide for the Developer’s Journey to Event-Driven Development

Tamimi — Thu, 21 Jan 2021 14:10:41 +0000

After almost a year working in the world of event-driven architecture and development – and given that I come from a REST API background, which I’m sure a lot of you can identify with – I thought I would share my learning experience. So all aboard! You’re going to embark on my journey to learning event-driven architecture (EDA) from a software developer’s perspective. This journey will take you through:

The origin and evolution of web services
Popularity of REST APIs and the impact on education
Mapping REST concepts/terms to their EDA equivalents
Building web applications: REST vs. event-driven
Is EDA a silver bullet?

So, you can either choose your own adventure and read the section of your choice or jump aboard and take the event-driven development ride from the beginning (recommended!).

1. The origin and evolution of web services

Before taking your seat on this event-driven development train, let me give you a quick history lesson on why REST APIs are so popular and how the industry adopted this protocol for most web applications.

In the late 90s and early 2000s, HTTP started to gain traction in the internet stack by giving people the ability to interact with web services over a network. Web applications were a gold mine in the world of the internet, allowing individuals and businesses to provide services and interact with end users.

During my undergraduate degree in Communications Engineering, I learned about the OSI model that describes the functions of a networking system through a conceptual framework and a list of abstractions. Web applications rely on the application layer (the only layer the end user interacts with) to transmit information across the lower layers underneath. These web applications and services interact with the application layer through different protocols; HTTP was (and still is) the most popular choice for its simple implementation (relatively speaking!).

Image Source: https://www.bmc.com/blogs/osi-model-7-layers/

Protocols that communicate with web services and objects over HTTP started rising; the first of which was Simple Object Access Protocol (SOAP). Without delving into the SOAP implementation details, its complexity motivated the rise of REST to communicate with HTTP web services, which made things even simpler to build. There had to be a way for systems to communicate with each other over a protocol. That system architecture is conceptually simple: client-server architecture. A client asks for resources, waits for a response from a server, and performs an action based on this response. Commands through CRUD operations provided a skeleton framework for request-reply and point-to-point communication patterns between services.

Image source: https://en.wikipedia.org/wiki/Client%E2%80%93server\_model

With REST APIs gaining more popularity and industry adoption, more tooling around using REST started evolving. Tooling such as OpenAPI, Swagger, and API Portals facilitated the adoption of REST APIs for building web applications. Automation and code generation tools made learning, understanding, and implementing REST APIs simple, resulting in this implied reputation that they should be the de facto for system communication and for building web applications.

Interesting Fact : In 2000, eBAY was the first enterprise to adopt a REST architecture. Flickr followed in 2004, Amazon in 2006, and then Facebook and Twitter eventually followed suit. At the time of these companies’ adoption of REST architecture, there was no concept of event-driven APIs to interact with web services from a user-facing perspective.

2. Popularity of REST APIs and the Impact on Education

You might be wondering why REST APIs are so popular. Well, when it comes to learning foundational concepts in the software industry, whether it’s through theoretical education (schooling, online courses, college…etc.) or real-life experiences through on the job training, the learning process is based on two main factors:

Theoretical background and fundamentals
Industry relevance

The theoretical background piece is inevitable, and is not influenced by what’s hot and trending. Industry relevance, however, affects the educational content. It only makes sense that course content must be able to draw a direct connection to what’s being implemented in real-life use cases. And let’s face it, most people go to school not just to learn but also get a job and get paid! REST had (and still has) a high industry relevance, so naturally my education path in developing and architecting web applications did not cover learning SOAP services or even event-driven architecture since there wasn’t much in terms of industry relevance. Does this mean that event driven development was completely new to me when I was introduced to it? On the contrary! Next, I’ll explain how the concept of event-driven architecture was ingrained in different areas of learning and how this enabled me to map foundational concepts to EDA fundamentals.

During my university undergraduate degree, the concepts of event-driven systems and real-time applications were covered as different subjects. The missing element in my event-driven development journey was the bridge between those theoretical concepts and their actual implementation . I will briefly highlight four areas that covered messaging and event-driven systems – Frontend and Game Development, Programming Design Patterns, Networking Fundamentals, and Operating Systems Fundamentals – followed by a table that maps the theoretical concepts to event-driven architecture lingo.

Frontend and Game Development

Interacting with UI elements in front end development is event-driven in nature. A user executes an “event” that immediately “drives” an action in return. For example, a button does not poll its state to check if it is pressed or not, it simply registers an EventListener with a callback function that gets triggered upon an event being fired. A simple implementation takes the following format:

element.addEventListener(event, function);

Let’s look underneath the hood and digest this further. Since JavaScript is single-threaded and can only do one thing at a time, the browser has an event loop that keeps track of all the events in an event queue which can be accessed through interfaces built on top of the core JavaScript language. Whenever an event occurs in the web browser, the event loop dispatches this event to any registered listener. Sweet and simple. Note that the event loop implements the reactor pattern design discussed in the following section.

Image source: https://www.geeksforgeeks.org/what-is-reactor-pattern-in-node-js/

In game development, the concepts of EventHandlers and EventDispatchers are very prevalent for asynchronous interactions with a game. The only way to use communication between objects where you don’t care about the target object receiving the event is via EventDispatchers.

Programming Design Patterns

Software design patterns present reusable solutions to common problems. For real-time applications, I will give a high-level overview of two common design patterns that present an event-driven approach versus a thread-based approach:

Reactor Pattern

Image source: https://developers.redhat.com/blog/2019/05/13/building-and-understanding-reactive-microservices-using-eclipse-vert-x-and-distributed-tracing/pasted-image-0-1/

This pattern relies on the concepts of dispatching events. It includes a “Reactor” that runs in a separate thread, responding to any I/O events and dispatches the work to “Handlers”. The benefit of the reactor pattern is to avoid having dedicated threads for each message, connection, or request. As seen in the diagram above, events are demultiplexed and dispatched to the responsible handler.

Observer Pattern

Image source: https://subscription.packtpub.com/book/web\_development/9781783287314/1/ch01lvl1sec12/the-observer-pattern

The observer pattern differs from the reactor pattern in a way that each object maintains a list of its dependents. A change in state for an object will simply trigger a notification (i.e., event) to all the dependents signaling a change. The observer pattern implements a one-to-many relationship where multiple dependents could be “observing” the change in an object.

Networking Fundamentals

Recalling the OSI model covered earlier, the network layer was a HUGE help to me when it came to understanding event-driven architecture and development. This layer is responsible for receiving packets of data in the internet stack and delivering them to the intended destination through IP routing. Every packet traveling through the internet has a logical address to its destination (IP) that is handled by the router (middleware hardware) that literally routes the packet to other destinations in the network. These packets could either be destined to hosts within the network boundary (egress traffic) or traffic coming in from outside the network boundary (Ingress traffic).

Image source: https://www.cloudflare.com/learning/network-layer/what-is-routing/

The keywords to keep in mind here are:

Router : a device that contains IP lookup tables mapping from and to destinations.
IP : the address where the packet is going to or came from.
Host : network device that sends/receives packets.
Packet format : contains metadata about the packet including destination and data content.

Operating Systems Fundamentals

Similar to the reactor pattern discussed earlier, scheduling and threading in an operating system embodies the concepts of dispatching and demultiplexing. Events (I/O, interrupts, thread returns, etc.) are either dispatched to the target endpoint or added to a queue. There are six popular scheduling algorithms, but in a very simplistic overview, scheduling enables asynchronous activities in operating systems and minimizes any resource starvation.

3. Mapping foundational concepts to their EDA equivalents

Matching the foundational theoretical concepts discussed in the section above to the event-driven architecture equivalent aided me in my event-driven development journey. I summarized them in this table:

Category	Foundational Concept	EDA Equivalent
Front-end & Game Development	Event Dispatcher	Publisher
Event Handler	Subscriber
Event Loop	Event Broker
Programming Paradigms	Reactor Pattern	Event Broker
Observer Pattern	Publish/Subscribe
Networking	IP Routing	Topic Routing
Egress and Ingress Routing	Event Mesh
IP Address	Topic
Packet Anatomy	Message Anatomy
Subnet masks	Wildcard filters
Operating Systems	Scheduler	Event Broker
Process Scheduling Queues	Broker Queues
Threads	Consumers
I/O	Publish/Subscribe

In Front-End and Game Development:

Event Dispatchers and Event Handlers relate to publishers and consumers respectively for the nature of publishing and listening to events.
An event loop relates to a message broker due to the nature of dispatching events to registered listeners.

In the Programming Paradigms category:

The Reactor pattern relates to an event broker through the concept of dispatching events.
The Observer pattern relates to Publish/Subscribe message exchange patterns through the one-to-many relationship of a publisher to listeners.

On the Networking front:

Every packet has an IP address which is synonymous to Topics that events are sent or received on.
The concepts of IP packet routing relates to message routing in the event broker via topic matching.
Egress and Ingress traffic routing through multiple networks relates to the behaviour of an event mesh in event-driven architecture.
The anatomy of packets in a network is broken down in a way that relates to a message anatomy
The way subnet masks act as filters allowing specific IP ranges relates to using wildcards in topic matching and filtering

And finally on the Operating Systems front:

The behaviour of a scheduler relates to an event broker through dispatching and demultiplexing of events to threads.
Process scheduling queues keeps track of all the processes in the system similar to how a broker queue keeps track of the guaranteed messages on the broker.
Operating system threads consume resources similar to how a *consumer * in an event-driven architecture consumes messages.
Input/Output ** operations relate to **Publish/Subscribe message exchange patterns in a way that an input device sends instructions to the operating system and the output device receives commands.

4. Building web applications: REST vs. event-driven

Now that the theoretical background of my journey is covered, I’d like to talk about how this is related to building web applications. If REST APIs have already been adopted for building applications and is considered the de facto solution in many industries, why should we bother ourselves with finding an alternative? Why should we consider an event-driven approach for operational use cases?

Increased demand for having immediate access to insights had proven that time-critical applications could not rely on REST for real-time feedback and actions. With the recent increase in demand for high availability services, trendy architectural designs like microservices and containerizations started gaining a lot of popularity. Tooling around microservices deployment strategies and container orchestration (e.g. Kubernetes, EKS, AKS, Docker Swarms, etc.) also started to become more popular. According to Forbes, data is becoming the new oil, with data being constantly streamed representing a constant flow of events. An example of this is a stock ticker; every time a stock price changes, a new event is created. It’s called “streaming data” because there are thousands of these events resulting in a constant stream of data. Reactive patterns advocate for resilient systems that are message driven as opposed to polling dependent. As per the Reactive Manifesto, “Today applications are deployed on everything from mobile devices to cloud-based clusters running thousands of multi-core processors. Users expect millisecond response times and 100% uptime.”

These design requirements motivated me to learn more about event-driven architecture and exposed me to the issues of REST-only and REST-heavy architectural choices for building applications. Keeping the Reactive Manifesto in mind, I’ve summarized a high-level comparative list between REST-based and messaging-based solutions below. Disclaimer: this is not a comprehensive list.

Pros

Tooling and documentation
Industry adoption and support
Simplicity of implementation
|

Cons
Built on request-reply communication assumptions. Synchronous in nature.
Point-to-point communication through the adoption of a client-server architecture and reliance on HTTP leading to tight coupling in microservices architectures resulting in a distributed monolith.
REST lock-in. If microservices communicate using REST, that will be the main form of communication preventing interoperability with non-REST applications.
|
|

Event-Driven Systems

Pros
Eventual consistency. The reason why I see this as a pro is because an event-driven system promotes a guaranteed high availability of the data where eventually all microservices in the system will receive the last updated value of the data.
Reduced coupling between microservices; a message sender does not need to know about any consumer. The only coupling in EDA is a topic hierarchy structure.
Loose coupling of applications instead of point-to-point relationships, resulting in easier scaling of the system.
Load leveling and shock absorbing though queuing mechanisms.
|

Cons
Complexity of handling asynchronous messages when using primitive event brokers. This complexity can be minimized by using an advanced event broker.
Coupling with the messaging infrastructure.
|

Adopting a REST-driven architectural design has several advantages to it. These include the availability of high quality, industry approved tooling and documentation; industry adoption and support; and simplicity of implementation. However, the drawback to a REST-heavy system is that it’s synchronous in nature. This means that it’s built on a request-reply communication concept, allowing only point-to-point communication: the client must always call the server, the server can not reach out to the client, and the client can not reach out to other clients directly. Through the adoption of a client-server architecture and reliance on HTTP, tight coupling in a microservices architecture could result in a distributed monolith. The one-to-one nature of REST means that a separate REST call is needed for every endpoint which makes it hard to extend the architecture without modifying the clients. I have seen that it is common to build a distributed monolith instead of the intended decoupled microservice architecture, and this leads to a lot of time and money wasted.

On the other hand, an event-driven system reduces the amount of coupling between microservices. This means that a message sender does not need to know about any consumer; the only coupling in EDA is a topic hierarchy structure. Loose coupling enables easier scaling of the system and handling of traffic spikes through shock absorbing and queuing mechanisms. In message-oriented systems, applications produce events without controlling how the consumption of those evens is done. This contrasts REST-based systems that rely on synchronous tight coupling between the client and the server.

Eventual consistency is intuitively handled in an event-driven system using a message broker, enabling parallel execution of microservices and making sure that each microservice will eventually come to a state of completion. This allows the system components to eventually become consistent with the state of the system without the need of a two-phase commit or distributed transactions. However, complexity of handling asynchronous messages when using event brokers could be a challenge. This challenge is associated with the adoption of any new architecture, which like all other developers, a healthy challenge is encouraging! Another drawback of an event-only driven system is the coupling to the messaging middleware infrastructure, so when adopting an messaging based architecture one should take into account the multi protocol compatibility along with interoperability with other message brokers.

5. Is EDA a silver bullet?

Neither REST nor event-driven architecture are best for every scenario, and choosing one depends on what you’re trying to accomplish. Large distributed systems and distributed architecture microservices are built on a polyglot of technologies and architectural patterns. An architecture that combines REST and event-driven architecture would deliver the full potential of real-time development and interoperability. Event-enabling a REST architecture allows you to combine traditional REST interactions with publish-subscribe communication, with a message broker acting as the middleware layer for message flow.

As developers in this modern world, we are no longer just concerned with applications that we write in a silo. Considering the effects of our applications on other parts of the business is becoming more important than ever as the notions of DevOps, microservices, and integrations all become every-day terminologies that a regular developer in the tech industry would hear and use. Furthermore, creating applications that would support a spike in traffic at any point in time is becoming more crucial.

Conclusion

This is my personal event-driven development journey, and like all tech journeys a developer chooses to take, there is always something to learn. I don’t know about you, but I am a developer driven by a challenge and have an urging thirst to solve technical problems using slick tools and architectures! Who knows what architectural trend will come up next with the rise of cloud infrastructures (hybrid/multi-cloud) and new technologies. Regardless of the new tech, it’s important to test it out for yourself, conduct research, keep an open mind, and be ready to learn and grow!

Thanks for taking the ride with me. Do YOU have an event driven journey or a different experience you’d like to share? Join the Solace Community or find me on Twitter @tweetTamimi!

The post A Guide for the Developer’s Journey to Event-Driven Development appeared first on Solace.

Live session: Event-Driven & code generation with AsyncAPI and Solace 10/06/2020

Tamimi — Fri, 05 Jun 2020 18:49:41 +0000

Two types of people

During The 2020 Quarantine, I had the opportunity to delve deep into my thoughts and come up with euphoric discoveries. One of the realization that I came to is that there are two types of people in this world:

Those who sit on the couch and watch Netflix all day
And those who read blog posts on dev.to and attend live coding sessions to learn new technologies and interact with other developers

Since you're reading this now, then you are definitely the second type. So "what is it I will be learning from this live coding session", you may ask? Asynchronous development in an event-driven world! Developing your applications in an event-driven manner is one of the most efficient way to achieve real-time interactions with your data and build a microservices infrastructure. I talk more about this topic and event-driven architectures in this blogpost

For those who are not aware of AsyncAPI, its the industry standard for defining asynchronous APIs to easily build and maintain event-driven architecture. So think Swagger and OpenAPI but for event-driven APIs instead of REST APIs.

Taking Event-driven to a META level

Now imagine a world where you can blend event-driven development AND code generation:

Say no more! AsyncAPI also created a generator open source tool that can be used to generate asynchronous event-driven applications based on a predefined AsyncAPI specification file as an input. You can check out the list of official generator templates on the generator github repository.

What's the deal with the live coding session?

Glad you asked! An online session will be held on June 10, 2020 at 10:00 ET that covers design to code using Solace PubSub+ Event Portals + AsyncAPI Code Generation. I will be the host interviewing Marc DiPasquale from Solace and Fran Méndez, the founder of AsyncAPI, while we get into an interactive live coding session where we’ll check out the design to code experience. We will also be building an event-driven microservice using Spring Cloud Stream and answer any questions you have.

How can I join

YouTube
Twitch
Solace Community

String youtubeLink = "https://youtu.be/oQbZf9DTnKc"
String twitchLink = "twitch.tv/solacedevs" 

System.out.println("Youtube Link: " + youtubeLink + "\nTwitch Link: " + twitchLink);

So come take part in this interactive session and chat with us on the live stream, I'll give you a shout out if you ask a question 😎

And to end on a good note, I’d like to leave you with one of my favorite gifs

Disclaimer: I am a Developer Advocate at Solace @SolaceDevs

LIVE coding during COVID: Solace ft CircleCI

Tamimi — Wed, 13 May 2020 15:11:03 +0000

The Developer Species

As a developer, I bet that you also get uneasy stagnating and not leaning something new. We are always on the look for something challenging and "cool" (technically speaking) whether be it the next HOT framework, programming language or reading up on new development patterns that will increase the efficiency of your program (confession session: I'm a victim for this).

One way I tend to learn about a new technology is by attending in-person conferences and workshop as it gives me the opportunity to have a hands-on learning experience. I particularly enjoy that as it gives me some face-to-face interaction with other technologists and developers, which sparks a lot of interesting conversations and back and forth questions entertaining the inner geek in me. I call this Tamimi's Brainstorming Thought Volleyball™, or TBTV, where we get the opportunity to bounce off ideas and questions with other people.

HOWEVER, we are currently surrounded by this thing called COVID (aka rona by some people, dont @ me) which stands in the way of the physical interaction learning process.

What's the alternative

You guessed it, LIVE coding sessions! (don't give yourself credit for guessing it, the title of this post gives it away). I'm taking part of a LIVE coding session with Angel from CircleCI where I build an event driven application and learn about build pipelines, a topic that I am alway curious to learn more about. The session will take place today, Wednesday May 13 2020, at 2:00 pm EST and you can check it out on this link
https://www.youtube.com/watch?v=P4ewUZRhVLA. If you're more of a twitch streams kind of person, you can also join here https://www.twitch.tv/circleci

What's in it for me, you may ask

Attending this, you will get the chance to

if (User.didAttend()) {
    console.log("Learn about eventdriven development by using Solace PubSub+ Event Broker")
    console.log("Learn NodeJS if you want to get into it or you're a beginner")
    console.log("Learn about CircleCI build pipelines")
    console.log("Teach attendees some cool things!")
}

So come take part in this interactive session and chat with us on the live stream, I'll give you a shout out if you ask a question 😎

What about you?

How do you deal with the lack of physical interaction and attending in-person events? How do you make the learning process more interactive? Have you attended any live coding sessions and actually learnt something from it?

And to end on a good note, I’d like to leave you with one of my favorite gifs

Disclaimer: I am a Developer Advocate at Solace @SolaceDevs

How I used COVID Tracking data to build an event-driven application and learn more Python 🤓

Tamimi — Sun, 03 May 2020 15:01:23 +0000

TL;DR – I built a python event driven application leveraging a publicly available real-time stream of COVID-19 data updates using Solace PubSub+ Event Broker as the messaging infrastructure. Check out the source code in this github repo under samples/python

SolaceLabs / covid19-stream-processors

Stream Information & Example Applications for Processing JHU and CovidTracking.com COVID-19 data available as streams over Solace

Phew! Now that the TL;DR elephant in the room is out of the way, let’s get to the core of it 🐘

Right now with the situation the world is currently in, some of us are probably thinking “hmm what can I do to contribute back to society? If I can't directly contribute something, what can I learn during quarantine to polish my skills and at the same time build something useful?” I don’t know about you, but I definitely enhanced my baking skills with the amount of banana bread loaves I baked (and ate). Apart from baking, I took this situation as an opportunity to learn new technologies and have something relatable to implement it on (i.e. COVID). I chose Python as the programming language of choice to learn and complement all what I know in event driven architectures.

In a previous blog post, I discussed how I built an event-driven NodeJS application on real-time COVID-19 data streams. I also covered some background on what event-driven is, why is it important and how to use the publish/subscribe messaging pattern to approach building and EDA application, so I recommend reading it if you haven’t already 🤗 . If JavaScript/NodeJs is not your cup of tea, then worry not! In this blogpost, I will be sharing how I used Python 🐍 to build a simple event driven application using the same COVID-19 streams of data with the same three basic requirements.

Tech Stack

Before building any application and work on a project, I start by drafting my architecture design, the different tools to be used and the programming language I need to successfully get an MVP up and running. With this approach in mind, I will be using the following tech stack:

Programming Language: Python
Messaging Protocol: MQTT
Message Broker: Solace PubSub+
Data source: COVID-19 data provided by the publisher application as per this documentation.

Note that in my tech stack, I didn’t list any REST API or client/server architecture 🤝 Done. Next!

1. Environment Setup & Prerequisites

To setup my Python environment, I will be using virtualenv. In a nutshell, virtualenv is a tool used to isolate Python environments by creating a directory structure encapsulating all the installed packages you need in the application you’re building. This way, you wont be contaminating your global Python packages with anything required by your application. You can read more about it here and on why it is a recommended approach for Python development.

Note: If you come from a NodeJS background, think in terms of node_modules.

In a new terminal, execute the following command to create a new directory for your project, setup the Python virtual environment and activate it:

mkdir covid_python && cd "$_" && virtualenv env && source env/bin/activate

Now install the paho-mqtt python package:

pip install paho_mqtt

2. Skeleton code setup

Create a new covid.py file and initialize it with the following:

import paho.mqtt.client as mqtt

# Solace Broker Info
url = "mr2r9za6fwi0wf.messaging.solace.cloud"
username = "covid-public-client"
password = "covid19"

def on_connect(client, data, flags, rc):
    assert (rc == 0), "Error Connecting. Return code: " + str(rc)
    print("Connected to the COVID PubSub+ Solace Broker!")


client = mqtt.Client()
client.username_pw_set(username=username, password=password)

client.on_connect = on_connect

client.connect(url)
client.loop_forever()

What you did above is import the paho-mqtt package and initialized an mqtt client using the configuration parameters.

It’s important to note the following:

The client.loop_forever() is a blocking function call that processes network traffic, dispatched callbacks and handles reconnecting. You can read more about other loop*() functions in the main documentation page.
.on_connect is a function called when the broker responds to our connection request. We assign this callback to the on_connect(client, userdata, flags, rc) function that will get executed upon a successful connection.

You now have a runnable skeleton to test out our connection. From your terminal execute the following

python covid.py

You should see Connected to the COVID PubSub+ Solace Broker! on your terminal. Saweeet! Connection established 🎉. To stop the execution of the program, press CMD+c (or Ctrl+c if you swim in the Windows domain).

3. Handle events

To handle the incoming messages from the PubSub+ Broker in an event-driven manner, we have to first subscribe to the topic of interest (according to the available topics) and handle the messages with the second type of callback function: .on_message.

import paho.mqtt.client as mqtt

def on_connect(client, data, flags, rc):
    assert (rc == 0), "Error Connecting. Return code: " + str(rc)
    print("Connected to the COVID PubSub+ Solace Broker!")

    for t in topics:
        print("Subscribing to: " + t)
        client.subscribe(t)

def on_message(client, data, msg):
    print("Received message on: %s\n %s" % (msg.topic, msg.payload.decode('ascii')))

# Broker Info
url = "mr2r9za6fwi0wf.messaging.solace.cloud"
username = "covid-public-client"
password = "covid19"

topics = [
    "jhu/csse/covid19/test/cases/+/update/Canada/Ontario/#",
    "com/covidtracking/states/current/update/New York",
]

client = mqtt.Client()
client.username_pw_set(username=username, password=password)

client.on_connect = on_connect
client.on_message = on_message

client.connect(url)
client.loop_forever()

In the above code program:

I added an on_message(client, data, msg) function to handle the .on_message callback from the client. Note the decoding of the received message payload to ascii since we receive the payload in binary format.
There are subscriptions to two different topics
1. All cases updates in Ontario, Canada obtained from the John Hopkins University data source. I used a test stream.
2. All cases updates in New York obtained from The Covid Tracking project data source. Note you can use the topic com/covidtracking/states/current/get/raw to get raw updates from all the states in the United States

Run the code and wait for updates to come in!

A note on JSON

Since the returned format of the message payload is in JSON, you can use the python json package to handle, manipulate and BEAUTIFY your output. Here is the full code for your application

import paho.mqtt.client as mqtt
import json

def on_connect(client, data, flags, rc):
    assert (rc == 0), "Error Connecting. Return code: " + str(rc)
    print("Connected to the COVID PubSub+ Solace Broker!")

    for t in topics:
        print("Subscribing to: " + t)
        client.subscribe(t)

def on_message(client, data, msg):
    msg_json_object = json.loads(msg.payload.decode('ascii'))
    msg_json_pretty = json.dumps(msg_json_object, indent=2)
    print("Received message on: %s\n %s" % (msg.topic, msg_json_pretty))

# Broker Info
url = "mr2r9za6fwi0wf.messaging.solace.cloud"
username = "covid-public-client"
password = "covid19"

topics = [
    "jhu/csse/covid19/test/cases/+/update/Canada/Ontario/#",
    "com/covidtracking/states/current/update/New York",
]

client = mqtt.Client()
client.username_pw_set(username=username, password=password)

client.on_connect = on_connect
client.on_message = on_message

client.connect(url)
client.loop_forever()

What’s Next 🤔

I would like to explore more front-end frameworks that I can use in conjunction with Python. So ideally, I will have my Python application communicate with the Solace PubSub+ Event Broker as the backend piece of the application infrastructure and a cool front-end framework to present it. Hey what’s up Django?

Closing Remarks

So here is the real question, what did YOU learn during quarantine? Share in the comments below and let’s start a discussion!

Oh and if you have any recommendations for front end frameworks I would love to 👀 what you have and 👂 your suggestions.

And to end on a good note, I’d like to leave you with one of my favorite gifs

How I built an event-driven NodeJS app on realtime COVID-19 data streams

Tamimi — Thu, 23 Apr 2020 17:06:51 +0000

In this unprecedented time we’re living in, we are all coming together to leverage our joint efforts and use it for COVID19 relief efforts to benefit the public. Everyone is pitching in with their own expertise. Some people have been pitching in by creating PPE products with their 3D printers, some have been developing software and others have been providing technical support for their loved ones or community. In this post, I would like to share how I used a stream of real-time COVID19 data updates to build a NodeJS event-driven application using a simple messaging protocol that can be used to build public facing applications.

In this application, I will be using the different streams Solace made available to the public for anyone to subscribe to. Documentation on how to use the streams can be found in this github repo.

SolaceLabs / covid19-stream-processors

Stream Information & Example Applications for Processing JHU and CovidTracking.com COVID-19 data available as streams over Solace

Research: A bit of background & requirements

Before building this application, I listed out three basic requirements that I wanted to achieve:

Real-time data updates
Light weight application (I didn’t want to continuously poll or check for new data)
Reactive to any data change

Options

From a higher level perspective, there are two different architectures I could've chosen:

A synchronous REST-driven approach or
An Event-driven architecture (EDA) approach

Option 1: REST

With the first option, there are lots of online resources that I could’ve tapped into including APIs or CSV datasets, such as the ones released by John Hopkins University in their github repo. While this is a viable option with lots of online samples and resources, I wanted something more real-time since

a) the data is most valuable when its first released (see figure 1 below) and

b) I wanted an application that reacts to data updates instead of continuously polling for updates. For example, the CSV file in the JHU github repo is updated once or twice a day. If I used this database, I would have to continuously poll it and check updates.

Figure 1: Value of data diminishes with passing of time

Also, since I would be dealing with a large data set, I want to only react to new updated data when changes come in. So a REST approach would not be a light weight implementation. This negates requirement 1 and 2.

Option 2: EDA

With an event driven architecture, I can use a publish-subscribe pattern approach to building my application. What is pub-sub you may ask? In a nutshell, it boils down to having a “Publisher” of the data (e.g. COVID-19 data source) and a “Subscriber” to this data (e.g. my NodeJs application) that reacts only when there is new data published.

With the PubSub+ COVID-19 Broker that Solace made available to the public, updates on COVID-19 data are published on different streams. So, application developers wanting to develop an event driven application (IoT, mobile/web app) could consume the stream of data by subscribing to any of the topics available. Since the consumption of this data is framework/platform/language agnostic, I could use any messaging protocols (MQTT, AMQP, JMS) or open APIs (Python, JS, NodeJS…) that support these protocols. I could also use REST!

Figure 2: High level end-to-end architecture overview

Decision

So, after evaluating the two options stated above, I decided to take the EDA approach to build my application. Also, since I wanted to use a lightweight messaging API, get real-time COVID-19 updates and be reactive to these updates, EDA was the clear winner.

Let’s get to business; well, I mean coding.

Based on the supported languages and protocols the Solace PubSub+ broker deals with, I decided to go with using MQTT since there is a native NodeJS API for this

Figure 3: Languages and protocolas supported by Solace

1. Initial setup

Let’s go ahead and start a NodeJS project. Open a new terminal and execute the following command which creates a new directory for your project, initializes it and installs the mqtt package


mkdir covidproject && cd "$_" && npm init -y && npm i mqtt

2. Connect to the Broker

Create a new file


touch index.js

And open it in your favourite text editor. Insert the following


var mqtt = require('mqtt') 

var host = "tcp://mr2r9za6fwi0wf.messaging.solace.cloud:1883" 

var config = { 

    username: "covid-public-client", 

    password: "covid19", 

} 

var client  = mqtt.connect(host, config) 

client.on('connect', () => { 

    console.log("Connected to COVID PubSub+ Broker!") 

})

What you did above is initialize an mqtt client and connected to the broker URL using the host and config variables. The mqtt object then returns back signals that your client application can leverage for callback behaviours, in this example it’s the “connect” signal we are listening to client.on(‘connect’). We will cover more signals later.

Note: the broker URL and credentials are from here https://github.com/SolaceLabs/covid19-stream-processors#1-connection-information

Now test out your connection by executing the following from terminal


node index.js

You should see Connected to COVID PubSub+ Broker! output . Voila!

3. Subscribe to the topic

Now that you are connected to the broker, all you need to do is subscribe to topics.


var topics = [ 

    "jhu/csse/covid19/raw", 

] 

client.on('connect', () => { 

    console.log("Connected to COVID PubSub+ Broker!") 

    topics.forEach( (topic) => { 

        console.log("Subscribing to topic: ", topic) 

        client.subscribe(topic) 

    }) 

})

4. Listen to incoming messages

The second signal we want to listen to is the message as follows:


client.on('message', (topic, message) => { 

    console.log("Received message on Topic: ", topic,"\nMessage:\n", JSON.parse(message.toString()))

})

Note that the received message from the broker is in binary format. To change it to a human readable format we use message.toString() and we JSON parse it. Note that the messages are sent in JSON format based on the schemas defined in the repo.

Your final application looks like this:


var mqtt = require('mqtt') 



var host = "tcp://mr2r9za6fwi0wf.messaging.solace.cloud:1883" 

var config = { 

    username: "covid-public-client", 

    password: "covid19", 

} 



var topics = [ 

   "jhu/csse/covid19/raw", 

] 



var client  = mqtt.connect(host, config) 



client.on('connect', () => { 

    console.log("Connected to COVID PubSub+ Broker!") 

    topics.forEach( (topic) => { 

        console.log("Subscribing to topic: ", topic) 

        client.subscribe(topic) 

    }) 

}) 



client.on('message', (topic, message) => { 

    console.log("Received message on Topic: ", topic,"\nMessage:\n", JSON.parse(message.toString()))

})

And we’re done! Your application is now connected to the broker and subscribes to one or more topics defined in the array and reacts only when new messages are published.

Sample run

I have modified my application to subscribe to all case updates (deaths, active, confirmed and recovered) in Canada Ontario, the recovered cases in Jordan and the confirmed cases in all the provinces in the the United Kingdom using the following topics on the test stream

var topics = [
    "jhu/csse/covid19/test/cases/+/update/Canada/Ontario/#",
    "jhu/csse/covid19/test/cases/recovered/update/Jordan/#",
    "jhu/csse/covid19/test/cases/confirmed/update/United Kingdom/#"
]

Notice the use of MQTT wild cards ('+' and '#') for topic level matches and multi level matches respectively.

You can check out https://www.marcd.dev/COVIDStreamViewer/mqtt/mqttListener.html and subscribe to jhu/csse/covid19/raw topic for a sample stream viewer.

Next steps

When you look at the topic hierarchy, you can subscribe to different topics and use mqtt wildcards to further customize how your client application consumes the event streams.

I would happy to see your ideas so feel free to share them and create a pull request to the SolaceLabs github repo! And if you have any questions, leave them in the comments sections below. And feel free to checkout my attempt to build a Python application with the same approach in this blog post!

DEV Community: Tamimi

Golang and Event-Driven Architecture

What's all the hype about?

Hold on, what is an event-driven architecture (EDA)?

How I built an event-driven NodeJS app on realtime COVID-19 data streams

Tamimi for Solace Developers ・ Apr 23 '20

fmt.Println("Show me the CODE!")

TamimiGitHub / conf42-golang-eda

Conf42 Golang 2022: Gopher in an Event-Driven Playground

Talk

Source Code

How to run

Extra Resources

Hands-on 👩‍💻🧑‍💻

MQTT

Solace PubSub+ Messaging API for Go

SolaceSamples / solace-samples-go

Getting Started Samples for the Solace Go API.

Solace PubSub+ Messaging API for Go

Environment Setup

Run Patterns

Note on protocol interoperability

Why use Solace native APIs??

Use Solace Messaging APIs for Advanced Event Mesh Features and the Best Performance | Solace

This is all cool. How can I know more 👀

What about you?

3 steps to configure Vanilla Forum with Orbit.love

Table Of Contents

What's Orbit?

Integrations

Where's my integration at?

Goal

1. Extract existing members. Done once!

Generate your VF Token

Write your script

Notes

Issues and Improvements

2. Import your CSV file to orbit

3. Connect Webhooks to Orbit Activities

Vanilla Forum new user webhook

Note

Zapier Webhook Actions

Orbit Step: Add new user

Tips and Tricks

How to query specific line(s) from a file using GitHub APIs

Permalinks

Show me the API!

Show me the code!

What about you?

5 Event-Driven Architectures Myth Busters!

Event-Driven Architecture: What's the sitch?

1. I will have to re-design my REST-heavy architecture from scratch to adopt EDA

🤯 MYTH BUSTED!

2. APIs are only asynchronous in EDA

🤯 MYTH BUSTED!

3. Designing EDA involves lots of moving parts

✅ CONFIRMED!

4. An event broker is the only thing you need to implement EDA

🤯 MYTH BUSTED!

5. Using Asynchronous API patterns means processing is done “later”

🤯 MYTH BUSTED!

SHOW ME MORE!

Event-Driven Architecture Myth Busting — Part 1: Five Common EDA Claims

1. I will have to re-design my REST-heavy architecture from scratch to adopt EDA

2. APIs are only asynchronous in EDA

3. Designing EDA involves lots of moving parts

4. An event broker is the only thing you need to implement EDA

5. Using Asynchronous API patterns means processing is done “later”

Stay tuned for more myth busting (or confirming) in Part 2!

My journey to learning Event-Driven development

Myth Busters: How to interact with online Web Services?

Elevator pitch

HTTP, REST and EDA walk into a bar

What is my Journey to learning event-driven development

A Guide for the Developer’s Journey to Event-Driven Development

1. The origin and evolution of web services

2. Popularity of REST APIs and the Impact on Education

Frontend and Game Development

Programming Design Patterns

Reactor Pattern