DEV Community: Elijah Wilson

Certes - Asynchronous Event Handling Standard

Elijah Wilson — Fri, 22 May 2020 00:20:46 +0000

Certes is a standard for asynchronous events, traditionally known as "webhooks". This post will give a conversational explanation around Certes, while the full technical specification can be read at spec.certes.dev.

Background

Webhooks are complicated and inconsistent. I've worked with webhooks at every company I've worked at with many different third parties sending the data. Each third party sent data in a different format, with some having security features like HMAC and some not.

In February I started working full-time on a company called HookActions to try to solve this problem. I built an MVP and started getting feedback from users. The overwhelming feedback was that webhooks are a problem but most people were not willing to pay for my service or didn't see the immediate use of HookActions. After having a conversation with a software developer in Israel something clicked for me. He had experienced the annoyances of webhooks but also thought what I was building might work well if built in a similar manner to Elasticsearch and Elastic.

This opened my eyes to what I could build and how to build it, which brings me to introduce the Certes open source project.

Goals

My goal at the end of the day has always been to make webhooks easy to use, I believe that Certes will do that efficiently and effectively. Here are the formalized goals I have set out for the project:

Standardize asynchronous event handling
Standardize an interface for creating and managing event subscriptions
Standardize the monitoring and tracing of asynchronous events
Standardize the format of asynchronous event data
Minimal latency increase (up to 150 ms from event received to processed)
Maintain an open-source community for changes, feedback, and guidance
Deploy anywhere with many options (e.g. single binary, docker, k8s, etc.)
Programming language agnostic client usage

Non-goals

Sometimes as important as goals are the non-goals of a project.

Introduce a new internet protocol (we should build on existing protocols such as HTTP, GRPC, DNS, etc.)
Solve 100% of asynchronous event problems (i.e. solving the top 20% of pain points/issues may actually solve all the problems for 80% of people)
Re-invent the wheel (i.e. use existing open-source tools when possible as long as they add minimal latency)

Subscribing to and consuming events

Certes is not yet implemented but imagine being able to write declarative event subscriptions for webhooks in your code:

package main

import (
  "fmt"
  "net/http"

  certes "github.com/hookactions/certes-sdk/go"
  gh "github.com/hookactions/certes-contrib/github"
)

func init() {
  certes.Init("events://events.meetly.com")  // Meetly's event gateway, local or hosted by 3rd party

  certes.EnsureSubscriptions(
    certes.Event("community.certes.dev/github/1/push", nil),
    certes.Event("community.certes.dev/github/1/issues", &gh.Opts{
      Repo: "meetly/app",
    }),
    certes.Event("community.certes.dev/github/1/membership", &gh.Opts{
      Org: "meetly",
    }),
    certes.Event("community.certes.dev/github/1/*", &gh.Opts{
      Repo: "meetly/meetly",
    }),
  )
}

func main() {
  certes.On("community.certes.dev/github/1/push", func(raw *certes.RawEvent) error {
    var event gh.PushEvent
    if err := raw.To(&event); err != nil {
      return err
    }

    fmt.Printf("Got Github push event: %#v\n", event)
    return nil
  })

  http.HandleFunc("/", certes.HttpHandler())
  log.Fatal(http.ListenAndServe(":8080", nil))
}

Producing events

Instead of managing back-offs, HMAC signing, and all the other complications sending webhooks bring, imagine being able to send them as easily as this:

package main

import (
  certes "github.com/hookactions/certes-sdk/go"
  pbv1 "./pb/v1"  // locally generated protobuf code
  pbv2 "./pb/v2"  // locally generated protobuf code
)

func init() {
  certes.Init("events://events.github.com") // GitHub's event gateway, local or hosted by 3rd party
}

func main() {
  meetlyGhId := "org_123"

  // Something in our code triggers a "push" event for Meetly
  certes.SendOutgoingEvent(meetlyGhId, &pbv1.PushEvent{
    Ref: "refs/tags/simple-tag",
    Before: "6113728f27ae82c7b1a177c8d03f9e96e0adf246",
    // ... data here related to "push"
  })
  certes.SendOutgoingEvent(meetlyGhId, &pbv2.PushEvent{
    Ref: "refs/tags/simple-tag",
    BeforeCommitSha: "6113728f27ae82c7b1a177c8d03f9e96e0adf246",  // versioning example of field name changing
    // ... data here related to "push"
  })
}

High-level overview

Events are defined in protobuf 3
Producers send events to their local (or 3rd party) Certes instance, these are then sent to the subscriber's Certes instance
Subscribers authenticate to Producers via the Management UI
Subscribers declare subscriptions to authenticated Producers in code
Subscribers handle incoming events from their local (or 3rd party) Certes instance

The "High-level architecture" section of the spec will help give you an understanding of the architecture behind Certes. The pages that follow that will go into depth for the technical reasons behind each decision and each component.

Full event flow examples

The last section of the spec will go over some event flow examples which will help you see how everything fits together from both the Subscriber and Producer perspectives. Let's go over a simple example of a made-up company called "Meetly" who wants to subscribe to some events from GitHub. For the purpose of this example, GitHub is not using Certes but the community has defined an adapter and events at community.certes.dev/github.

As Meetly, subscribe to GitHub events

Go to mgmt.events.meetly.com
- This can be a locally hosted Management UI by Meetly or a third-party Management UI.
Enter events://community.certes.dev/github
- The Management UI will call the Master API to retrieve events for this URI & Namespace.
- The namespace "github" is optional, but can be used when there are many namespaces available.
- The Master API will cache the schema in Meetly's local Schema Registry
I can now see what events & schema are available to be used in the github namespace. I will also see a short explanation of how authentication works for this namespace
- Authentication will be either OAuth or an Authorization header token.
Connect via OAuth to GitHub which will grant community.certes.dev GitHub api access.
I am now able to use all events in the community.certes.dev/github namespace

  func init() {
    certes.EnsureSubscription(
      certes.Event("community.certes.dev/github/org/1/memberships", &gh.Opts{
        Org: "meetly",
      }),
      certes.Event("community.certes.dev/github/repo/1/push", nil),
    )
  }

I then update my code to handle these incoming events

  func main() {
    certes.On("community.certes.dev/github/org/1/memberships", func(raw *certes.RawEvent) error {
      var event gh.PushEvent
      _ = raw.To(&event)
      // handle event
      return nil
    })
  }

Test & deploy!

Relationship to HookActions

My vision is that there will be many Certes providers, like there are for Elasticsearch. Specifically, Certes will be open source under the GPLv3 license, which means that you can run your own Certes instance but cannot distribute closed source versions of Certes. Read more about the license here: GNU General Public License v3.

HookActions will provide a Management UI for self-hosted Certes instances as well as being a provider of Certes instances if you don't want to run Certes yourself.

The Spec

I haven't started implementing the spec yet and I hope people will give me comments and feedback both here and on GitHub about this idea. Webhooks keep me up at night and I want to never have to fret about implementing them again. The full spec is available at spec.certes.dev, please leave your feedback in the comments here or on GitHub.

Results in Go - an experimental way of handling errors

Elijah Wilson — Wed, 20 May 2020 14:44:00 +0000

Error handling in Go is less than straightforward. There are many examples of how to handle errors and in Go 2 there may even be changes to how errors work.

I recently started learning Rust which has the best error handling I've worked with for a programming language. There can still be panics but most often you will use the Result<T, E> type.

Your function might look like this:

pub fn parse_str(s: &str) -> Result<i32, &str> {
  let parse_result = s.parse::<i32>();
  match parse_result {
    Ok(v) => Ok(v),
    Err(_) => Err("unable to parse string"),
  }
}

This says, parse the string input to an int value, if successful return v which is an int, otherwise return an error. Using this code would look like this:

fn main() {
    println!("Parsed! {:?}", parse_str("123").unwrap());
}

The call to .unwrap() will return an int and if there is an error it will panic. This is useful when showing examples of code we're writing, but we generally wouldn't want to use .unwrap() in production code, unless we know something the compiler doesn't.

Rust also benefits from the match statement, which is similar to Go's switch statement. If you're familiar with Go error handling, it can be very redundant to write:

func ParseStr(s string) (int, error) {
  v, err := strconv.Atoi(s)
  if err != nil {
    return 0, errors.Wrap(err, "unable to parse string")
  }
  return v, nil
}

In this single function it's not too bad, but when we want to use this function to say, parse 3 strings, it gets very verbose:

func main() {
  num1, err := ParseStr("123")
  if err != nil {
    panic(err)
  }

  num2, err := ParseStr("456")
  if err != nil {
    panic(err)
  }

  num3, err := ParseStr("789c")
  if err != nil {
    panic(err)
  }

  fmt.Printf("%d, %d, %d\n", num1, num2, num3)
}

In many code examples, error handling will be omitted by using the underscore variable:

func main() {
  num, _ := ParseStr("123")
  fmt.Printf("Parsed %d\n", num)
}

This definitely makes code examples a little bit nicer, but what if we could use a Result type like in Rust?

The main problem is that Go does not have generics, although this is being worked on for Go 2. What could we use right now?

I built a small, experimental library called results to see what this might look like.

Let's first work with some primitive scalar values like an int:

package main

import (
    "fmt"
    "strconv"

    "github.com/tizz98/results"
)

func betterParser(v string) (result results.IntResult) {
    val, err := strconv.Atoi(v)
    if err != nil {
        result.Err(err)
        return
    }

    result.Ok(val)
    return
}

func betterParser2(v string) (result results.IntResult) {
    result.Set(strconv.Atoi(v))
    return
}


func main() {
    result := betterParser("123").Unwrap()
    fmt.Printf("Got: %d\n", result)

    result2 := betterParser2("456").Unwrap()
    fmt.Printf("Got: %d\n", result2)

    // This will panic if you uncomment
    // _ = betterParser2("foo").Unwrap()
}

Personally, I think the calls to .Unwrap() make the examples easier to read and less underscores to ignore errors. betterParser2 also shows an interesting feature of my result types: the .Set(...) method. If Go had generics, it would be defined like:

func (r *Result<T>) Set(v T, err error) {
  if err != nil {
    r.Err(err)
    return
  }
  r.Ok(v)
}

In Go, it's very common to return a tuple of two values, the first being an arbitrary type T and the second being an error. I hope this will make it easier to use this library with other libraries. The second unique method is called .Tup() which will return a tuple of type T and an error. To allow you to call result.Tup() when working with existing code that doesn't use the result types.

How does this code work without generics? Code generation. You can create these result types by adding a line like so to your files:

//go:generate go run github.com/tizz98/results/cmd -pkg foo -t *Bar -tup-default nil -result-name BarPtrResult
//go:generate go run github.com/tizz98/results/cmd -pkg foo -t Bar -tup-default Bar{} -result-name BarResult
package foo

type Bar struct {
    Baz int
    Field string
}

This will generate two new result types called BarPtrResult and BarResult having an internal "value" of *Bar and Bar, respectively.

My results library has generated result types for most of Go's scalar types which you could start using today. Again, this library is experimental and I'm really just hoping to get some feedback around it!