DEV Community: Stephen Afam-Osemene

How to use a terminal

Stephen Afam-Osemene — Thu, 11 Nov 2021 20:01:45 +0000

In this article, I’ll go over the fundamentals you need to understand the terminal. The aim of this article is not to explain all the possibilities, rather it is to give you a foundation from which you can build on.

The commands I will use in this article work on MacOS and other Unix base operating systems. However, the same principles should apply to any other command line interface (e.g. Windows).

What is a terminal?

You may be already familiar with the graphical user interface (GUI), that is one way of telling your computer what to do.

The terminal is another. It is mostly text-based and can feel more primitive, but it is also often more powerful.

To open your terminal, you can just search for an application called terminal and open it. It will likely look like a blank box, with a cursor and the name of your computer

It can look quite scary but it really isn’t. Let’s jump into it and run a few commands.

Running a command

To use a terminal, we type out a command and press enter to send it as our input, then the computer runs the command and sends out an output.

A very common command is ls. This will list out the contents of the current directory.

That may look a little familiar, that is because it is the same as this:

Another command you will likely use very often is cd. What that does is to change directory. Meaning, if we want to look inside the desktop folder, we would type cd Desktop and then press enter. We can then list the files inside the Desktop folder by using the ls command.

To see some more details about the files, for example when the file was last modified, we can run ls -l.

Now that you’re getting a feel for it, let’s break it down.

The anatomy of a command

Commands are generally made up of 3 parts.

The name of the program, for example, cd or ls.
Arguments. These are any extra values passed to the program, for example, we passed Desktop as an argument to cd when we ran cd Desktop.
Flags. These are normally used to modify the behavior of the command. You will be able to identify them because they are usually prefixed with one or two hyphens. For example, we pass the -l flag to the ls command to make it print more details when we ran ls -l.

At this point, you may have some questions:

How do I know which commands are available?

If you want to learn how to do a new thing, you can search for it through google. I assure you there will be tons of examples and recommendations.

The commands you will interact with most often are not too many, and you will likely get the hang of those pretty quickly. Here are a few

touch: Used to create a new file
mkdir: To create a new directory
rm: To delete files of directories
cp: To copy files or directories
mv: To move(i.e. rename) files or directories

It is possible that you will install other command line programs on your computer, such as git, homebrew, docker and so on.

How do I know the arguments and flags to pass to a command?

Most inbuilt programs come with a manual that you can access using the command man, which is fittingly a command-line tool itself. For example, to view the manual of the ls command, we would run man ls.

A convention among most external tools is to use the —help flag to display the help for the command. As an example, see git:

What to do next?

Google, Google, Google!

Search for how to do something, and try it out. If you encounter an error, search for the error and figure out what is wrong.

Don’t try to memorize things. The commands you use frequently will become second nature pretty quickly, and the ones you don’t you can always find when you need it.

How to schedule a task at a specific time in Go

Stephen Afam-Osemene — Tue, 05 Oct 2021 16:32:05 +0000

Looking at Google search traffic, I've seen an increase in queries for "How to schedule task at specific time in Go" leading to my other article about scheduling in Go.

So I've decided to create this article to directly answer the question. You can read the other article for a more detailed look at scheduling in Go.

The one-liner

You can block until your specified time before continuing execution using this one-liner.

time.Sleep(time.Until(until))

For example:

func main() {
    // when we want to wait till
    until, _ := time.Parse(time.RFC3339, "2023-06-22T15:04:05+02:00")

    // and now we wait
    time.Sleep(time.Until(until))

    // Do what ever we want..... 🎉
}

Exiting early

In real world use, it is likely that there are cases where you want to stop the schedule prematurely, to do this, we can accept a context.Context and exit if the context is canceled.

This also has the benefit of properly stopping the timer if it is no longer needed

func waitUntil(ctx context.Context, until time.Time) {
    timer := time.NewTimer(time.Until(until))
    defer timer.Stop()

    select {
    case <-timer.C:
        return
    case <-ctx.Done():
        return
    }
}

Example:

func main() {
    // our context, for now we use context.Background()
    ctx := context.Background()

    // when we want to wait till
    until, _ := time.Parse(time.RFC3339, "2023-06-22T15:04:05+02:00")

    // and now we wait
    waitUntil(ctx, until)

    // Do what ever we want..... 🎉
}

Handling PostgreSQL Errors

Stephen Afam-Osemene — Mon, 26 Jul 2021 07:00:06 +0000

PostgresSQL is one of the most popular relational database systems and most developers work with it daily.

Understanding how to retrieve and handle database errors can save us from making unnecessary queries or handling extra logic in our code. I'll go over the main points to note and then list a few practical use cases at the end of the article.

In this article, I will be using Go and lib/pq, but the same techniques also apply to most languages and drivers.

Here are the main points to note:

1. PostgreSQL Errors have codes

Every PostgreSQL error has an error code (see the full list). Whatever library or language you use to access Postgres, you should be able to retrieve the error code to handle specific cases.

For example, if our email column in our users table has a unique constraint, we know that a new insert/update query will fail if that constraint is violated. From the documentation, we can see that the error code for a unique_violation is 23505, and so we can check for that like this:

_, err = db.Exec("INSERT INTO users(email) VALUES($1)", "existing@example.com")
// Check specifically for a unique constraint error
var pqErr *pq.Error
if errors.As(err, &pqErr) && pqErr.Code == "23505" {
    // handle the violation
}

// Check if it was a different error
if err != nil {
    panic(err)
}

2. PostgreSQL Errors can contain even more information

In many cases, the error code is not enough. Thankfully, PostgresSQL returns even more information as part of the error diagnostics. Usually, your driver will parse this information for you, but you have to know how to access it. See documentation.

For example, we may have multiple unique constraints(23505), or multiple check constraints(23514) that could be violated by our query.

In the cases, we may want to get the specific constraint name that was violated.

_, err = db.Exec("INSERT INTO users(email) VALUES($1)", "existing@example.com")
// Check specifically for a violation of the "unique_user_email" constraint
var pqErr *pq.Error
if errors.As(err, &pqErr) && pqErr.Constraint == "unique_user_email" {
    // handle the violation
}

// Check if it was a different error
if err != nil {
    panic(err)
}

Some practical use cases

If our database table has a unique constraint on the email column, instead of doing an initial query to check if a user exists with that email address, before inserting a new user, we can directly add the email, and if we get an error understand that a unique constraint violation on that column means that the email address existed before.
If we have a check constraint to make sure product quantities are not negative, we can freely add/subtract from the quantity and handle the check constraint violation to know if the quantity went negative.

The best way to style broken images

Stephen Afam-Osemene — Thu, 06 May 2021 06:38:16 +0000

Most techniques to style broken images rely on browser hacks and are very inconsistent. Let me show you a consistent way that works the same way across every browser.

This technique relies on the HTMLMediaElement.onerror property, which is supported in almost every browser, all the way back to IE9. Take a look:

On all our images, we will add an onerror event handler which will change the src of our image to a predefined one

<img src="/path/toimage" alt="my image" 
onerror="this.onerror=null;this.src='/broken-img.svg';this.classList.add('broken-img');">

What’s happening here?

We are hooking into the onerror event which fires when some form of error occurs while attempting to load or perform the media. When the event fires, we replace the src of the image to point to A NEW image and adds the class broken-img to our image.

On Swish, I use this image, an SVG that is only 855B in size, and will be loaded once no matter how many times it appears on the page.

This part this.onerror=null prevents the browser from falling into an infinite loop if the onerror handler has a problem, maybe if the new image was also unable to be loaded. Although some browsers evaluate the onerror handler just once, others will attempt to run it again and again if an error keeps occurring.

With the broken-img class, we can add any further styling that only applies when the image is broken. On Swish, I limit the height of the image to 5em

.broken-img {
    max-height: 5em;
}

Conclusion

I think this is the best way to handle broken images, we use a system that is guaranteed to work the same way across browsers and has practically no adverse effects on our website’s peformance.

Compounding Competence

Stephen Afam-Osemene — Sun, 11 Apr 2021 23:19:21 +0000

Compounding is mostly talked about in terms of financial investments, but almost everything in life compounds.

Even more powerfully, our skills, abilities and competence compounds.

I just want to draw a brief timeline of how one task after the other has led me to be able to create a GitHubFlavoured Markdown to HTML converter in pure Go with a few hours of work.

From Blackfriday to Goldmark

First time working with Markdown in Go

When building Iteretta in 2019, I later decided to have the email editor in Markdown, and then convert it to HTML before sending the email.

There are two places where this conversion needs to happen.

In the frontend editor: I used EasyMDE for this.
On the backend when generating the emails: For this, I chose a popular Go markdown library BlackFriday.

Switching to Goldmark

An Iteretta user reported a mismatch between the preview and the emails sent.

Debugging the issue led me to start learning about the different markdown specifications.

Commonmark is currently the most widely used specification, along with GitHub Flavoured Markdown, which is an extension of Commonmark.

To make sure things did not diverge, I needed to pick a specification and stick with it. Thankfully, EasyMDE already followed Commonmark, it was Blackfriday that was a little buggy.

After evaluation a number of other Go markdown libraries, I settled on Goldmark. It was well written, recently updated and easily extensible.

Extending Goldmark

Rewriting Images

When I started building Swish last year, I of course just reached for Goldmark to power the conversion on Markdown to HTML.

The first extension I had to add to Goldmark was rewriting image URLs.

Since the blogs on Swish can change their domains, that means that the link to any image can be changed.

Therefore, when a user adds an image from their uploaded files into the editor, we only put a relative URL. However, the absolute URL is needed when rendering the HTML.

Getting all images

Later, I wanted to get all the images in a post, download any missing ones, and rewrite the image url in the post to use only local images.

This was necessary for ensuring reliability, so that an external image does not eventually break your post cause it suddenly went missing.

For this, I wrote a custom renderer that walks through Goldmark's AST(abstract syntax tree) and gets the src of every image found.

Using Goldmark for PDF rendering

Having written a custom renderer before, when I first thought of converting Markdown to PDF, it didn't seem too daunting, after all, the only part I had to figure out was the PDF.

Thankfully, I found a Go package (mdtopdf) that already does this, but it used Blackfriday. I then only had to take a few hours to rewrite it using Goldmark to have a base to work from.

Update: Custom fonts are working

Here's a screenshot with Roboto pic.twitter.com/9ZCqimMlZr

— StephenAfamO - @swish_ink (@stephenafamo) April 11, 2021

## Conclusion

Most of the skills you have today are an combination and evolution of the things you picked up on your journey. And it is the same for everyone.

The things that look like genius are often just long hard work.

I wrote this post as a reminder to myself. Maybe I can read it in the future and be more patient with myself.

Authenticating 3rd Party Integrations in Go

Stephen Afam-Osemene — Sat, 03 Apr 2021 02:13:21 +0000

While building Swish, I came across an interesting problem: How do I create an Interface for authenticating integrations?.

When you want to enable a user integrate with a specific service, that can be easy to model. But it is trickier to model an authentication flow that works for almost any service.

In the article, I'll walk you through the process I went through, several versions I tried, and my current solution.

Why is this necessary?

In Swish, the plan is to support a multitude of integrations, allowing a user to connect their blog to the many tools they are also using.

Now, when authenticating to external services, there are an almost infinite amount of ways the authentication flow will work. But because I needed to create a common interface for integration authentication, I needed to figure out how this will be modeled.

Version 1: Using Forms

Describing our Form

To display our form to a user, we need a few parts.

The name of the form as a whole. Used to make sure we are reading the correct imput
Instructions to show the user to understand what the options mean
The options which represent a part of the form

package integrations

import (
    "fmt"
    "net/url"
    "template/html"

    "gitlab.com/swishink/swishink/internal/types"
)

// An authForm represents a form to be shown to the user
// The name of the form is combined with the ID of the option to
// get the name in the HTML field
// e.g. <input name="form_name[option_id]">
// This is so that we can differentiate the options when submitted
// Instructions are any information to show the user to understand what the options mean
type AuthForm struct {
    Name         string
    Instructions template.HTML
    Options      []types.Option
}

// Used to get the values for the form when given url.Values
func (a AuthForm) GetVals(source url.Values) map[string]string {
    vals := map[string]string{}
    if a.Name != "" {
        for _, key := range a.Options {
            val := source.Get(fmt.Sprintf("%s[%s]", a.Name, key.ID))
            if val != "" {
                vals[key.ID] = val
            }
        }
    }

    return vals
}

About types.Option

types.Option is a struct I have to represent a HTML form input. I will write about it in a separate article, I don't want to clutter this one.

Modeling the authentication

To model these, my Integration authentication interface would look like this:

package integrations

import "gitlab.com/swishink/swishink/internal/types"

type IntegrationAuth interface {
    // Returns the authetication form
    Form() AuthForm
    // When the auth form is submitted, the values are passed to this function
    // to validate and get the credentials
    FormValidate(vals map[string]string) types.IntegrationCreds
}

Problems with v1

The method assumes that there is only one type of authentication. We will display a form to the user, and they will submit it.

With this method we cannot support integrations that use OAuth1 (like Twitter), or OAuth2(like LinkedIn).

Version 2: Supporting More Authentication Types

Trying to add more integrations, we then need to extend our authentication to support other authentication types.

Forms: This is for integrations that require the user to enter information such as an API key, or username/password. E.g. DEV, Medium, Hashnode.
Oauth1: For integrations that are authenticated using the OAuth1 protocol. E.g. Twitter.
OAuth2: For integrations that use the OAuth2 protocol: E.g. LinkedIn

Modeling the authentication

To model these, my Integration struct would look like this:

package integrations

import (
    "context"

    "github.com/dghubble/oauth1"
    "gitlab.com/swishink/swishink/internal/types"
    "golang.org/x/oauth2"
)

type authType string

var (
    authTypeForm   authType = "form"
    authTypeOauth1 authType = "oauth1"
    authTypeOauth2 authType = "oauth2"
)

type IntegrationAuth interface {
    // Returns the authentication type to use
    Type() authType

    // Returns the authetication form
    // Used when the authentication type is "form"
    Form() AuthForm
    // When the auth form is submitted, the values are passed to this function
    // to validate and get the credentials
    FormValidate(vals map[string]string) types.IntegrationCreds

    // Gets the Oauth1 config in the oauth1 flow
    Oauth1Config() oauth1.Config
    // In the Oauth1 flow, this function is used to retrieve the oauth1 credentials
    GetOauth1Creds(ctx context.Context, config *oauth1.Config, token *oauth1.Token) (types.IntegrationCreds, error)

    // Gets the Oauth2 config in the oauth2 flow
    Oauth2Config() *oauth2.Config
    // In the Oauth2 flow, this function is used to retrieve the oauth1 credentials
    GetOauth2Creds(ctx context.Context, config *oauth2.Config, token *oauth2.Token) (types.IntegrationCreds, error)
}

This would work quite well, however, the immediate worst part of it is that every implementation will have to implement ALL the methods to be a valid Integration even if they don't use it.

We can immediately fix that by breaking up the interface.

package integrations

import (
    "context"

    "github.com/dghubble/oauth1"
    "gitlab.com/swishink/swishink/internal/types"
    "golang.org/x/oauth2"
)

type IntegrationFormAuth interface {
    // Returns the authetication form
    // Used when the authentication type is "form"
    Form() AuthForm
    // When the auth form is submitted, the values are passed to this function
    // to validate and get the credentials
    FormValidate(vals map[string]string) types.IntegrationCreds
}

type IntegrationOauth1Auth interface {
    // Gets the Oauth1 config in the oauth1 flow
    Oauth1Config() oauth1.Config
    // In the Oauth1 flow, this function is used to retrieve the oauth1 credentials
    GetOauth1Creds(ctx context.Context, config *oauth1.Config, token *oauth1.Token) (types.IntegrationCreds, error)
}

type IntegrationOauth2Auth interface {
    // Gets the Oauth2 config in the oauth2 flow
    Oauth2Config() *oauth2.Config
    // In the Oauth2 flow, this function is used to retrieve the oauth1 credentials
    GetOauth2Creds(ctx context.Context, config *oauth2.Config, token *oauth2.Token) (types.IntegrationCreds, error)
}

In the process, we also got rid of the AuthType() method. Since they are now different interfaces, we can check the AuthType by seeing what interface was implemented. For example:

package integrations

import "errors"

type authType string

var (
    authTypeForm   authType = "form"
    authTypeOauth1 authType = "oauth1"
    authTypeOauth2 authType = "oauth2"
)

func GetAuthType(integration interface{}) (authType, error) {
    switch integration.(type) {
    case IntegrationFormAuth:
        return authTypeForm, nil
    case IntegrationOauth1Auth:
        return authTypeOauth1, nil
    case IntegrationOauth2Auth:
        return authTypeOauth2, nil
    default:
        return "", errors.New("No valid authentication type")
    }
}

Alright, everything seems clean, modular. This should be great right? NO

Problems with v2

This solution starts to fall apart when we encounter a more custom authentication flow. Here are some examples.

Self hosted software: We may need to get the URL of the server, along with generated clientID/Secrets. Then maybe switch to Oauth1/Oauth2 authentication.
Multiple supported authentication types. We may need to ask the user what method they want to use to authenticate, and then use that method. This would not only require multiple steps, it would require multiple steps based on what the user chose on a previous step.

Version 3: A recursive flow

Requirements

1. A flexible number of steps

We do not know how many steps an integration authentication will take. So our flow must be able to take an arbitrary amount of steps.

First thought could be, each integration will return a list of the steps and then we will take the users through the flow.

However, this will not work because we could have situations where the number of steps changes because of something in a previous step. And this leads to our next requirement.

If the authentication flow can change depending on the values from a previous step, then each step must receive the values from the previous step.

2. Ability to display a form at the end of any step

Some authentication flows will likely require that we show a form to the user. This could be options on the authentication method, getting an API key, getting the self-hosted URL, e.t.c.

3. Ability to redirect the user to a URL at the end of any step

Some authentication flows may require a redirect, this is common on OAuth flows.

4. A way to know the flow has finished

If we will keep calling this authentication method recursively, we need to be able to signal that the authentication has been completed.

Inputs

The authentication URL. For something like OAuth, we need to know the URL to set as the Callback URL.
The values gotten from the previous step.

package integrations

import  "net/url"

type AuthReq struct {
    // The URL is the URL that the Flow should redirect to for extra steps
    // For example, it will be used to set the callbackURL in Oauth flows
    URL string

    // FormValues are the submitted values from the request
    // We use `url.Values` because that is also the type of http.Request.Form
    // Makes it easy to retrieve these values
    FormValues url.Values
}

Output

What to do next?
- A form to display to the user. OR
- A URL to redirect the user to. OR
- The final credentials of the Integration.
An error if we encountered any

package integrations

import  "gitlab.com/swishink/swishink/internal/types"

type AuthResp struct {
    // Set when we need to display a form
    Form AuthForm

    // RedirectTo is a URL we want to redirect the user to
    // Useful in OAuth authentication flows
    RedirectTo string

    // Credentials are the credentials to save
    // When this value is not nil, the authentication flow ends
    Credentials types.IntegrationCreds
}

The Flow

With these, we can create an AuthFlow interface like this:

package integrations

import "context"

type IntegrationAuth interface {
    Do(ctx context.Context, req AuthReq) (resp *AuthResp, err error)
}

Quite elegant if I do say so myself 😎.

Fitting that like many of the interfaces in the standard library, we end up with a single method interface.

Implementations

Let's go over some implementations of this IntegrationAuth.

No Authentication

Some integrations don't require credentials. Just a one-click activation.

This is what powers the Blogstreak and Disqus integrations on Siwsh.ink, so you can check it out in the demo to see how it works.

VIEW

package integrations

import (
    "context"

    "gitlab.com/swishink/swishink/internal/types"
)

type AuthFlowNone struct{}

func (a AuthFlowNone) Do(ctx context.Context, req AuthReq) (resp *AuthResp, err error) {
    return &AuthResp{Credentials: types.IntegrationCreds{}}, nil
}

Single Form

A common type of integration we will see is a single form authentication. We display one form, and we validate the values submitted.

The effect is identical to v1.

This is what powers the DEV, Medium and Hashnode integrations on Siwsh.ink, so you can check it out in the demo to see how it works.

VIEW

package integrations

import (
    "context"
    "fmt"

    "gitlab.com/swishink/swishink/internal/types"
)

type AuthFlowForm struct {
    Form      AuthForm
    Validator func(ctx context.Context, val map[string]string) (types.IntegrationCreds, error)
}

func (a AuthFlowForm) Do(ctx context.Context, req AuthReq) (*AuthResp, error) {
    ourVals := a.Form.GetVals(req.FormValues)

    // if we have no form values, display the form
    if len(ourVals) == 0 {
        return &AuthResp{Form: a.Form}, nil
    }

    // If the form is submitted, validate
    creds, err := a.Validator(ctx, a.Form.GetVals(req.FormValues))
    if err != nil {
        return nil, fmt.Errorf("Could not validate submitted form: %w", err)
    }

    return &AuthResp{Credentials: creds}, nil
}

Oauth1

This is an implementation for an Oauth1 flow. An integration such as Twitter will be able to use this.

This is what powers the Twitter integration on Siwsh.ink, so you can check it out in the demo to see how it works.

VIEW

package integrations

import (
    "context"
    "errors"
    "fmt"

    "github.com/dghubble/oauth1"
    "gitlab.com/swishink/swishink/internal/types"
)

type AuthFlowOauth1 struct {
    // The Oauth1 config
    Config *oauth1.Config

    // Used to save a request token/secret pair
    SaveTokenPair func(token, secret string)

    // Uset to retrieve a request secret given the token
    GetSecretByToken func(token string) (secret string)

    // Set by the user, use the config/token to get and marshal the credentials to save
    GetCreds func(ctx context.Context, config *oauth1.Config, token *oauth1.Token) (types.IntegrationCreds, error)
}

func (a AuthFlowOauth1) Do(ctx context.Context, req AuthReq) (*AuthResp, error) {
    if req.URL == "" {
        return nil, fmt.Errorf("No authURL in request context")
    }

    if a.Config == nil {
        return nil, fmt.Errorf("oauth2 config is nil")
    }
    a.Config.CallbackURL = req.URL

    // To know if we are in the Oauth flow, we check if the oauth verifier parameter is present
    // If it is not present, redirect to authorizationURL
    if req.FormValues.Get("oauth_verifier") == "" {
        return a.redirectToAuthorizationURL()
    }

    // If it is present, it means we got a callback
    // Validate the callback
    return a.validateCallback(ctx, req)
}

func (a AuthFlowOauth1) redirectToAuthorizationURL() (*AuthResp, error) {
    requestToken, requestSecret, err := a.Config.RequestToken()
    if err != nil {
        return nil, fmt.Errorf("could not get request token: %w", err)
    }
    a.SaveTokenPair(requestToken, requestSecret)

    authorizationURL, err := a.Config.AuthorizationURL(requestToken)
    if err != nil {
        return nil, fmt.Errorf("could not get authorizationURL: %w", err)
    }

    return &AuthResp{RedirectTo: authorizationURL.String()}, nil
}

func (a AuthFlowOauth1) validateCallback(ctx context.Context, req AuthReq) (*AuthResp, error) {
    requestToken := req.FormValues.Get("oauth_token")
    verifier := req.FormValues.Get("oauth_verifier")
    if requestToken == "" || verifier == "" {
        return nil, errors.New("Request missing oauth_token or oauth_verifier")
    }

    // Get request secret
    requestSecret := a.GetSecretByToken(requestToken)
    accessToken, accessSecret, err := a.Config.AccessToken(requestToken, requestSecret, verifier)
    if err != nil {
        return nil, fmt.Errorf("could not get access token and secret: %w", err)
    }

    creds, err := a.GetCreds(ctx, a.Config, oauth1.NewToken(accessToken, accessSecret))
    if err != nil {
        return nil, fmt.Errorf("Could not get oauth1 credentials: %w", err)
    }

    return &AuthResp{Credentials: creds}, nil
}

OAuth2

OAuth2 authentication is pretty much the industry standard these days. Facebook, Google, MailChimp, GitHub, LinkedIn and many more use OAuth2 authentication.

This is what powers the LinkedIn integration on Siwsh.ink. So you can check it out in the demo to see how it works.

VIEW

package integrations

import (
    "context"
    "fmt"

    "gitlab.com/swishink/swishink/internal"
    "gitlab.com/swishink/swishink/internal/types"
    "golang.org/x/oauth2"
)

type AuthFlowOauth2 struct {
    Config *oauth2.Config

    // Set by the user, will be called in the Finish method
    GetCreds func(ctx context.Context, config *oauth2.Config, token *oauth2.Token) (types.IntegrationCreds, error)
}

func (a AuthFlowOauth2) Do(ctx context.Context, req AuthReq) (*AuthResp, error) {
    if req.URL == "" {
        return nil, fmt.Errorf("No authURL in auth request")
    }

    if a.Config == nil {
        return nil, fmt.Errorf("oauth2 config is nil")
    }
    a.Config.RedirectURL = req.URL

    // To know if we are in the Oauth flow, we check if the code parameter is present
    // If it is not present, redirect to the authorization URL
    if req.FormValues.Get("code") == "" {
        return a.redirectToAuthorizationURL()
    }

    // If it is present, validate the callback
    return a.validateCallback(ctx, req)
}

func (a AuthFlowOauth2) redirectToAuthorizationURL() (*AuthResp, error) {
    // This is a dashboard page, we already validate that is is a legitimate user/request
    // So we can just use a random code
    authorizationURL := a.Config.AuthCodeURL(internal.GenRandomToken(12))
    return &AuthResp{RedirectTo: authorizationURL}, nil
}

func (a AuthFlowOauth2) validateCallback(ctx context.Context, req AuthReq) (*AuthResp, error) {
    code := req.FormValues.Get("code")
    token, err := a.Config.Exchange(ctx, code)
    if err != nil {
        return nil, fmt.Errorf("code exchange failed: %s", err.Error())
    }

    creds, err := a.GetCreds(ctx, a.Config, token)
    if err != nil {
        return nil, fmt.Errorf("Could not get oauth2 credentials: %w", err)
    }

    return &AuthResp{Credentials: creds}, nil
}

WordPress: A more complex flow

To integrate with WordPress, we needed an interesting flow. Here are the things to consider.

There's 2 flavors of WordPress. Hosted version(.com) and the Self hosted version(.org).
The hosted version (.com) uses OAuth2, while the self-hosted version needs a plugin to generate an application password which we would then use to validate.

Here's how the authentication flow for WordPress goes:

Ask if it is a Hosted or Self-hosted WordPress install
If it is Hosted, revert to an OAuth2 flow
If it is self-hosted, ask for the website URL, the username and the application password. Then validate after it is submitted.

This is what powers the WordPress integration on Siwsh.ink. So you can check it out in the demo to see how it all works.

The full implementation is shown below.

VIEW

package wordpress

import (
    "bytes"
    "context"
    "encoding/json"
    "fmt"
    "html/template"
    "io/ioutil"
    "net/http"
    "net/url"
    "path/filepath"
    "time"

    "gitlab.com/swishink/swishink/internal/types"
    "gitlab.com/swishink/swishink/services/integrations"
    "golang.org/x/oauth2"
)

const (
    // Keys for our credentials
    wpAuthKey = "wp_auth"

    // Keys for self-hosted wordpress credentials
    wpURLKey         = "wp_url"
    wpUsernameKey    = "wp_user_login"
    wpAppPasswordKey = "wp_app_password"

    // Keys for wordpress.com credentials
    wpUserIDKey = "wp_user_id"
    wpBlogIDKey = "wp_blog_id"
    wpTokenKey  = "wp_token"

    wpTypeKey = "wp_type"
    wpTypeCom = "wp_type_com"
    wpTypeOrg = "wp_type_org"
)

var wordpressOauth2Endpoint = oauth2.Endpoint{
    AuthURL:  "https://public-api.wordpress.com/oauth2/authorize",
    TokenURL: "https://public-api.wordpress.com/oauth2/token",
}

type wordpress struct {
    ClientID     string
    ClientSecret string
}

func (w *wordpress) Auth() integrations.AuthFlow {
    return wpAuthFlow{
        Config: w.getOauth2Config(),
    }
}

type wpAuthFlow struct {
    Config *oauth2.Config
}

func (w wordpress) getOauth2Config() *oauth2.Config {
    return &oauth2.Config{
        ClientID:     w.ClientID,
        ClientSecret: w.ClientSecret,
        // Needs a "global" scope to access the v2 API
        // Scopes:       []string{"auth", "posts", "media"},
        Endpoint: wordpressOauth2Endpoint,
    }
}

func (w wpAuthFlow) oauth2() integrations.AuthFlowOauth2 {
    return integrations.AuthFlowOauth2{
        Config:   w.Config,
        GetCreds: w.validateComCreds,
    }
}

func (w wpAuthFlow) Do(ctx context.Context, req integrations.AuthReq) (*integrations.AuthResp, error) {
    if req.URL == "" {
        return nil, fmt.Errorf("No authURL in request context")
    }

    comOrOrgVals := comORorg.GetVals(req.FormValues)
    if len(comOrOrgVals) > 0 {
        wpType := comOrOrgVals["wp_type"]
        switch wpType {
        case wpTypeOrg:
            return &integrations.AuthResp{Form: wpOrgDetails}, nil
        default:
            return w.oauth2().Do(ctx, req)
        }
    }

    wpOrgVals := wpOrgDetails.GetVals(req.FormValues)
    if len(wpOrgVals) > 0 {
        creds, err := wpValidateOrgCreds(wpOrgVals)
        if err != nil {
            return nil, fmt.Errorf("Could not validate wp org credentials: %w", err)
        }

        return &integrations.AuthResp{Credentials: creds}, nil
    }

    // If "code" exists in the query parameters, continue the Oauth2 authentication
    if req.FormValues.Get("code") != "" {
        return w.oauth2().Do(ctx, req)
    }

    // If nothing else, sent the first form
    return &integrations.AuthResp{Form: comORorg}, nil
}

func (w wpAuthFlow) validateComCreds(ctx context.Context, config *oauth2.Config, token *oauth2.Token) (types.IntegrationCreds, error) {
    req, err := http.NewRequestWithContext(ctx, http.MethodGet,
        fmt.Sprintf("https://public-api.wordpress.com/oauth2/token-info?client_id=%s&token=%s",
            w.Config.ClientID, token.AccessToken), nil)
    if err != nil {
        return nil, fmt.Errorf("could not create WordPress token verify request: %w", err)
    }

    resp, err := http.DefaultClient.Do(req)
    if err != nil {
        return nil, fmt.Errorf("could not do WordPress token verify request: %w", err)
    }
    defer resp.Body.Close()

    bodyBytes, _ := ioutil.ReadAll(resp.Body)

    if resp.StatusCode != 200 {
        return nil, fmt.Errorf("Got status code %d in WordPress token verify response: %s", resp.StatusCode, string(bodyBytes))
    }

    var response = struct {
        ClientID string `json:"client_id"`
        UserID   string `json:"user_id"`
        BlogID   string `json:"blog_id"`
        Scope    string `json:"scope"`
    }{}
    if err = json.NewDecoder(bytes.NewBuffer(bodyBytes)).Decode(&response); err != nil {
        return nil, fmt.Errorf("could not decode WordPress token verify response: %w", err)
    }

    return types.IntegrationCreds{
        wpTypeKey:   wpTypeCom,
        wpUserIDKey: response.UserID,
        wpBlogIDKey: response.BlogID,
        wpTokenKey:  token.AccessToken,
    }, nil
}

func wpValidateOrgCreds(vals map[string]string) (types.IntegrationCreds, error) {

    wpURL, err := url.Parse(vals[wpURLKey])
    if err != nil {
        return nil, fmt.Errorf("could not parse wordpress url")
    }

    wpURL.Path = filepath.ToSlash(filepath.Join(wpURL.Path, "wp-json", "wp", "v2", "users", "me"))
    ctx, cancel := context.WithTimeout(context.Background(), time.Second*5)
    defer cancel()

    req, err := http.NewRequestWithContext(ctx, http.MethodGet, wpURL.String(), nil)
    if err != nil {
        return nil, fmt.Errorf("could not create org /me request")
    }

    req.SetBasicAuth(vals[wpUsernameKey], vals[wpAppPasswordKey])

    resp, err := http.DefaultClient.Do(req)
    if err != nil {
        return nil, fmt.Errorf("Could not do WordPress /me request: %w", err)
    }
    defer resp.Body.Close()

    if resp.StatusCode != http.StatusOK {
        return nil, fmt.Errorf("Got status code %d from WordPress /me request", resp.StatusCode)
    }

    creds := types.IntegrationCreds{}
    creds[wpTypeKey] = wpTypeOrg
    creds[wpURLKey] = vals[wpURLKey]
    creds[wpUsernameKey] = vals[wpUsernameKey]
    creds[wpAppPasswordKey] = vals[wpAppPasswordKey]

    return creds, nil
}

var comORorg = integrations.AuthForm{
    Name: wpTypeKey,
    // Instructions: "Is your WordPress blog hosted on WordPress.com or do you have a self-hosted instance?",
    Options: []types.Option{
        {
            ID:       wpTypeKey,
            Required: true,
            Label:    "Type of website",
            Kind:     types.MustKind("select"),
            HelpText: "Is your WordPress blog hosted on WordPress.com or do you have a self-hosted instance?",
            Options: []types.SelectOption{
                {
                    Label: "WordPress.com",
                    Value: wpTypeCom,
                },
                {
                    Label: "Self Hosted (WordPress.org)",
                    Value: wpTypeOrg,
                },
            },
        },
    },
}

// Form to get the URL of the self-hosted wordpress instance
var wpOrgDetails = integrations.AuthForm{
    Name:         "wp_api_details",
    Instructions: template.HTML(`You should have <a href="https://wordpress.org/plugins/application-passwords/" target="_blank" rel="noopener">this plugin</a> installed on your site so we can authenticate to the API.<br>You should then generate a new application password from your profile settings page.`),
    Options: []types.Option{
        {
            ID:       wpURLKey,
            Label:    "Website URL",
            HelpText: "The URL of your WordPress website.",
            Required: true,
            Kind:     types.MustKind("url"),
        },
        {
            ID:       wpUsernameKey,
            Label:    "Username",
            HelpText: "Your username on the WordPress website.",
            Required: true,
            Kind:     types.MustKind("text"),
        },
        {
            ID:       wpAppPasswordKey,
            Label:    "Application Password",
            HelpText: "The generated application password. <b>DO NOT</b> use your login password!",
            Required: true,
            Kind:     types.MustKind("text"),
        },
    },
}

Conclusion

We went through a couple iterations and ended up with an authentication flow that can be used for almost any possible flow.

Works somewhat like recursion. We'll keep calling the method with the values from the previous step until we get the final credentials or an error.

We also looked a few implementations of this authentication flow to show how it looks in practice.

If you have any questions/comments, let me know on Twitter.

Gracefully shutting down multiple workers in Go

Stephen Afam-Osemene — Sat, 16 May 2020 21:30:56 +0000

In this post, we’ll see how to write workers in Go, how to gracefully shut them down. We’ll also look at how to coordinate multiple workers.

Finally, I’ll introduce Orchestra, a package I built which makes it a lot easier to manage and gracefully shutdown workers.

What are workers?

When we write programs, we are often in situations when we have function that runs until either it faces an error, or we explicitly order it to stop. A good example of this is the http.Server. When we start a web server in Go, the program will block until we ask it to stop or it encounters.

Here’s a simple webserver:

package main

import (
    "io"
    "net/http"
)

func main() {
    // Hello world, the web server

    helloHandler := func(w http.ResponseWriter, req *http.Request) {
        io.WriteString(w, "Hello, world!\n")
    }

    http.HandleFunc("/hello", helloHandler)
    err := http.ListenAndServe(":8080", nil)
    if err != nil {
        panic(err)
    }
}

We can write a function that behaves the same way:

package main

func main()
    err := myWorker()
    if err != nil {
        panic(err)
    }
}

func myWorker() error {
    for {
        err := doSomethingRepeatedly()
        if err != nil {
            return err
        }
    }

    return nil
}

In both these programs, we can ask them to stop by killing the main process (ctrl + C) and that will forcefully close the program.

Gracefully Shutting Down Workers

To shutdown a worker grace fully we should listen for signals passed to the program. The main ones to look out for are:

SIGINT: The interrupt signal. The terminal sends it to the foreground process when the user presses ctrl-c. The default behavior is to terminate the process, but it can be caught or ignored. The intention is to provide a mechanism for an orderly, graceful shutdown.
SIGTERM: The termination signal. The default behavior is to terminate the process, but it also can be caught or ignored. The intention is to kill the process, gracefully or not, but to first allow it a chance to cleanup.

Credit for the explanation of what the signals mean

For the webserver, since Go 1.8 (which was a long time ago), the http.Server type has a Shutdown method which gracefully closes the connections.

Here’s an example of how to use it:

package main

import (
    "context"
    "log"
    "net/http"
    "os"
    "os/signal"
    "syscall"
)

func main() {
    var srv http.Server

    idleConnsClosed := make(chan struct{})
    go func() {
        signals := make(chan os.Signal, 1)
        signal.Notify(signals, os.Interrupt, syscall.SIGTERM)
        <-signals

        // We received an interrupt signal, shut down.
        if err := srv.Shutdown(context.Background()); err != nil {
            // Error from closing listeners, or context timeout:
            log.Printf("HTTP server Shutdown: %v", err)
        }
        close(idleConnsClosed)
    }()

    if err := srv.ListenAndServe(); err != http.ErrServerClosed {
        // Error starting or closing listener:
        log.Fatalf("HTTP server ListenAndServe: %v", err)
    }

    <-idleConnsClosed
}

Let’s create something similar for our worker:

package main

import (
    "context"
    "log"
    "os"
    "os/signal"
    "syscall"
)

var shouldStop = false

func main() {
    complete := make(chan struct{})

    go func() {
        signals := make(chan os.Signal, 1)
        signal.Notify(signals, os.Interrupt, syscall.SIGTERM)
        <-signals

        shouldStop = true

        close(complete)
    }()

    if err := myWorker(); err != nil {
        // Error in our worker:
        log.Fatalf("Worker error: %v", err)
    }

    <-complete
}

func myWorker() error {
    for !shouldStop {
        err := doSomethingRepeatedly()
        if err != nil {
            return err
        }
    }

    return nil
}

Multiple Workers

This seems somewhat straightforward with just a single long running process. However, it becomes trickier.

At first glance, it seems we can just do something like this:

package main

import (
    "context"
    "log"
    "net/http"
    "os"
    "os/signal"
    "syscall"
)

var shouldStop = false

func main() {
    var srv http.Server

    complete := make(chan struct{})

    go func() {
        signals := make(chan os.Signal, 1)
        signal.Notify(signals, os.Interrupt, syscall.SIGTERM)
        <-signals

        // Ask our worker to stop
        shouldStop = true

        // Shutdown the server
        if err := srv.Shutdown(context.Background()); err != nil {
            // Error from closing listeners, or context timeout:
            log.Printf("HTTP server Shutdown: %v", err)
        }

        close(complete)
    }()

    go func() {
        if err := myWorker(); err != nil {
            // Error in our worker:
            log.Fatalf("Worker error: %v", err)
        }
    }()

    go func() {
        if err := srv.ListenAndServe(); err != http.ErrServerClosed {
            // Error starting or closing listener:
            log.Fatalf("HTTP server ListenAndServe: %v", err)
        }
    }()

    <-complete
}

func myWorker() error {
    for !shouldStop {
        err := doSomethingRepeatedly()
        if err != nil {
            return err
        }
    }

    return nil
}

But that does not work as expected. Here’s why:

First, we did not stop the workers asynchronously. It does not seem really needed in the particular example, so let’s leave that problem for now.

More importantly, in the example above, if any of the workers encounters an error, the program rudely exits! Our entire shutdown sequence is ignored.

Coordinating with Context

In idiomatic Go, the ideal way would be to pass a context.Context to our worker, and ask it to shut itself down once the context is done.

Using this method, instead of calling log.Fatalf() when a worker encounters an error, we can simply cancel the context, and it will then initiate the shutdown sequence.

Since we are not calling the shutdown functions directly, let’s use sync.WaitGroup to know when all our workers have exited.

This means we’d have to rework our code to look like this:

package main

import (
    "context"
    "fmt"
    "log"
    "net/http"
    "os"
    "os/signal"
    "sync"
    "syscall"
    "time"
)

func main() {
    ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)

    var wg sync.WaitGroup

    go func() {
        signals := make(chan os.Signal, 1)
        signal.Notify(signals, os.Interrupt, syscall.SIGTERM)
        <-signals

        cancel()
    }()

    wg.Add(1)
    go func() {
        if err := myWorker(ctx); err != nil {
            cancel()
        }
        wg.Done()
    }()

    wg.Add(1)
    go func() {
        if err := startServer(ctx); err != nil {
            cancel()
        }
        wg.Done()
    }()

    wg.Wait()
}

func myWorker(ctx context.Context) error {

    var shouldStop = false

    go func() {
        <-ctx.Done()
        shouldStop = true
    }()

    for !shouldStop {
        err := doSomethingRepeatedly()
        if err != nil {
            return err
        }
    }

    return nil
}

func startServer(ctx context.Context) error {

    var srv http.Server

    go func() {
        <-ctx.Done() // Wait for the context to be done

        // Shutdown the server
        if err := srv.Shutdown(context.Background()); err != nil {
            // Error from closing listeners, or context timeout:
            log.Printf("HTTP server Shutdown: %v", err)
        }
    }()

    if err := srv.ListenAndServe(); err != http.ErrServerClosed {
        // Error starting or closing listener:
        return fmt.Errorf("HTTP server ListenAndServe: %w", err)
    }

    return nil
}

This neatly satisfies all our conditions. Yay!

Using Orchestra

To reduce all this boilerplate, I created a package to do most of this.

Orchestra is a library to manage long running go processes.

At the heart of the library is an interface called Player

// Player is a long running background worker
type Player interface {
    Play(context.Context) error
}

All a type needs to do to satisfy the interface is to have a Play method that will gracefully shutdown when the context is done.

It can also return an error if it encounters a problem when playing.

Next, there’s the Conductor type (which itself is a Player)

// Conductor is a group of workers. It is also a Player itself **evil laugh**
type Conductor struct {
    Timeout time.Duration
    Players map[string]Player
}

With the conductor, you add Players to it, and when you call the Play method on the conductor, it will start the Players under it and gracefully shut them all down when the main context is done.

The timeout is there incase there is a Player that refused to stop.

It also contains a few helper functions:

PlayUntilSignal(p Player, sig ...os.Signal): This will start a player with a context, and close the context once it receives any of the signals provided.
PlayerFunc(func(context.Context) error): This is a quick way to convert a standalone function into a type that satisfies the Player interface.
ServerPlayer{*http.Server}: This is a type that embeds the *http.Server and extends it to satisfy the Player interface.

More details can be found in the README.

With these, we can simplify our example code into this:

package main

import (
    "context"
    "net/http"
    "os"
    "syscall"
    "time"

    "github.com/stephenafamo/orchestra"
)

func main() {
    var srv = &http.Server{}

    // creates a player from a myWorker function
    workerPlayer := orchestra.PlayerFunc(myWorker)
    // A player from a server
    serverPlayer := orchestra.ServerPlayer{srv}

    // A conductor to control them all
    conductor := &orchestra.Conductor{
        Timeout: 5 * time.Second,
        Players: map[string]orchestra.Player{
            // the names are used to identify the players
            // both in logs and the returned errors
            "worker": workerPlayer,
            "server": serverPlayer,
        },
    }

    // Use the conductor as a Player
    err := orchestra.PlayUntilSignal(conductor, os.Interrupt, syscall.SIGTERM)
    if err != nil {
        panic(err)
    }
}

func myWorker(ctx context.Context) error {

    var shouldStop = false

    go func() {
        <-ctx.Done()
        shouldStop = true
    }()

    for !shouldStop {
        err := doSomethingRepeatedly()
        if err != nil {
            return err
        }
    }

    return nil
}

Conclusion

As stated in the first paragraph, we’ve seen how to write workers in Go, how to gracefully shut them down and I’ve introduced Orchestra, a package I built which makes it a lot easier to manage and gracefully shutdown workers.

The post Gracefully shutting down multiple workers in Go appeared first on Stephen AfamO's Blog.

Better scheduling in Go

Stephen Afam-Osemene — Sun, 22 Sep 2019 11:43:55 +0000

In a lot of situations, we would like to be able to schedule functions in Go. While there are many current tools for doing scheduling (such as Cron), I would much prefer if the entire working of my program was contained in the same code/binary and I did not have to go to the system crontab to discover what is going on.

Thankfully, the time standard library already comes with A LOT of tools to schedule events. Let's look at some of them and when to use it.

Check out Kronika, a package that adds some nice helpers around the time package.

https://github.com/stephenafamo/kronika

Using `time.After()`

time.After() allows us to perform an action after a duration. For example:

NOTE: The helper constants time.Second, time.Minute and time.Hour are all of type time.Duration. So if we have to supply a duration, we can multiply these constants by a number (e.g. time.Second * 5) and the expression returns a time.Duration.

package main

import (
    "fmt"
    "time"
)

func main() {
    // This will block for 5 seconds and then return the current time
    theTime := <-time.After(time.Second * 5)
    fmt.Println(theTime.Format("2006-01-02 15:04:05"))
}

2019-09-22 09:33:05

Using `time.Ticker`

time.After() is great for one-time actions, but the power of cron jobs are in performing repeated actions.

So, to do that we use a time.Ticker. For most use cases, we can use the helper function time.Tick() to create a ticker. For example:

package main

import (
    "fmt"
    "time"
)

func main() {
    // This will print the time every 5 seconds
    for theTime := range time.Tick(time.Second * 5) {
        fmt.Println(theTime.Format("2006-01-02 15:04:05"))
    }
}

2019-09-22 10:07:54
2019-09-22 10:07:59
2019-09-22 10:08:04
2019-09-22 10:08:09
2019-09-22 10:08:14

NOTE: When using a Ticker, the first event will be triggered AFTER the delay.

Dangers of using `time.Tick()`

When we use the time.Tick() function, we do not have direct access to the underlying time.Ticker and so we cannot close it properly.

If we never need to explicitly stop the ticker (for example if the ticker will run all the time), then this may not be an issue. However, if we simply ignore the ticker, the resources will not be freed up and it will not be garbage collected.

Limitations using `time.Tick()`

There are several things we can't easily do with time.Ticker:

Specify a start time
Stop the ticker

Extending `time.Tick()` using a custom function

To overcome the limitations of time.Tick(), I've created a helper function which I use in my projects.

func cron(ctx context.Context, startTime time.Time, delay time.Duration) <-chan time.Time {
    // Create the channel which we will return
    stream := make(chan time.Time, 1)

    // Calculating the first start time in the future
    // Need to check if the time is zero (e.g. if time.Time{} was used)
    if !startTime.IsZero() {
        diff := time.Until(startTime)
        if diff < 0 {
            total := diff - delay
            times := total / delay * -1

            startTime = startTime.Add(times * delay)
        }
    }

    // Run this in a goroutine, or our function will block until the first event
    go func() {

        // Run the first event after it gets to the start time
        t := <-time.After(time.Until(startTime))
        stream <- t

        // Open a new ticker
        ticker := time.NewTicker(delay)
        // Make sure to stop the ticker when we're done
        defer ticker.Stop()

        // Listen on both the ticker and the context done channel to know when to stop
        for {
            select {
            case t2 := <-ticker.C:
                stream <- t2
            case <-ctx.Done():
                close(stream)
                return
            }
        }
    }()

    return stream
}

What's happening in the function

The function receives 3 parameters.

A Context: The ticker will be stopped whenever the context is cancelled. So we can create a context with a cancel function, or a timeout, or a deadline, and when that context is cancelled, the function will gracefully release it's resources.
A Time: The start time is used as a reference to know when to start ticking. If the start time is in the future, it will not start ticking until that time. If it is in the past. The function calculates the first event that will be in the future by adding an appropriate multiple of the delay.
A Duration: This is the interval between ticks. Calculated from the start time.

Examples of using the custom function

Run on Tuesdays by 2pm

ctx := context.Background()

startTime, err := time.Parse(
    "2006-01-02 15:04:05",
    "2019-09-17 14:00:00",
) // is a tuesday
if err != nil {
    panic(err)
}

delay := time.Hour * 24 * 7 // 1 week

for t := range cron(ctx, startTime, delay) {
    // Perform action here
    log.Println(t.Format("2006-01-02 15:04:05"))
}

Run every hour, on the hour

ctx := context.Background()

startTime, err := time.Parse(
    "2006-01-02 15:04:05",
    "2019-09-17 14:00:00",
) // any time in the past works but it should be on the hour
if err != nil {
    panic(err)
}

delay := time.Hour // 1 hour

for t := range cron(ctx, startTime, delay) {
    // Perform action here
    log.Println(t.Format("2006-01-02 15:04:05"))
}

Run every 10 minutes, starting in a week

ctx := context.Background()

startTime, err := time.Now().AddDate(0, 0, 7) // see https://golang.org/pkg/time/#Time.AddDate
if err != nil {
    panic(err)
}

delay := time.Minute * 10 // 10 minutes

for t := range cron(ctx, startTime, delay) {
    // Perform action here
    log.Println(t.Format("2006-01-02 15:04:05"))
}

Conclusion

With this function, I have much better control over scheduling in my projects. Hopefully, it is also of some use to you.

Check out Kronika, a package that adds some nice helpers around the time package.

https://github.com/stephenafamo/kronika

Let me know what you think.

The post Better scheduling in Go appeared first on Stephen AfamO's Blog.

Implementing an Event Driven System in Go

Stephen Afam-Osemene — Sun, 01 Sep 2019 14:12:23 +0000

In this post, we will implement an event driven system in Go.

We are going to imagine a fictional application where we want to send out events for when a new account is created and another for when an account is deleted.

Let’s assume that the current structure of our program looks like this:

working-dir
|
|__auth.go/
| |__auth.go
|
|__main.go
|__go.mod
|__go.sum

We would like this system to:

Be type safe. No interface{}, no need to type cast.
Be able to define a custom payload for each event.

Defining Events

To make it safe, we will create our events in a separate package and only export the complete events. Let’s call this package events. We’ll update our directory structure like this.

working-dir
|
|__auth.go/
| |__auth.go
|
|__events/
|__main.go
|__go.mod
|__go.sum

Each event would be a unique type and would define the required payload for each event. Every handler would explicitly know what data it will receive.

Here would be our event for when a user is created:

// events/user_created.go
package events

import (
    "time"
)

var UserCreated userCreated

// UserCreatedPayload is the data for when a user is created
type UserCreatedPayload struct {
    Email string
    Time  time.Time
}

type userCreated struct {
    handlers []interface{ Handle(UserCreatedPayload) }
}

// Register adds an event handler for this event
func (u *userCreated) Register(handler interface{ Handle(UserCreatedPayload) }) {
    u.handlers = append(u.handlers, handler)
}

// Trigger sends out an event with the payload
func (u userCreated) Trigger(payload UserCreatedPayload) {
    for _, handler := range u.handlers {
        go handler.Handle(payload)
    }
}

We can then create another event for when a user is deleted:

// events/user_deleted.go
package events

import (
    "time"
)

var UserDeleted userDeleted

// UserDeletedPayload is the data for when a user is Deleted
type UserDeletedPayload struct {
    Email string
    Time  time.Time
}

type userDeleted struct {
    handlers []interface{ Handle(UserDeletedPayload) }
}

// Register adds an event handler for this event
func (u *userDeleted) Register(handler interface{ Handle(UserDeletedPayload) }) {
    u.handlers = append(u.handlers, handler)
}

// Trigger sends out an event with the payload
func (u userDeleted) Trigger(payload UserDeletedPayload) {
    for _, handler := range u.handlers {
        go handler.Handle(payload)
    }
}

Our directory structure now looks like this:

working-dir
|
|__auth.go/
|   |__auth.go
|
|__events/
|   |__user_created.go
|   |__user_deleted.go
|
|__main.go
|__go.mod
|__go.sum

A good thing about this system is that the event variable types are not exported, therefore, they cannot be changed or assigned to something different outside the package.

Listening for Events

To listen for an event, we import our events package and then register our handler to an event.

First, we create a listener that sends notifications to an admin and to slack when a user is created.

// create_notifier.go
package main

import (
    "time"

    "github.com/stephenafamo/demo/events"
)

func init() {
    createNotifier := userCreatedNotifier{
        adminEmail: "the.boss@example.com",
        slackHook: "https://hooks.slack.com/services/...",
    }

    events.UserCreated.Register(createNotifier)
}

type userCreatedNotifier struct{
    adminEmail string
    slackHook string
}

func (u userCreatedNotifier) notifyAdmin(email string, time time.Time) {
    // send a message to the admin that a user was created
}

func (u userCreatedNotifier) sendToSlack(email string, time time.Time) {
    // send to a slack webhook that a user was created
}

func (u userCreatedNotifier) Handle(payload events.UserCreatedPayload) {
    // Do something with our payload
    u.notifyAdmin(payload.Email, payload.Time)
    u.sendToSlack(payload.Email, payload.Time)
}

Let’s add another listener that does the same when a user is deleted.

// delete_notifier.go
package main

import (
    "time"

    "github.com/stephenafamo/demo/events"
)

func init() {
    createNotifier := userCreatedNotifier{
        adminEmail: "the.boss@example.com",
        slackHook: "https://hooks.slack.com/services/...",
    }

    events.UserCreated.Register(createNotifier)
}

type userCreatedNotifier struct{
    adminEmail string
    slackHook string
}

func (u userCreatedNotifier) notifyAdmin(email string, time time.Time) {
    // send a message to the admin that a user was created
}

func (u userCreatedNotifier) sendToSlack(email string, time time.Time) {
    // send to a slack webhook that a user was created
}

func (u userCreatedNotifier) Handle(payload events.UserCreatedPayload) {
    // Do something with our payload
    u.notifyAdmin(payload.Email, payload.Time)
    u.sendToSlack(payload.Email, payload.Time)
}

Now, we will have a directory structure that looks like this:

working-dir
|
|__auth.go/
|   |__auth.go
|
|__events/
|   |__user_created.go
|   |__user_deleted.go
|
|__create_notifier.go
|__delete_notifier.go
|__main.go
|__go.mod
|__go.sum

Triggering Events

Now that we have our listeners set up, we can then trigger these events from our auth package (or anywhere else).

// auth.go
package auth

import (
    "time"

    "github.com/stephenafamo/demo/events"
    // Other imported packages
)

func CreateUser() {
    // ...
    events.UserCreated.Trigger(events.UserCreatedPayload{
        Email: "new.user@example.com",
        Time: time.Now(),
    })
    // ...
}

func DeleteUser() {
    // ...
    events.UserDeleted.Trigger(events.UserDeletedPayload{
        Email: "deleted.user@example.com",
        Time: time.Now(),
    })
    // ...
}

Conclusion

We saw a way to define events in a type safe manner, how to listen for these events and how to trigger them.

Nothing fancy. As with all things Go, a good solution is a boring solution.

The post Implementing an Event Driven System in Go appeared first on Stephen AfamO's Blog.

Tailwind CSS: Getting Set Up in Minutes

Stephen Afam-Osemene — Fri, 05 Jul 2019 14:26:00 +0000

Tailwind CSS is a great CSS framework. Utility first, fully customizable and just a pleasure to use.

I had been seeing it around and wanted to try it myself, however the set up instructions can feel a bit tedious, especially if you’re not familiar with all the front-end build tools and CSS preprocessors.

So, after going through it myself, I decided to write a guide on the fastest way to set up Tailwind CSS.

Prerequisites

Node
NPM

Assumptions

You are familiar with working from the command line
All commands are run from the root of the project
Our main CSS file is at src/css/app.css
We want our compiled file to be at build/css/app.css

Step 1: Install Laravel Mix

Laravel Mix is a great tool that greatly simplifies webpack configuration. I have written a previous article on it here.

While we’re going to use it only to compile CSS here, it can do so much more. Check out my previous article and the official documentation.

npm init -y
npm install laravel-mix --save-dev
cp node_modules/laravel-mix/setup/webpack.mix.js ./

We also need to add some scripts to our package.json

"scripts": {
    "dev": "NODE_ENV=development webpack --progress --hide-modules --config=node_modules/laravel-mix/setup/webpack.config.js",
    "watch": "NODE_ENV=development webpack --watch --progress --hide-modules --config=node_modules/laravel-mix/setup/webpack.config.js",
    "hot": "NODE_ENV=development webpack-dev-server --inline --hot --config=node_modules/laravel-mix/setup/webpack.config.js",
    "production": "NODE_ENV=production webpack --progress --hide-modules --config=node_modules/laravel-mix/setup/webpack.config.js"
}

Step 2: Install and Configure Tailwind CSS

Now we install Tailwind

npm install tailwindcss
npx tailwind init

Next, we’ll edit our src/css/app.css file to add tailwind:

@import "tailwindcss/base";
@import "tailwindcss/components";
@import "tailwindcss/utilities";

Step 3: Configure Laravel Mix

We will now edit webpack.mix.js to the following:

let mix = require('laravel-mix');
mix.postCss('src/css/app.css', 'build/css', [
    require('postcss-import'), // So we can use @import statements
    require('tailwindcss'),
    require('autoprefixer'), // to add prefixes like --webkit
]);

We’re using two other PostCSS plugins: postcss-import and autoprefixer. And our Tailwind CSS

Step 4: Compile our CSS

To compile our CSS, we have several options:

To build the CSS in “development” mode: npm run dev
To build in development mode and watch for changes: npm run watch
To build for production: npm run production

Building in development mode skips a couple things like minification. Makes the builds faster, and makes it easier to debug.

While developing, I just run npm run watch and code away!

Step 5: Minimizing File Size (Optional)

The main Tailwind CSS compiled file is relatively big.

Before using in production, we are advised to remove unused CSS.

Version 1.4+

We can do this by adding a custom configuration file. We place this at ./tailwind.config.js. It should look like this:

module.exports = {
    theme: {},
    // Specify the paths to all of the template files in your project
    purge: [
        '../templates/*.html',
        './templates/*.vue',
        './templates/*.jsx',
    ],
    variants: {}
}

When we build our CSS for production by running npm run production, all unused CSS classes will be purged giving us a much smaller final file size.

Version 1.3 and earlier

We can do this with a PostCSS plugin called Purgecss. There is a more detailed article about this in the Tailwind CSS documentation.

To add Purgecss to our Laravel Mix configuration is pretty straightforward:

First, we install Purgecss

npm install @fullhuman/postcss-purgecss --save-dev

Next we add modify our webpack.mix.js to look like this:

const purgecss = require('@fullhuman/postcss-purgecss')({
    // Specify the paths to all of the template files in your project
    content: [
        './templates/\*.html',
        './templates/\*.vue',
        './templates/\*.jsx',
    ],
    // Include any special characters you're using in this regular expression
    defaultExtractor: content => content.match(/[A-Za-z0-9-\_:/]+/g) || [] })

let mix = require('laravel-mix');
mix.postCss('src/css/app.css', 'build/css', [
    require('postcss-import'), // So we can use @import statements
    require('tailwindcss'),
    require('autoprefixer'), // to add prefixes like --webkit
    ...process.env.NODE\_ENV === 'production'? [purgecss] : [] // Purge only in production
]);

Conclusion

Tailwind CSS can seem pretty tedious to set up, but using Laravel Mix, it can become a pretty straightforward process. Following the above steps should get you up and running in minutes.

Let me know any questions, or extra tips you may have in the comments section.

The post Tailwind CSS: Getting Set Up in Minutes appeared first on Stephen AfamO's Blog.

Loops in Go

Stephen Afam-Osemene — Mon, 25 Mar 2019 08:04:08 +0000

There is only one keyword to perform loops in Go: for. The implementation is very flexible. In this article, we’ll consider the various ways to use it. If you’re interested in the details, you should definitely go through the official specification.

For loops in Go

First, let’s see how to replicate the common C-style for loop. In Go, it works similar to other languages. We supply 3 statements after the keyword. The first is an initial statement to run, the second is the condition that is evaluated before each iteration and the third is run at the end of an iteration.

package main

import "fmt"

func main() {
    for i := 1; i <= 10; i++ {
        fmt.Println(i)
    }
}

While Loops in Go

In most computer programming languages, a while loop is a control flow statement that allows code to be executed repeatedly based on a given Boolean condition. The while loop can be thought of as a repeating if statement.

In Go, we use the for keyword to write a while loop by providing only a single statement. The loop will continue to run as long as the statement is true.

package main

import "fmt"

func main() {
    var a = 1

    for a <= 5 {
        fmt.Printf("a is: %v \n", a)
        a++
    }
}

a is: 1 
a is: 2 
a is: 3 
a is: 4 
a is: 5

The Range Clause

The range clause allows us to iterate over an array, slice, map, string, or channel.

Looping over an array using range

When we use range to loop over an array, we will receive 2 values for each iteration. The index of each element and a copy of the value. Receiving the 2nd value is optional.

package main

import "fmt"

func main() {
    var myArray = [...]int{1,2,3,4}

    for k, v := range myArray {
        fmt.Printf("index=%v, value=%v \n", k, v)
    }
}

index=0, value=1 
index=1, value=2 
index=2, value=3 
index=3, value=4

Looping over a slice using range

Just like with an array, when we use range to loop over a slice, we will receive 2 values for each iteration. The index of each element and a copy of the value. Receiving the 2nd value is optional.

package main

import "fmt"

func main() {
    var mySlice = []int{1,2,3,4}

    for k, v := range mySlice {
        fmt.Printf("index=%v, value=%v \n", k, v)
    }
}

index=0, value=1 
index=1, value=2 
index=2, value=3 
index=3, value=4

Looping over a map using range

As you may expect, when we loop over a map using range, we also receive 2 values. This time the values are the key of the element in the map and a copy of the value. Receiving the 2nd value is optional.

package main

import "fmt"

func main() {
    var myMap = map[string]string{
        "foo": "bar",
        "boo": "baz",
    }

    for k, v := range myMap {
        fmt.Printf("key=%v, value=%v \n", k, v)
    }
}

key=foo, value=bar 
key=boo, value=baz

Looping over a string using range

When we loop over a string using range, we also receive 2 values in each iteration. The first value is the byte position where the character begins, the second value is a rune which represents a single Unicode character. As usual, receiving the 2nd value is optional.

package main

import "fmt"

func main() {
    var myString = "Stephen"

    for index, runeValue := range myString {
        fmt.Printf("%#U starts at byte position %d\n", runeValue, index)
    }
}

U+0053 'S' starts at byte position 0
U+0074 't' starts at byte position 1
U+0065 'e' starts at byte position 2
U+0070 'p' starts at byte position 3
U+0068 'h' starts at byte position 4
U+0065 'e' starts at byte position 5
U+006E 'n' starts at byte position 6

Looping over a channel using range

When we loop over a channel using range, we receive a single value, which is the next element in the channel.

package main

import "fmt"

func main() {
    queue := make(chan string, 2)
    queue <- "one"
    queue <- "two"
    close(queue)

    for elem := range queue {
        fmt.Println(elem)
    }
}

one
two

Conclusion

We’ve seen the various loop mechanics in Go. Now Go forth and prosper. .

The post Loops in Go appeared first on Stephen AfamO's Blog.

Docker Volumes: An Introduction

Stephen Afam-Osemene — Sun, 04 Feb 2018 19:34:45 +0000

Docker volumes are very useful when we need to persist data in Docker containers or share data between containers.

Docker volumes are important because when a Docker container is destroyed, it’s entire file system is destroyed too. So if we want to keep this data in some way, it is necessary that we use Docker volumes.

Docker volumes are attached to containers during a docker run command by using the -v flag

When Docker Volumes are needed

Let’s look at the docker image we created in a previous article. The stephenafamo/adonisjs image.

The working directory is /var/www, and the application code is there. If we want to run an Adonis application in a container from this image and we don’t want all our files to be deleted once we stop the container, then we have several choices.

1. Bind mounts

A bind mount is just mapping a directory on the host machine to a directory in the container. However, when the container is removed, it does not affect the directory.

If the -v or --volume flag’s value is a path, then it is assumed to be a bind mount. If the directory does not exist, then it will be created. When we do this, the directory /path/to/app/directory will contain the same files as /var/www in the container.

docker run -d --name adonis \
    -v /path/to/app/directory:/var/www \
    stephenafamo/adonisjs:1.0.0

NOTE : If you are confused about how to use the docker run command or how to run containers, then you can go through the the previous article.

2. Docker volumes

Volumes are the preferred mechanism for persisting data generated by and used by Docker containers. While bind mounts are dependent on the directory structure of the host machine, volumes are completely managed by Docker. Volumes have several advantages over bind mounts:

Volumes are easier to back up or migrate than bind mounts.
You can manage volumes using Docker CLI commands or the Docker API.
Volumes work on both Linux and Windows containers.
Volumes can be more safely shared among multiple containers.
Volume drivers allow you to store volumes on remote hosts or cloud providers, to encrypt the contents of volumes, or to add other functionality.
A new volume’s contents can be pre-populated by a container.

In addition, volumes are often a better choice than persisting data in a container’s writable layer, because using a volume does not increase the size of containers using it, and the volume’s contents exist outside the lifecycle of a given container.

Creating and Removing a Docker volume

We can create a Docker volume using the command

docker volume create volume_name

To list all the volumes, we use the following command.

docker volume ls

docker volume ls

If we ever choose to delete a Docker volume, we do it using the command.

docker volume rm volume_name

Attaching a Docker volume to a container

We can also use the -v or --volume flag to attach a Docker volume to a container. However, instead of putting the path to the directory on the host machine as we did with bind mounts, we simply put the volume name.

docker run -d --name adonis \
    -v adonis_vol:/var/www \
    stephenafamo/adonisjs:1.0.0

NOTE : If we specify a volume that does not exist, it will be created

Mounting new Docker volumes or empty directories

If we start a container that creates a new volume and attach it to a destination in the container that contains files, then the files in the destination are copied into the volume.

For example, in our stephenafamo/adonisjs image, a new application will be created in the /var/www directory if an empty volume is mounted there.

Using a mount in read-only mode

Since a Docker volume (or bind mount) can be attached to multiple containers, it may be useful for only some of the containers to be able to write to it.

To achieve this, we will need to attach the volume to the other containers in read-only mode. This is done by adding the ro option like this;

docker run -d --name adonis \
    -v adonis_vol:/var/www:ro \
    stephenafamo/adonisjs:1.0.0

This works for both Docker volumes and bind mounts.

Conclusion

So, this covers the common uses of Docker volumes. There are also several volume drivers that allow for some more functionality. Perhaps we’ll talk about that in another article, or maybe I’ll update this later.

As always, I appreciate comments, suggestions and corrections. Thank you.

The post Docker Volumes: An Introduction appeared first on Stephen AfamO's Blog.

DEV Community: Stephen Afam-Osemene

How to use a terminal

What is a terminal?

Running a command

The anatomy of a command

How do I know which commands are available?

How do I know the arguments and flags to pass to a command?

What to do next?

How to schedule a task at a specific time in Go

The one-liner

Exiting early

Handling PostgreSQL Errors

1. PostgreSQL Errors have codes

2. PostgreSQL Errors can contain even more information

Some practical use cases

The best way to style broken images

What’s happening here?

Conclusion

Compounding Competence

From Blackfriday to Goldmark

First time working with Markdown in Go

Switching to Goldmark

Extending Goldmark

Rewriting Images

Getting all images

Using Goldmark for PDF rendering

Authenticating 3rd Party Integrations in Go

Why is this necessary?

Version 1: Using Forms

Describing our Form

About types.Option

Modeling the authentication

Problems with v1

Version 2: Supporting More Authentication Types

Modeling the authentication

Problems with v2

Version 3: A recursive flow

Requirements

Inputs

Output

The Flow

Implementations

No Authentication

Single Form

Oauth1

OAuth2

WordPress: A more complex flow

Conclusion

Gracefully shutting down multiple workers in Go

What are workers?

Gracefully Shutting Down Workers

Multiple Workers

Coordinating with Context

Using Orchestra

Conclusion

Better scheduling in Go

Using time.After()

Using time.Ticker

Dangers of using time.Tick()

Limitations using time.Tick()

Extending time.Tick() using a custom function

What's happening in the function

Examples of using the custom function

Conclusion

Implementing an Event Driven System in Go

Defining Events

Listening for Events

Triggering Events

Conclusion

Tailwind CSS: Getting Set Up in Minutes

Prerequisites

Assumptions

Step 1: Install Laravel Mix

Step 2: Install and Configure Tailwind CSS

Step 3: Configure Laravel Mix

Step 4: Compile our CSS

Step 5: Minimizing File Size (Optional)

Version 1.4+

Version 1.3 and earlier

Conclusion

Using `time.After()`

Using `time.Ticker`

Dangers of using `time.Tick()`

Limitations using `time.Tick()`

Extending `time.Tick()` using a custom function