DEV Community: Fikayo Adepoju

File Uploads with HTMX and Golang

Fikayo Adepoju — Mon, 19 Aug 2024 22:15:34 +0000

Ofcourse you have already heard about the awesomeness of HTMX (you haven’t? well, good thing you’re here 😃)

Today, we will be combining the simplicity of HTMX with the power of Golang to upload files to our server. Yeah, we are going to be building another exciting web feature with HTMX and Go.

By the way, if you really want a practical project-based guide on building fullstack apps with HTMX, check out my HTMX + Go: Build Fullstack Applications with Golang and HTMX course [Discount Included].

So, let’s begin.

Setting Up the Go Project

The first step is to setup a simple Go project. We can do that by creating a folder, going into it and initialising it as a Go project using the commands below:

mkdir go-htmx-file-uploads
cd go-htmx-file-uploads
go mod init go-htmx-file-uploads

Once we have the project initialized, let’s now install some dependencies we will be requiring within our project.

This will be a simple server that will contain a single page with our upload form and also an endpoint that will be used to upload the file.

For routing, we will be using the Gorilla Mux routing library, but feel free to use any routing solution of your choice. We will also be using Google’s UUID library for Go to generate random names for our files when we upload them. This is a personal preference as you can generate file names in different ways.

Install these two with the commands below:

Gorillla Mux

go get -u github.com/gorilla/mux

Google UUID

go get github.com/google/uuid

With these two installed, our project is fully set up, and we can move to the next step.

Creating our Templates

We will be creating two HTML templates for this little project.

The first template will be an HTML fragment that simply takes a slice of string messages that we can send from the server to the client.

This fragment will take this slice of messages and loop through it to create an HTML list to be returned to the client (remember how HTMX works with Hypermedia APIs, pretty cool huh 😎).

So, let’s create that first.

At the root of the Go project, first create a templates folder inside which we will be storing all our templates.

Next, create a file messages.html inside the templates folder and add the following code to it:

{{define "messages"}}
<ul>
    {{range .}}
        <li>{{ . }}</li>
    {{end}}
</ul>
{{end}}

This defines a messages template and loops through the incoming slice of string messages to form an HTML list.

For our next template, we will be creating the file upload page itself.

Inside the templates folder, create a new file upload.html and paste in the code below:

{{define "upload"}}
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <link href="https://cdn.jsdelivr.net/npm/bootstrap@5.1.3/dist/css/bootstrap.min.css" rel="stylesheet">
    <script src="https://unpkg.com/htmx.org@1.9.12"></script>
    <title>Upload File</title>
</head>
<body>

    <div class="row">
        <div class="col-md-6 p-5 mt-5">
            <h4>Upload File</h4>
            <form  class="form">
                <div id="messages"></div>
                <div class="mb-3">
                    <label for="avatarInput" class="form-label">Select Image</label>
                    <input type="file" class="form-control" id="avatarInput" name="avatar" required>
                </div>
                <button 
                    hx-post="/upload" 
                    hx-encoding="multipart/form-data" 
                    hx-target="#messages" 
                    type="submit" class="btn btn-primary">Upload</button>
            </form>
        </div>
    </div>



</body>
</html>
{{end}}

Perfect!

Now let’s go through the code in this file.

First, we have defined the template with the name upload, this is the name we will use to reference it later in our route handlers.

We then have some boilerplate HTML code in the head section, but I have included two important libraries here (well, just one is really important, the other is just for CSS vibes).

The HTMX library has been included with the <script> tag to bring in HTMX, just the library, no dependencies required.

Then I have also brought in the Bootstrap CSS library, this is just to give our page elements some nice styling. It is not compulsory for this demo.

In the page itself, we have a form that does the upload. Let’s break down what is within the <form> tags.

First we have a <div> with an id of messages , this is the container where we will be loading all our server messages that come in as HTML. Remember the messages template, yeah, this is where the list of messages will be going into.

After that, we have the form input element that is set to file to ensure that it displays a file upload widget. We have given it the name avatar to reference it at the backend, but you can give this any name you want. I gave it avatar because i was using it to upload profile images.

Finally, we have the button which has been supercharged with HTMX. I have displayed it again below so we can go through it

<button 
                    hx-post="/upload" 
                    hx-encoding="multipart/form-data" 
                    hx-target="#messages" 
                    type="submit" class="btn btn-primary">Upload</button>

First, I have added hx-post="/upload" , this tells it to submit the form to the /upload endpoint that we will be creating very soon and will handle the file upload.

Next is hx-encoding="multipart/form-data", this is compulsory for uploading files with HTMX to let the server know you’re sending along a file with the request.

Then we have hx-target="#messages" which tells the button to insert any response from the server into the <div> with an id of messages.

These three define our configuration for uploading the file to our backend.

Below is a preview of what our page looks like:

Processing the File Upload

Now that we have our templates, it’s time to write the code that will display our upload page and also handle our file uploads.

To begin, at the root of the Go project, create a uploads folder. This is the folder where all our uploaded files will be stored.

With that in place, let’s write our main file.

Create the file main.go at the root of your project and add the following code:

package main

import (
    "html/template"
    "log"
    "net/http"
    "io"
    "os"
    "path/filepath"
    "github.com/google/uuid"
    "github.com/gorilla/mux"
)

var tmpl *template.Template

func init(){
    tmpl, _ = template.ParseGlob("templates/*.html")
}

func main() {


    router := mux.NewRouter()

    router.HandleFunc("/", homeHandler).Methods("GET")

    router.HandleFunc("/upload", UploadHandler).Methods("POST")

    log.Println("Server starting on :8080")
    log.Fatal(http.ListenAndServe(":8080", router))
}

func homeHandler(w http.ResponseWriter, r *http.Request) {

    tmpl.ExecuteTemplate(w, "upload", nil)

}

func UploadHandler(w http.ResponseWriter, r *http.Request) {


        // Initialize error messages slice
        var serverMessages []string

        // Parse the multipart form, 10 MB max upload size
        r.ParseMultipartForm(10 << 20)

        // Retrieve the file from form data
        file, handler, err := r.FormFile("avatar")
        if err != nil {
            if err == http.ErrMissingFile {
                serverMessages = append(serverMessages, "No file submitted")
            } else {
                serverMessages = append(serverMessages, "Error retrieving the file")
            }

            if len(serverMessages) > 0 {
                tmpl.ExecuteTemplate(w, "messages", serverMessages)
                return
            }

        }
        defer file.Close()

        // Generate a unique filename to prevent overwriting and conflicts
        uuid, err := uuid.NewRandom()
        if err != nil {
            serverMessages = append(serverMessages, "Error generating unique identifier")
            tmpl.ExecuteTemplate(w, "messages", serverMessages)

            return
        }
        filename := uuid.String() + filepath.Ext(handler.Filename) // Append the file extension

        // Create the full path for saving the file
        filePath := filepath.Join("uploads", filename)

        // Save the file to the server
        dst, err := os.Create(filePath)
        if err != nil {
            serverMessages = append(serverMessages, "Error saving the file")
            tmpl.ExecuteTemplate(w, "messages", serverMessages)

            return
        }
        defer dst.Close()
        if _, err = io.Copy(dst, file); err != nil {
            serverMessages = append(serverMessages, "Error saving the file")
            tmpl.ExecuteTemplate(w, "messages", serverMessages)
            return
        }


        serverMessages = append(serverMessages, "File Successfully Saved")
        tmpl.ExecuteTemplate(w, "messages", serverMessages)


}

Yope, that’s a bunch of code. Don’t worry, we’ll go through it all step by step to figure out what this is all doing.

First we define our package main and import a bunch of libraries we will be making use of. These imports include the Gorilla mux router and the Google UUID library that we installed earlier.

After that, I create a global tmpl variable to hold all the HTML templates in the project and in the init() function, the templates are all loaded from the templates folder.

The main() Function

Now to the main() function. Here, we have initlialized the Gorilla Mux router and set up two routes.

The GET / base route which will be handled by a homeHandler function and displays our upload form, and the POST /upload route that will be handled by UploadHandler and handles the upload itself.

Finally, we print out a message to indicate that our server is running, and run the server on port 8080.

The Handler Functions

First we have homeHandler . This is the function that handles our base route, and it simply calls ExecuteTemplate on the tmpl variable with the name we gave to our template

tmpl.ExecuteTemplate(w, "upload", nil)

This call is enough to simply render our upload page to the screen when we visit the base route.

After that is the UploadHandler function. This is where the real magic happens, so let’s walk through the function.

First, we create a slice of strings called serverMessages to hold any message we want to send back to the client.

After that, we call ParseMultipartForm on the request pointer to limit the size of uploaded files to within 20MB.

r.ParseMultipartForm(10 << 20)

Next, we get a hold on our file by referencing the name of the file field with FormFile on the request pointer.

With our reference to the file, we check if there is actually a file, and if not, we return a message saying that no file was submitted or an error was encountered when trying to retrieve the file to account for other errors.

file, handler, err := r.FormFile("avatar")
        if err != nil {
            if err == http.ErrMissingFile {
                serverMessages = append(serverMessages, "No file submitted")
            } else {
                serverMessages = append(serverMessages, "Error retrieving the file")
            }

            if len(serverMessages) > 0 {
                tmpl.ExecuteTemplate(w, "messages", serverMessages)
                return
            }

        }

At this point, if our messages slice is not empty, we return the messages to the client and exit the function.

If a file is found, we keep the file open and move to generating a new name for it with the UUID library and also handle the errors in that process accordingly.

We build a new file name with the generated string and the file extension and set it’s path to the uploads folder.

    uuid, err := uuid.NewRandom()
        if err != nil {
            serverMessages = append(serverMessages, "Error generating unique identifier")
            tmpl.ExecuteTemplate(w, "messages", serverMessages)

            return
        }
        filename := uuid.String() + filepath.Ext(handler.Filename) 

        // Create the full path for saving the file
        filePath := filepath.Join("uploads", filename)

Once the new file path is constructed, we then use the os library to create the file path

After that, we use the io library to move the file from it’s temporary location to the new location and also handle errors accordingly.

    dst, err := os.Create(filePath)
        if err != nil {
            serverMessages = append(serverMessages, "Error saving the file")
            tmpl.ExecuteTemplate(w, "messages", serverMessages)

            return
        }
        defer dst.Close()
        if _, err = io.Copy(dst, file); err != nil {
            serverMessages = append(serverMessages, "Error saving the file")
            tmpl.ExecuteTemplate(w, "messages", serverMessages)
            return
        }

If we get no errors from the file saving process, we then return a successful message to the client using our messages template as we have done with previous messages.

serverMessages = append(serverMessages, "File Successfully Saved")
tmpl.ExecuteTemplate(w, "messages", serverMessages)

And that’s everything.

Now let’s take this code for a spin.

Testing the File Upload

Save the file and head over to the command line.

At the root of the project, use the command below to run our little file upload application:

go run main.go

Now go to your browser and head over to http://localhost:8080, you should see the upload screen displayed.

Try testing with no file to see the error message displayed. Then test with an actual file and also see that you get a successful message.

Check the uploads folder to confirm that the file is actually being saved there.

And Wholla! You can now upload files to your Go servers using HTMX.

Conclusion

If you have enjoyed this article, and will like to learn more about building projects with HTMX, I’ll like you to check out HTMX + Go: Build Fullstack Applications with Golang and HTMX, and The Complete HTMX Course: Zero to Pro with HTMX to further expand your knowledge on building hypermedia-driven applications with HTMX.

HTMX + Go : Build a CRUD App with Golang and HTMX

Fikayo Adepoju — Mon, 15 Jul 2024 19:49:54 +0000

One question I see going around alot about HTMX, especially amongst developers that have just tried the library is “But what can you really build with it though?”

Great question, and in this article, we will start with baby steps by building a database-backed CRUD application with HTMX and Go as our backend language.

By the way, if you really want a practical project-based guide on building fullstack apps with HTMX, check out my **HTMX + Go: Build Fullstack Applications with Golang and HTMX [Discount included] course.**

Let’s begin.

What exactly are we building?

I’ll like to call it a Task Management Application but I know you already figured that that’s just a fancy name for another Todo application. Don’t worry, Todo apps are great for learning fundamental operations with languages, libraries and frameworks so we will be using that same tested and trusted strategy.

Our application will be able to do the following:

Display tasks
Add new tasks
Update an existing task and …
Delete a task

Database Setup

So first, we need a database, and for this demo project, I will be using MySQL. Feel free to use any database of your choice and make the necessary code changes to reference your database as you follow along with this article.

We will keep things simple, no complicated schema design. First we create a database with the name testdb and inside this database, we create a todos table (feel free to give your database and table any name you prefer but ensure you use the same names in your SQL statements)

Inside the todos table, implement the schema below:

id: PK, Auto incrementing
task : VARCHAR(200) - Contains the task item
done: INT(1), default = 0 (Boolean field)

You can choose to seed the database table with some tasks so that we can start seeing some tasks the first time we load the application.

Creating the Hypermedia API

To begin setting up our little application, create a folder for the project at any convenient location in your development computer.

mkdir task-management

Run the following command at the root of the project folder to initialize it as a Golang project:

go mod init task-management

Next, we need to install some dependencies. We already know we are using MySQL as our database, thus, we need to install the MySQL driver for Golang.

We also need to install the Gorilla Mux Router which will be the routing library for our project. Run the two commands below at the root of your project to get these libraries installed into your project

MySQL:

go get -u github.com/go-sql-driver/mysql

Gorilla Mux:

go get -u github.com/gorilla/mux

With these libraries in place, create your main.go file at the root of the project and add the code below:

package main

import (
    "database/sql"
    "fmt"
    "html/template"
    "log"
    "net/http"
    "strconv"
    "strings"

    _ "github.com/go-sql-driver/mysql"
    "github.com/gorilla/mux"
)

var tmpl *template.Template
var db *sql.DB

type Task struct {
    Id   int
    Task string
    Done bool
}

func init() {
    tmpl, _ = template.ParseGlob("templates/*.html")

}

func initDB() {
    var err error
    // Initialize the db variable
    db, err = sql.Open("mysql", "root:root@(127.0.0.1:3333)/testdb?parseTime=true")
    if err != nil {
        log.Fatal(err)
    }

    // Check the database connection
    if err = db.Ping(); err != nil {
        log.Fatal(err)
    }
}

func main() {

    gRouter := mux.NewRouter()

    //Setup MySQL
    initDB()
    defer db.Close()

    gRouter.HandleFunc("/", Homepage)

    //Get Tasks
    gRouter.HandleFunc("/tasks", fetchTasks).Methods("GET")

    //Fetch Add Task Form
    gRouter.HandleFunc("/newtaskform", getTaskForm)

    //Add Task
    gRouter.HandleFunc("/tasks", addTask).Methods("POST")

    //Fetch Update Form
    gRouter.HandleFunc("/gettaskupdateform/{id}", getTaskUpdateForm).Methods("GET")

    //Update Task
    gRouter.HandleFunc("/tasks/{id}", updateTask).Methods("PUT", "POST")

    //Delete Task
    gRouter.HandleFunc("/tasks/{id}", deleteTask).Methods("DELETE")

    http.ListenAndServe(":4000", gRouter)

}

func Homepage(w http.ResponseWriter, r *http.Request) {

    tmpl.ExecuteTemplate(w, "home.html", nil)

}

func fetchTasks(w http.ResponseWriter, r *http.Request) {
    todos, _ := getTasks(db)
    //fmt.Println(todos)

    //If you used "define" to define the template, use the name you gave it here, not the filename
    tmpl.ExecuteTemplate(w, "todoList", todos)
}

func getTaskForm(w http.ResponseWriter, r *http.Request) {

    tmpl.ExecuteTemplate(w, "addTaskForm", nil)
}

func addTask(w http.ResponseWriter, r *http.Request) {

    task := r.FormValue("task")

    fmt.Println(task)

    query := "INSERT INTO tasks (task, done) VALUES (?, ?)"

    stmt, err := db.Prepare(query)

    if err != nil {
        log.Fatal(err)
    }
    defer stmt.Close()

    _, executeErr := stmt.Exec(task, 0)

    if executeErr != nil {
        log.Fatal(executeErr)
    }

    // Return a new list of Todos
    todos, _ := getTasks(db)

    //You can also just send back the single task and append it
    //I like returning the whole list just to get everything fresh, but this might not be the best strategy
    tmpl.ExecuteTemplate(w, "todoList", todos)

}

func getTaskUpdateForm(w http.ResponseWriter, r *http.Request) {

    vars := mux.Vars(r)

    //Convert string id from URL to integer
    taskId, _ := strconv.Atoi(vars["id"])

    task, err := getTaskByID(db, taskId)

    if err != nil {
        http.Error(w, err.Error(), http.StatusInternalServerError)
    }

    tmpl.ExecuteTemplate(w, "updateTaskForm", task)

}

func updateTask(w http.ResponseWriter, r *http.Request) {

    vars := mux.Vars(r)

    taskItem := r.FormValue("task")
    //taskStatus, _ := strconv.ParseBool(r.FormValue("done"))
    var taskStatus bool

    fmt.Println(r.FormValue("done"))

    //Check the string value of the checkbox
    switch strings.ToLower(r.FormValue("done")) {
    case "yes", "on":
        taskStatus = true
    case "no", "off":
        taskStatus = false
    default:
        taskStatus = false
    }

    taskId, _ := strconv.Atoi(vars["id"])

    task := Task{
        taskId, taskItem, taskStatus,
    }

    updateErr := updateTaskById(db, task)

    if updateErr != nil {
        log.Fatal(updateErr)
    }

    //Refresh all Tasks
    todos, _ := getTasks(db)

    tmpl.ExecuteTemplate(w, "todoList", todos)
}

func deleteTask(w http.ResponseWriter, r *http.Request) {

    vars := mux.Vars(r)

    taskId, _ := strconv.Atoi(vars["id"])

    err := deleTaskWithID(db, taskId)

    if err != nil {
        http.Error(w, err.Error(), http.StatusInternalServerError)
    }

    //Return list
    todos, _ := getTasks(db)

    tmpl.ExecuteTemplate(w, "todoList", todos)
}

func getTasks(dbPointer *sql.DB) ([]Task, error) {

    query := "SELECT id, task, done FROM tasks"

    rows, err := dbPointer.Query(query)

    if err != nil {
        return nil, err
    }

    defer rows.Close()
    var tasks []Task

    for rows.Next() {
        var todo Task

        rowErr := rows.Scan(&todo.Id, &todo.Task, &todo.Done)

        if rowErr != nil {
            return nil, err
        }

        tasks = append(tasks, todo)
    }

    if err = rows.Err(); err != nil {
        return nil, err
    }

    return tasks, nil

}

func getTaskByID(dbPointer *sql.DB, id int) (*Task, error) {

    query := "SELECT id, task, done FROM tasks WHERE id = ?"

    var task Task

    row := dbPointer.QueryRow(query, id)
    err := row.Scan(&task.Id, &task.Task, &task.Done)

    if err != nil {
        if err == sql.ErrNoRows {
            return nil, fmt.Errorf("No task was found with task %d", id)
        }
        return nil, err
    }

    return &task, nil

}

func updateTaskById(dbPointer *sql.DB, task Task) error {

    query := "UPDATE tasks SET task = ?, done = ? WHERE id = ?"

    result, err := dbPointer.Exec(query, task.Task, task.Done, task.Id)

    if err != nil {
        return err
    }

    rowsAffected, err := result.RowsAffected()

    if err != nil {
        return err
    }

    if rowsAffected == 0 {
        fmt.Println("No rows updated")
    } else {
        fmt.Printf("%d row(s) updated\n", rowsAffected)
    }

    return nil

}

func deleTaskWithID(dbPointer *sql.DB, id int) error {

    query := "DELETE FROM tasks WHERE id = ?"

    stmt, err := dbPointer.Prepare(query)

    if err != nil {
        return err
    }
    defer stmt.Close()

    result, err := stmt.Exec(id)
    if err != nil {
        return err
    }

    rowsAffected, err := result.RowsAffected()
    if err != nil {
        return err
    }
    if rowsAffected == 0 {
        return fmt.Errorf("no task found with id %d", id)
    }

    fmt.Printf("Deleted %d task(s)\n", rowsAffected)
    return nil

}

Yeah, that was alot of code. Don’t worry, we will take it from the very top and walk down

So first we import all our necessary packages. The MySQL driver and Gorilla Mux router we installed, and a bunch of packages from the Go standard library that will be useful in our code operations.

import (
    "database/sql"
    "fmt"
    "html/template"
    "log"
    "net/http"
    "strconv"
    "strings"
    _ "github.com/go-sql-driver/mysql"
    "github.com/gorilla/mux"
)

Next, we create a tmpl variable that will be used to hold our loaded templates and a db variable that will be a pointer to our database connection for running database tasks. We then create a custom Task struct that defines a task type.

Inside the init() function, we load all our templates from a templates folder. All our templates are expected to have the .html extension as since HTMX expects us to return HTML, this makes a ton of sense.

Go ahead and create the templates folder at the root of the project so that we can begin loading all our templates from there.

We also have an initDB() function that takes care of setting up our connection to the database and returns a pointer reference to our database. Ensure to change the connection string to match that of your database (credentials, host, port, database name etc)

Inside the main function, we initialize our router and call our initDB() database function to initialize our database. This is then followed by all our routes and route handlers and finally, we listen on port 4000 which is where we will be serving the application.

Routes and Handlers

Now let’s begin breaking down our routes and their respective handlers.

The GET / Base Route: This is our base route and loads the home page of the application. The handler, Hompage returns the home.html file to the client.
The GET /tasks Route: This route uses the fetchTasks handler to get all our tasks from our database and return them in an HTML list to the client using a todoList template.
The GET /newtaskform Route: This route will load a new task form from the server each time a user wants to create a new task or clicks a Add New Task button. It uses a addTaskForm template to display a new HTML form for adding a new task
The POST /tasks Route: This route calls the addTask handler to add a new task to the database and return an updated list of all tasks.
The GET /gettaskupdateform/{id} Route: Uses the Id of a task to load the task into an update form with the updateTaskForm template and returns this form to the client when the user clicks the Edit button.
The PUT/POST /tasks/{id} Route: Takes the Id of a task to be updated and updates it using the updateTask handler. After the update operation, the most recent version of the list is returned as HTML.
The DELETE /tasks/{id} Route: Uses the deleteTask handler and a task Id to delete a specific task. Once the task is deleted, an updated list of tasks is returned back to the client.

And that’s all the routes and handlers used in this application.

You may have noticed some other functions asides the route handlers also defined in our main.go file. These are functions for performing database operations for fetching tasks (getTasks), getting a single task using its Id (getTaskByID), updating a task using its Id (updateTaskById), and deleting a task using the tasks’ Id (deleTaskWithID).

These helper functions are used within our route handlers to facilitate database operations and keep the handlers lean.

Creating Our Templates

Now that we are familiar with our Hypermedia API, let’s begin creating the HTML templates that will be retuned in the response to our API calls.

First, we create home.html file in the templates folder. This will load the home page of our task management application. Add the following code to the file after creating it.

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <link href="https://cdn.jsdelivr.net/npm/bootstrap@5.3.3/dist/css/bootstrap.min.css" rel="stylesheet" integrity="sha384-QWTKZyjpPEjISv5WaRU9OFeRpok6YctnYmDr5pNlyT2bRjXh0JMhjY6hW+ALEwIH" crossorigin="anonymous">
    <script src="https://unpkg.com/htmx.org@1.9.12"></script>
    <title>To Do App</title>
</head>
<body>

    <div class="row">

        <div class="col">
            <h2>Tasks</h2>

            <div>
                <a href="#" hx-get="/newtaskform" hx-target="#addTaskForm">Add New Item</a>
            </div>

            <div id="taskList" hx-get="/tasks" hx-trigger="load" hx-swap="innerHTML">

            </div>

        </div>
        <!-- <div class="col">

        </div> -->
        <div class="col">
            <h2>Add New Task</h2>

            <div id="addTaskForm">
                {{template "addTaskForm"}}
            </div>

        </div>

    </div>

</body>
</html>

This templates forms the shell and layout of the entire application. We have the boilerplate HTML structure and I have also added the Bootstrap CSS library for some basic styling. The HTMX library has also been included through a CDN link.

The application layout contains two sections. One section for displaying tasks and the other for showing the new task and task update forms.

The first section contains a button for requesting a new task form from the hypermedia API. Once the form is returned, we then use hx-target to load the form into the div with an id of addTaskForm in the forms section of the page.

<a href="#" hx-get="/newtaskform" hx-target="#addTaskForm">Add New Item</a>

The next component in the first section is the div where our tasks will be loaded into. This div uses hx-trigger to initiate a GET request to the /tasks route once the page loads, thus immediately loading the tasks into the page.

<div id="taskList" hx-get="/tasks" hx-trigger="load" hx-swap="innerHTML">

</div>

In the second section, as mentioned earlier, we have a div with an id of addTaskForm for loading both our new task and update forms. We have also preloaded the form for adding a new task into this div using Go template import syntax so as to have a default form in place.

Now let’s create the form for adding a new task next. Inside the templates folder, create the file addTaskForm.html and add the following code inside it:

{{define "addTaskForm"}}
<form>
    <div>
        <input type="text" class="form-control" name="task">
    </div>
    <div class="mt-2">
        <button class="btn btn-primary" hx-post="/tasks" hx-target="#taskList" hx-target="innerHTML">
            Save
        </button>
    </div>

</form>
{{end}}

This templates loads a fresh form in the UI for adding a new task. When the submit button is clicked, it uses HTMX to send a POST request to the /tasks route to add a new task. When the operation is done, it uses HTMX once again to load the response, an updated list of tasks, into the div with an id of taskList.

Next is our update form template. Inside the templates folder, create the file updateTaskForm.html and add the following code:

{{define "updateTaskForm"}}
<form>
    <div>
        <input type="text" class="form-control" name="task" value="{{.Task}}">
    </div>
    <div>
        <input type="checkbox" name="done" {{if .Done}} checked {{end}} id="">
    </div>
    <div class="mt-2">
        <button class="btn btn-primary" hx-put="/tasks/{{.Id}}" hx-target="#taskList" hx-target="innerHTML">
            Update Task
        </button>
    </div>

</form>
{{end}}

This template takes in a task to be updated and uses it to pre-populate the update form so that the user can see the previous state of the task to be updated.

When the Update Task button is clicked, it will send the updated values to the hypermedia API for the task to be updated. Once updated, it loads the updated list into the page.

Finally, we create the template the returns our list of task items. Inside the templates folder, create the file todoList.html and add the following code:

{{define "todoList"}}
<ul>
    {{range .}}
        <li>
            <span {{if .Done}} style="text-decoration:line-through" {{end}}>{{.Task}}</span>
            [<a href="#" hx-get="/gettaskupdateform/{{.Id}}" hx-target="#addTaskForm" hx-swap="innerHTML">Edit</a>] | 
            <a href="#" hx-delete="/tasks/{{.Id}}" 
                        hx-confirm="Are you sure you want to Delete this Task?"
                        hx-target="#taskList">[Delete]</a>
        </li>
    {{end}}
</ul>

{{end}}

Yeah, a lot is going on in this template, so let’s break it down.

First, the template takes in a Go slice of Task types and loops over it using the range function to create an HTML list of unordered items.

The task it displayed in each list item and the Done property is used to check if the task is completed. If so, we use CSS to strike the task as being completed.

Just after the task text, we have an Edit button. This button calls the /gettaskupdateform endpoint to load an update form using the id of the specific task that was clicked. The user can then update the task and get an updated list of task items.

After the Edit button, we have a Delete button that uses hx-delete to call the DELETE /tasks/{id} endpoint so that we can delete the task. But before we can send the delete request, we use hx-confirm to display a confirmation dialog to the user so that they can confirm if they really want to delete this task item. Once deleted, a new updated list is returned and the task will be gone.

And with that we wrap up our application, so let’s move on to the fun part, checking it out.

Running the Application

With all the code in place, now let’s test our application.

Ensure that all files are saved and run the following command at the root of your project:

go run main.go

Now go to your browser and load the application page at http://localhost:4000. If you have used a different port, ensure that you’re using that port to load the app.

Now you should see your application as displayed below. See below as we add a new task, update an existing task and delete a task from our task list

Conclusion

Happy Coding :)

Sending Data in HTMX Requests (3 Ways)

Fikayo Adepoju — Mon, 08 Apr 2024 05:42:04 +0000

Sending Data in HTMX Requests

Introduction

TL;DR: We can all agree that HTMX is great right?

After several years of heaping up huge node_modules folders with hundreds of packages just to build a simple homepage in frameworks like Reactjs, Vuejs and Angular, we now have an alternative approach that brings back the simplicity of the old days where we just add one Javascript link at the top of our page and start building amazing interactivity in our Frontend applications.

Instead of Javascript-heavy frontends and virtualizing the DOM, HTMX takes a different approach by making “HTML Great Again” through the addition of enhanced hypermedia controls to native HTML elements.

One of the biggest examples is the ability to use any tag (<div>, <span>, <input />, etc) to trigger an HTTP request based on an event, using the standard HTTP methods: POST , GET , PUT, and DELETE

In this article, I’ll be showing you how you can send data along with your HTTP requests in HTMX and we will be looking at 3 ways you can do that.

By the way, if you’re still trying to get your hands solid on HTMX, you can checkout my course: The Complete HTMX Course: Zero To Pro with HTMX [Discount included]

Enough talk, let’s get started.

#1: Using a form Tag

The first, and probably easiest strategy, is to wrap the element making the request inside an HTML <form> tag. Inside this <form> tag, we can then add <input /> elements for each piece of data that we want to send.

Once the request is sent, all the data contained in the <input /> fields within the form will be sent along with the request

Example: Let’s say we want to process a login request by sending an email/password combo and the button below triggers the request with an hx-post

<button hx-post="http://localhost:1330/login">
    Submit
</button>

The first step is to place this button inside an HTML form like below:

<form>

   <button hx-post="http://localhost:1330/login">
       Submit
     </button>

</form>

We will then add <input> tags for the email and password we want to send along by specifying the name of the parameters with the name attribute of the <input> elements.

<form>

   <input type="email" name="userEmail" />

   <input type="password" name="userPassword" />

   <button hx-post="http://localhost:1330/login">
       Submit
     </button>

</form>

This way, HTMX will automatically recognize the data in the input fields as the data you want to send with the POST request and they will be sent accordingly.

Do note that this request is sent as Form Data, thus you need to make sure that your server accepts form data. For example, on an Express.js server, make sure that you have this middleware setup:

app.use(bodyParser.urlencoded({extended: true}))

#2: Using hx-vals (Sending JSON Data)

Our previous data sending strategy sends data to the server as Form data, this might not be appropriate in a number of cases. So, let’s look at how we can send data in JSON format using a JSON object.

Instead of wrapping our data items in a form tag this time we use a special HTMX tag, hx-vals , on the requesting element to load our JSON object as the payload of the request.

Example: Sending Login Details with a JSON object

So let’s say we want to send this JSON data along with our request:

{
  "userEmail": "htmxceo@x.com"
  "userPassword": "htmxrocks"
}

We will simply pass this object to the hx-vals attribute on our requesting element. However, it is important to note that because this JSON object uses double quotes for its keys and values, the hx-vals attribute should wrap the object in single quotes like below:

<button hx-post="http://localhost:1330/login"
        hx-vals='{
                          "userEmail": "htmxceo@x.com"
                          "userPassword": "htmxrocks"
                }'>
       Submit
</button>

Now when you click this button, the JSON payload will be sent along with the request.

Just like I noted in the previous section, ensure that your server can receive requests in JSON format in order to access this payload on the server-side.

On an Express.js server, you should have this middleware setup:

app.use(bodyParser.json());

Passing Variables in your Request

You might be asking, what if I want to send variables within my JSON object? As we all know, you can’t always hardcode values into your JSON payload, some of the values will come from user-submitted data or state variables.

So let’s assume we have these two variables that hold our email and password

const email = "htmxceo@x.com";
const password = "htmxrocks"

We can simply pass this into our JSON object by prefixing the value of hx-vals with js: and then replace our values inside the JSON object with variables. Below is our previous example refactored to use variables instead of hardcoded strings.

<script>
   const email = "htmxceo@x.com";
     const password = "htmxrocks";
</script>

...

<button hx-post="http://localhost:1330/login"
        hx-vals='js:{
                          "userEmail": email
                          "userPassword": password
                }'>
       Submit
</button>

This format also allows you to use Javascript functions or expressions as values of your JSON object keys.

#3: Using hx-include

This strategy is used for input fields that are not in anyway positionally close to or related to the requesting element (not collocated) but the requesting element needs to include their data in the HTTP request.

To better understand this, in our first strategy, the input fields were wrapped as siblings with requesting element, the HTML button, inside a form tag.

But what if the input field is at a totally different position away from the button in the DOM but you need to include the information in that input field in your request? that’s where hx-include comes in.

Example: We need to send the user id of a user in order to process their payment request. Let’s say we have the HTML structure below:

<div>
    <input type="text" name="userID" />

    <div>
        <h3>Product Name</h3>
        <img src="/imags/product-001" />
        <p>
            Product Description
        </p>
    </div>

</div>

<div>
  <button hx-post="http://localhost:1330/processpayment">
       Pay Now
     </button>
<div>

As you can see from the markup above, the button making the request and the input field for the userID data item are miles apart and are not even located in the same container element. However, we still want to be able to pick the data in this input field and send it in our request. So how do we achieve that?

The first thing we will need to do is to give the <input /> element that we want to target an id in order to access it:

<input type="text" name="userID" id="myUserID" />

We can then include it in our request by placing a hx-include attribute on the requesting element and use a CSS selector with the ID of the input field as the value like below:

<button hx-post="http://localhost:1330/processpayment"
        hx-include="#myUserID">
       Pay Now
</button>

This way, when the button is clicked to trigger the request, the data in the input field will be passed along in the request body.

How awesome is that 😃.

So, those are the 3 strategies for passing data in HTTP requests when using HTMX.

If you have found this article useful and you need a full comprehensive guide on what HTMX is and how to use it to build frontend applications, you can get my HTMX course here for 90% off.

You can also subscribe to my newsletter for more amazing HTMX and software engineering content.

Happy Coding ☺️

5 Books Every (Web) Developer MUST Read

Fikayo Adepoju — Fri, 31 Mar 2023 07:58:17 +0000

Yes, I have been there. Feeling the need to level up, dealing with Impostor syndrome, wondering how you can move from your current level to a better, more experienced, and marvelously skilled programmer.

Or you often feel you missed the class on concepts like memory management, cookies, sessions, sockets, proxies, compilers, Microservices, load-balancers, etc., these 5 books will help you fix all that.

Common questions I get asked a lot (and have often asked myself) are:

How do I go from Junior/Intermediate to Senior developer?
Am I an engineer or just a developer?
How do those senior guys get it all the time?
I didn’t study computer science. Do I need to learn computer fundamentals to be a better developer?
How do I know I am making the right decisions when working on a project?
….and many more.

In this video, I describe 5 books that will help web developers and any developer that has anything to do with the web to level up and resharpen their skills by filling up all the holes in their knowledge.

These are not your typical HTML/CSS/Javascript books. These books ground you in fundamental computer science concepts and the web, teach you all you need to know about scalability, and help you learn software architectures and best practices that make you a professional, industry-standard, and battle-tested (web) developer.

Ready? Let’s dive in. https://youtu.be/kNbIthp1KU0

Shopify Webhooks Tutorial: How to Use the Shopify API

Fikayo Adepoju — Thu, 16 Dec 2021 19:46:54 +0000

Introduction

Shopify webhooks help build integrations with external applications that need to respond in real-time to events taking place in a Shopify store. You can create and manage Shopify webhooks using the Shopify admin dashboard. Although easy to use, the admin interface does not allow developers to perform automated tasks on their webhooks. Instead, users have to log into the admin interface and manually perform webhook-related tasks.

As a solution to this drawback, Shopify opened up its webhook API. This allows programmatic interaction with Shopify webhooks, thereby expanding the scope of what is possible with the technology.

In this article, I will walk you through how to set up authentication with the Shopify API, and then use the Shopify webhooks API to create and view your webhooks. You will also use a locally running API to receive and log your webhooks.

Let's get right into it!

Prerequisites

To follow along with this tutorial, you will need a couple of things:

A Shopify store
The ability to create Shopify private apps (you may not have this permission if you're not the direct owner of the store you're working with)
Node.js installed on your system (if you already have a local API that you prefer to use, that is fine as long as you make sure it exposes a POST endpoint and listens on a known port)
Hookdeck CLI installed (you can find details on installing the tool here)

Creating a Shopify store app

The first step to accessing and using the Shopify webhooks API is to create a Shopify store app. There are 3 types of Shopify apps: public, custom, and private. To learn more about the different types of Shopify apps and how to create them, read the documentation pages here.

To access the Shopify API, we are going to create a private app. Private apps are built to work exclusively for your Shopify store (unlike public apps that can be used by any Shopify store and are listed on the Shopify App store). Private apps also allow you to access your store's data directly using the Shopify API.

To create a Shopify app, click Apps on the side menu of the admin interface. Just below the main section on the Apps page, click the Manage Private Apps link and then click the Create private app button.

On the app creation page, enter the name for the application you are about to create (in this example, Purchase Log API) and enter your email. Next, in the Admin API section, expand the Show Inactive Admin API and give Read and write access to the following resources...

Order editing
Orders

...as shown below:

Once the permissions are set, scroll down and select the latest API version (at the time of this writing, the latest version is 2021-10).

Click Save to complete this process. You will be prompted with a pre-save message informing you that an API key will be generated to provide access to the store's data.

Click Create app on this prompt and you will be taken to the app page. On this page, you will find the following credentials in the Admin API section:

API key (this is your username when using Basic authentication)
Password (this is your API access token)
Example URL (a sample URL format for performing basic authentication using a username and password)
Shared Secret (used for verifying webhooks, more on this later)

With these credentials, you have all you need to authenticate with the Shopify API.

Authenticating with the Shopify API

Now that you have your authentication credentials, let's discuss how you will go about making authenticated calls to the Shopify API. It is important to note that the different types of Shopify apps do not use the same authentication strategy.

While public and custom apps use OAuth and session tokens, private apps use basic authentication. In this section, I will only be discussing authentication for private apps.

There are 3 ways you can authenticate with the Shopify API using credentials from your private app.

1) Username and password combo

If your client supports basic authentication, you can include authentication credentials in your URL. This is done by prepending username:password@ to the hostname in the URL as shown below:

https://{username}:{password}@{shop}.myshopify.com/admin/api/{version}/webhooks.json

username: Your API key.
password: Your private app password.
shop: Your Shopify store subdomain string, e.g. hookdeck-store.
version: The version of the API you're using, e.g. 2021-10.

Any request made to the Shopify API using this URL is automatically authenticated.

2) Using an authorization token

Some clients do not support basic authentication as used in the above method. You can go around this issue by using the Basic {token} value in the Authorization header. The token value is derived by joining your store app's API key and password with a colon (:) and encoding the resulting string in base64. The Authorization header can then be sent, as shown below:

Authorization: Basic NDQ3OGViN2FjMTM4YTEzNjg1MmJhYmQ4NjE5NTZjMTk6M2U1YTZlZGVjNzFlYWIwMzk0MjJjNjQ0NGQwMjY1OWQ=

3) Using the X-Shopify-Access-Token header

The last strategy for authenticating private apps is to send the X-Shopify-Access-Token header, along with your request, to the Shopify API. The value of this header is your private app's password, which is your access token. The header is sent as follows:

X-Shopify-Access-Token: {my_app_password}

This is the strategy we will be using for all our API calls in this tutorial.

Now that you know how to authenticate with the Shopify API, let's start creating and receiving Shopify webhooks.

Setting up and receiving Shopify webhooks using the Shopify API

As described earlier, we will create a webhook using the Shopify API. Then, we'll spin up a local API to receive and log part of the webhook payload.

Before we begin, let's walk you through the steps necessary to achieve this webhook logging setup.

Clone the demo API and run it locally
Use the Hookdeck CLI to generate a webhook URL that points to the API endpoint
Create a webhook using the API
Test your webhook by triggering the subscription event
Inspect your webhook and webhook logs
Verify the webhook payload

Now that you're caught up, let's begin.

Clone the demo API

Clone the application repository for the API by running the following command:

git clone --single-branch --branch shopify-webhooks https://github.com/hookdeck/nodejs-webhook-server-example.git

Navigate to the root of the project and install the required dependencies by running the following commands:

cd nodejs-webhook-server-example
npm install

When the installation completes, you can then run the Node.js server with the following command:

npm start

This will boot up the API application and print a message to the screen indicating that the API is now running and listening for connections on port 1337.

We are using two endpoints in this project.

/log-shopify-webhook: This is the endpoint that will be receiving the Shopify webhook and logging it into an in-memory database. It logs a simple object containing a subset of the information from the webhook payload.
/fetch-webhooks-logs: This endpoint can be called to retrieve a collection of the logged webhook data.

Generate the webhook URL using the Hookdeck CLI

The next step is to use the CLI to generate a webhook URL that points to the running API application. To do this, run the following command (note: replace the port 1337 value with the port number on which your API is running if you're not using the sample project):

hookdeck listen 1337

This command starts an interactive session where the CLI collects information about the endpoint you're about to create. Below are the questions and the answers you should supply to each question. Ensure to hit the Enter key after each answer.

What source should you select? Ans: select **Create new source* (this prompt does not show up if you have not created any previous connections)*
What should your new source label be? Ans: type the text Shopify
What path should the webhooks be forwarded to (i.e.: /webhooks)? Ans: type /log-shopify-webhook (if you're using your own custom server, replace this value with your endpoint)
What's the connection label (i.e.: My API)? Ans: type Purchase Log API

With this information, the CLI will begin the process of generating the URL and once it's done, you will see the URL printed to the screen and the CLI indicating that it is ready to receive requests.

Note: If you're running the CLI in guest mode, you will need to use the guest Login URL link in the console to access the dashboard. Copy and paste this into your browser to begin a guest login session.

Create a Shopify webhook using the API

Now that you have your webhook URL, let's create the webhook. Shopify's webhook API can be accessed at https://{shop}.myshopify.com/admin/{version}/webhooks.json (remember to replace the values in {} with yours).

To create a webhook, you need to send a POST request to this endpoint containing data. You can use any HTTP client of your choice (Postman, Curl, etc.). In this tutorial, I am using ReqBin because it is accessible and easy to use.

For the webhook you are creating in this tutorial, you must subscribe to the orders/create. This event is fired when a new order is created in your store.

The data to send along with your webhook creation request should be structured as follows:

{
    "webhook" : {
            "topic" : "orders/create",
            "address": "{webhook_url}",
            "format": "json"
    }
}

As seen in the above snippet, the format to receive the webhook is set as json, so remember to replace {webhook_url} with the webhook URL generated on the Hookdeck CLI session.

Open https://reqbin.com and paste the API URL in the address field, then set the request method to POST. For authentication, add the X-Shopify-Access-Token field in the Headers section as shown below (replace MY_ACCESS_TOKEN with your Shopify app password):

The header will now be reflected in the details under the Raw section. Lastly, click the Content section and paste in your request object. See below a screenshot of the request setup:

Now hit the Send button to create your new webhook. You should get back a successful response, like you can see below:

This indicates that you have successfully created a webhook for the orders/create topic on your Shopify store.

Other actions that can be performed on your Shopify webhooks using the API include updating a webhook, fetching a single webhook, and deleting an existing webhook. For more information on how to perform these operations, visit the Shopify webhooks API documentation.

Test your webhook

With a locally running log API set up to receive webhooks, and a webhook subscription created via the API, the time has come to test your setup.

To receive a webhook for the subscribed topic, you need to trigger the topic's event by creating a new order. You can create a new order easily on your Shopify admin interface. Below, I am placing an order for a Ben Simmons 76er's jersey:

Once the order is created, Shopify will trigger a webhook for the orders/create event.

This will be received in your Hookdeck CLI session as shown below:

Inspect your webhook and webhook logs

The Hookdeck CLI helps us easily inspect the details contained in a Shopify webhook by providing a URL to an event page that breaks down the webhook data. This URL is the last item on the webhook entry, as shown in the CLI output above. Copy this URL and load it in your browser, and you will see an event page where you can view details like the webhook headers in the Headers section below:

You also have a browsable representation of the webhook payload in the Body section:

Finally, let's check to confirm that the locally running API is indeed logging the webhook payload. Still on your browser, navigate to the /fetch-webhooks-logs endpoint of the locally running API, and you will see a display similar to the one below (ignore the first object):

As seen, you have been able to log the webhook ID, the total number of items in your order, and the time the webhook was delivered.

Verify your Shopify webhooks

Shopify sends you an X-Shopify-Hmac-Sha256 header that can be used to verify that the payload contained in the webhook is from Shopify and has not been spoofed or tampered with.

This header is an encrypted version of the payload and is computed using the private app's shared secret, the webhook payload, and the HMAC cryptographic algorithm.

To verify your webhook on your external API, you compute the same encrypted version of the webhook payload using the shared key and HMAC, then compare the result with the value sent in the X-Shopify-Hmac-Sha256 header. If there is a match, then the payload is valid and it originated from Shopify. If not, then it's suspicious and should be ignored.

You can find this verification logic in the server.js file as a commented middleware function validatePayload, shown below:

const sigHeaderName = 'x-shopify-hmac-sha256';
const sigHashAlg = 'sha256';
const secret = "xx-xx-x";

.....

/* function validatePayload(req, res, next) {
    if(req.method == "POST"){
        if (!req.rawBody) {
            return next('Request body empty')
        }
        const body = req.rawBody;
        const hmacHeader = req.get(sigHeaderName);
        //Create a hash based on the parsed body
        const hash = crypto
            .createHmac(sigHashAlg, secret)
            .update(body, "utf8", "hex")
            .digest("base64");
        // Compare the created hash with the value of the X-Shopify-Hmac-Sha256 Header
        if (hash !== hmacHeader) {
            return next(`Request body digest (${hash}) did not match ${sigHeaderName} (${hmacHeader})`)
        }
    }
    return next()
}
app.use(validatePayload); */

For more information on security considerations for your webhooks, check out our security checklist.

Conclusion

In this article, you have learned and demonstrated how to perform Shopify webhook operations using the webhook API. Being a programmatic interface, the Shopify API allows you to run automated tasks that can, for example, update and fetch webhooks on the fly. You can perform these tasks directly in your code or within CI/CD pipelines to tailor Shopify webhooks towards your application/infrastructure needs.

Happy coding!

Tutorial: How to Use Shopify's Admin Dashboard to Create Webhooks

Fikayo Adepoju — Thu, 16 Dec 2021 19:30:46 +0000

Introduction

In a previous article, we looked at how Shopify webhooks provide a one-way communication medium for Shopify to send notifications and data about events taking place in your store to an external application. In this article, we will take a more hands-on approach to learning and understanding how Shopify webhooks work.

Each Shopify store comes with an administrative interface to manage the store. Webhooks can also be created and managed using this interface. In this tutorial, we are going to create a webhook, test it, and later delete it through Shopify's admin interface.

Read this tutorial if you are more interested in using Shopify's webhook API.

Ready? Let's get started.

How Shopify webhooks work

As stated earlier, Shopify webhooks help you integrate external applications with the Shopify store by creating a channel through which Shopify sends notifications for events happening in the store to the external application.

A webhook is set up on Shopify by subscribing to a topic on your store. A topic is an event that can take place on a Shopify store resource e.g, a Cart can be updated. Cart in this example is the resource while updated is the event, and this topic on Shopify is represented as Cart/update.

When a webhook subscription has been created, an HTTP request is triggered each time an event for that topic occurs. This HTTP request is sent to an endpoint on your external application. This endpoint is known as the Webhook URL and is supplied when creating a webhook on Shopify.

Shopify requires that webhook URLs use the secure HTTPS protocol and the POST request method.

Your external application receives the webhook with its payload at the webhook URL endpoint and triggers any custom processes in response to the webhook.

Tutorial: Logging Shopify cart events

In this tutorial we will explore logging, a common use case for Shopify webhooks, as engineers often use webhooks to feed information into an external logging system. These logs are mostly integrated with tools that help visualize the webhook data.

Here's a quick summary of the steps we need to take to achieve the Shopify webhook logging setup:

Clone the workflow logger API.
Generate a webhook URL that will be used to create a webhook on Shopify.
Set up a webhook on Shopify.
Trigger and inspect the webhook.
View the webhook logged by the API.

Now let's look at what you need for the project.

Project requirements

To set up our demo logging system, we are going to use a local Node.js API with an endpoint that receives and logs a portion of the webhook payload.

To follow along with this tutorial, it's required that you have the following items set up:

Have Node.js installed on your system, version 12 or greater is fine (if you already have a local API that you prefer to use, that is fine. Just make sure it exposes a POST endpoint and listens on a known port).
Hookdeck CLI installed (you can find details on installing the tool here).
An active Shopify store with access to Shopify apps (Shopify apps are required for the webhook authentication step, but more on this later).

Cloning the demo API

Clone the application repository for the API by running the following command:

git clone --single-branch --branch shopify-webhooks https://github.com/hookdeck/nodejs-webhook-server-example.git

Navigate to the root of the project and install the required dependencies by running the following commands:

cd nodejs-webhook-server-example
npm install

When the installation completes, you can then run the Node.js server with the following command:

npm start

This will boot up the API application and print a message to the screen indicating that the API is now running and listening for connections on port 1337.

We are using two endpoints in this project.

/log-shopify-webhook: This is the endpoint that will be receiving the Shopify webhook and logging it into an in-memory database. It logs a simple object containing a subset of the information from the webhook payload.
/fetch-webhooks-logs: This endpoint can be called to retrieve a collection of the logged webhook data.

Generating a webhook URL using the Hookdeck CLI

We now have a locally running server that contains an endpoint to receive our webhook. However, this endpoint is not accessible to the public, thus it cannot be targeted by Shopify webhooks.

A mechanism is required to route our webhooks to the locally running API endpoint, and this is where the Hookdeck CLI comes in. The Hookdeck CLI is built specifically for working with webhooks, and it helps tunnel external HTTP requests into your local development environment by providing you with a publicly accessible HTTPS URL. You can also inspect the headers and payloads of your webhooks, as you will see later on in this tutorial.

Visit Hookdeck's CLI documentation to install and set up the tool on your operating system. For macOS users, you can run the following command to install the CLI tool:

brew install hookdeck/hookdeck/hookdeck

If you're using the Windows operating system, use the following command to install the CLI tool:

scoop bucket add hookdeck https://github.com/hookdeck/scoop-hookdeck-cli.git
scoop install hookdeck

Once the setup process is complete, the next step is to use the CLI to generate a webhook URL that points to the running API application. To do this, run the following command (note: replace the port 1337 value with the port number on which your API is running if you're not using the sample project):

hookdeck listen 1337

What source should you select? Ans: select **Create new source* (this prompt does not show up if you have not created any previous connections)*
What should your new source label be? Ans: type the text Shopify
What path should the webhooks be forwarded to (i.e.: /webhooks)? Ans: type /log-shopify-webhook (if you're using your own custom server, replace this value with your endpoint)
What's the connection label (i.e.: My API)? Ans: type Purchase Log API

With this information, the CLI will begin the process of generating the URL and once it's done, you will see the URL printed to the screen and the CLI indicating that it is ready to receive requests.

Creating a webhook subscription using the Shopify admin interface

Now that you have your webhook URL, the next step is to create a webhook subscription on Shopify. On your Shopify admin dashboard, navigate to Settings → Notifications → (scroll down to) Webhooks. Click the Create webhook button and fill the webhook form as follows:

Event: Cart Update
Format: JSON
URL: The Webhook URL printed to your Hookdeck CLI session screen
Webhook API version: Select the version marked **Latest**

See below an example of the filled form:

Click Save to complete the webhook creation process. You will then see your newly created webhook displayed in the Webhooks list as shown below:

Testing your webhook

Great! You now have your webhook logging system set up, and it's time to put it to the test. Conveniently, Shopify provides a way to send a test webhook notification, which you can do by clicking the Send test notification button on your webhook row as displayed in the previous image.

Click this button for your Cart update webhook subscription to send a test webhook. Once sent, you will see an entry in your Hookdeck CLI session like below:

As seen in the above image, the status of the request is 201, indicating that a new resource has been created on the logger API.

To view details of the webhook, copy the event URL, which is the last item on the logged webhook in the CLI. Load this URL in your browser and you will see a screen similar to the one below:

The webhook event page provides you with in-depth details regarding the webhook you just received. Starting from the top section you can see some metadata provided by Hookdeck, shown below:

Next is the Headers section. Here, you can access all the headers that came with the webhook and inspect their values:

For the webhook payload, the Body section contains a browsable object representing the webhook payload. You can expand this object to inspect all the properties contained in the data that came in the payload:

Viewing the webhook logs

So, we have successfully received a webhook on our logging application. Let's now check the application logs to confirm that our webhook is indeed being logged. To have an appreciable amount of webhooks logged, click the Send test notification button a few more times.

Then, on your browser, navigate to the /fetch-webhooks-logs endpoint of the locally running API, and you will see a display similar to the one below:

As seen in the above image, we have 3 webhooks logged (the first entry is a test entry already contained in the database). We have been able to log the webhook ID, the total line items in the cart, and the time the webhook was triggered for each webhook. Now, it's time to verify.

How to verify Shopify webhooks with HMAC Signature

Webhooks create a communication medium between your external apps and Shopify, so you need to make sure that only Shopify is able to use this medium to speak to your app. Attackers often take advantage of the open-ended nature of a webhook URL to send requests containing malicious content to the webhook endpoint.

Fortunately, in the X-Shopify-Hmac-Sha256 header sent with each webhook, Shopify sends an encrypted version of the payload. This encrypted value is created using your Shopify app's shared secret key and the HMAC cryptographic algorithm.

You can get a Shopify shared secret for your store by creating a Shopify application in the Apps section. The shared key is contained in the Admin API section of the app created.

When you receive a webhook, you can compute the same encrypted version of the payload using the shared key and HMAC, and then compare the result with the value sent in the X-Shopify-Hmac-Sha256 header. If there is a match, then the payload is valid and it originated from Shopify. If not, then it's suspicious and should be ignored.

You can find this verification logic in the server.js file as a commented middleware function validatePayload, shown below:

const sigHeaderName = 'x-shopify-hmac-sha256';
const sigHashAlg = 'sha256';
const secret = "xx-xx-x";

.....

/* function validatePayload(req, res, next) {
    if(req.method == "POST"){
        if (!req.rawBody) {
            return next('Request body empty')
        }
        const body = req.rawBody;
        const hmacHeader = req.get(sigHeaderName);
        //Create a hash based on the parsed body
        const hash = crypto
            .createHmac(sigHashAlg, secret)
            .update(body, "utf8", "hex")
            .digest("base64");
        // Compare the created hash with the value of the X-Shopify-Hmac-Sha256 Header
        if (hash !== hmacHeader) {
            return next(`Request body digest (${hash}) did not match ${sigHeaderName} (${hmacHeader})`)
        } 
    }
    return next()
}
app.use(validatePayload); */

For more information on security considerations for your webhooks, check out our security checklist.

How to delete Shopify webhooks

Deleting webhooks from your Shopify store is simple and straightforward using the admin interface. Simply click the bin icon next to your webhook on the webhooks list:

Shopify will prompt you for to confirm to ensure that you really want to take this action. Note that once a webhook is deleted, it cannot be recovered again.

Conclusion

The Shopify administrative interface provides easy-to-use tools to create and manage your store webhooks. In this tutorial, you have learned and demonstrated how to use the Shopify admin interface to set up a webhook, test it, and then delete the subscription.

If you like doing things programmatically, or want to set up automated tasks like scheduled cron jobs to interact with Shopify webhooks and other store resources, Shopify also provides an API that you can call. We have detailed how you can authenticate with Shopify's API and use it to set up your webhooks in this article.

Happy coding!

Shopify Webhooks Best Practices

Fikayo Adepoju — Thu, 16 Dec 2021 19:16:04 +0000

Introduction

In this series, we've explored how to get started with Shopify webhooks, and shared step-by-step tutorials on how to create webhooks using both the Shopify admin dashboard and Shopify API. Now, we'll conclude the series by looking at some important best practices for building resilience and reliability into Shopify webhooks in a production environment. We want reliable Shopify webhooks because sudden spikes in your Shopify webhook traffic can overload your API and cause it to shut down if the capacity of your API server is exhausted. Shopify can also send duplicate webhooks that could potentially cause inconsistencies in your application database.

In this article, we will share and suggest some key engineering practices that will help make your Shopify webhooks resilient to the usage pressures common in production environments.

Shopify webhooks features

Before getting into best practices for Shopify webhooks, let's take a look at its features. Understanding the strengths and weaknesses of Shopify webhooks and how they operate helps engineers design the most efficient solutions to mitigate any issues that might arise when using them.

Feature	Related Best Practice	Related Best Practice
Webhook Configuration	Admin API and Shopify admin (REST or GraphQL)	Integration development
Webhook URL	Allow one per subscription	Integration development
Response Format	JSON or XML	Integration development, troubleshooting and testing
Request Method	POST	Integration development, troubleshooting and testing
Browsable Log	Delivery metrics on your partner dashboard	Troubleshooting and testing, failure recovery
Hashing Algorithm	SHA256	Security and verification
Manual Retry	Not available	Failure recovery
Automatic Retry Logic	Exponential, 19 times over 48 hours	Failure recovery
Alert Logic	Shopify sends an alert for each failure right before it deletes any webhooks	Failure recovery
Timeout period	5 seconds	Scalability with asynchronous processing

Now that you're familiar with the features of Shopify webhooks, let's go ahead and break down solutions to issues that may arise in production environments.

Shopify webhooks best practices checklist

Local troubleshooting and testing

Properly testing your Shopify webhooks helps you avoid all the performance issues that arise due to buggy code. For example, let's say your endpoint on an external accounting application assumes that Shopify will send the total price for a store order as an integer value (e.g. 10). Meanwhile, Shopify sends the price as a floating-point number (10.15). Your accounting application then forcefully coerces the floating number into an integer value which does not reflect the true cost of the order, and will lead to inconsistencies in the accounting data.

You need to test your Shopify webhook endpoint to ensure the following:

The endpoint exists (to avoid 404 errors)
The endpoint does not perform a redirect (Shopify sees a redirect as an error)
The logic on the endpoint is bug-free and has the desired impact on your application
You return a proper status code for successfully processing the webhook (2xx range of HTTP status code) and for failure (4xx, 5xx range of HTTP status codes)

One drawback in testing Shopify webhooks is that webhooks cannot be received in a local development environment because they require a publicly accessible URL. Fortunately, tools like the Hookdeck CLI exist to solve this problem by tunneling your Shopify webhook requests to your local environment.

Asynchronous processing to ensure reliability

When Shopify sends a webhook, it waits for a response from the receiving application. This waiting period elapses after 5 seconds. Thus, if Shopify does not receive a response within the waiting period, Shopify assumes that the webhook delivery has failed.

As a best practice, you need to respond to Shopify as fast as possible. This practice is recommended on Shopify's webhook documentation. That said, it might not be possible to respond so quickly under certain conditions, such as:

1) Long-running processes

Your webhook can trigger a time-consuming task, such as file encryption, database queries, or computations that take long periods to complete. It may be difficult, and sometimes impossible, to send a response within the 5-second limit in this situation.

2) Multiple webhooks

Let's assume that the time it takes your application to process a single webhook is 3 seconds; you would be able to return a response within the 5-second limit. However, if your application receives 3 webhooks at the same time, the time required becomes 9 seconds and the second webhook will not complete before it times out.

Imagine you have a Shopify store that sells flowers, and you process an average of 10 orders daily. You have set up a webhook that targets an external delivery service when a customer places an order. Your delivery service comfortably handles your daily average load and then suddenly, Valentine's Day season rolls by, and the number of orders for flowers on your store surges to an all-time high of 300 orders per day.

You start receiving 300 webhooks daily on your delivery service, which causes the server resources for the application to be used up quickly and drop webhooks due to service failure. As webhooks begin to drop, many customers that have ordered and paid for their flowers won't have their flowers delivered.

So, how do we mitigate these often unavoidable situations? One of the standard solutions is to boost processing speed by expanding the server hardware capabilities. You can also adopt horizontal scaling techniques to distribute your webhook workload.

The above solutions are good, but the best way to respond to Shopify as quickly as possible is by processing the webhooks asynchronously.

With asynchronous processing, Shopify does not need to wait for your application to finish processing a webhook before it receives a response.

One of the ways to implement asynchronous processing for Shopify webhooks is by placing a message broker between Shopify and your application. The message broker ingests webhooks from Shopify and immediately sends a response back to Shopify. The broker then routes the webhooks to your application at a rate that your application can handle.

This way, your Shopify webhooks never time out, and your application does not run out of server resources.

Verifying your incoming webhooks

The next best practice we'll look at is security. The HTTPS endpoint used by your applications to receive Shopify webhooks is publicly accessible, which means that any client can call it.

This open-ended nature of your webhook URL is a huge security flaw, as attackers can spoof webhook requests to your application for various malicious purposes. You need a way to prevent your webhook endpoints from receiving requests that are not from Shopify.

Let's imagine that you have a Shopify store that sells basketball jerseys, with a webhook set up to message a delivery service about the order details for the jersey to be delivered to the customer's address. Without verification, an attacker can spoof order webhooks to your delivery service webhook URL and get free jerseys anytime they want.

Fortunately, Shopify provides a way to help us with this. Shopify sends a hashed version of the body of the payload in the X-Shopify-Hmac-Sha256 header (if you're using Ruby on Rails or Sinatra, the header is HTTP_X_SHOPIFY_HMAC_SHA256 ). The value of this header is a base64 encoded string and serves as the digital signature for the webhook payload. This signature is signed using your Shopify app's shared secret. For more information about Shopify app secrets and how to generate one, check the docs page.

To verify the payload on your endpoint handler, you need to re-compute the signature using your Shopify secret, the webhook payload, and some HMAC algorithm functions. Once you have your version of the payload signature, you can then compare it with the one sent in the X-Shopify-Hmac-Sha256 to confirm a match.

Below is an example of the verification process written in Node.js:

const getRawBody = require("raw-body"); //install raw-body from npm
const crypto = require("crypto");
const secret = MY_SHOPIFY_SECRET;

app.post("/webhook", async (req, res) => {
//Extract X-Shopify-Hmac-Sha256 Header from the request
  const hmacHeader = req.get("X-Shopify-Hmac-Sha256");

//Parse the request Body
  const body = await getRawBody(req);
//Create a hash based on the parsed body
  const hash = crypto
    .createHmac("sha256", secret)
    .update(body, "utf8", "hex")
    .digest("base64");

  // Compare the created hash with the value of the X-Shopify-Hmac-Sha256 Header
  if (hash === hmacHeader) {
    console.log("Webhook source confirmed. Continue processing");
    res.sendStatus(200);
  } else {
    console.log("Unidentified webhook source. Do not process");
    res.sendStatus(403);
  }
});

It is also worth mentioning that you should store your Shopify secret in an environment variable, especially in production environments. The entire process of verifying your webhook breaks down if an attacker can get hold of your Shopify secret.

Idempotency in webhook processing

Idempotency helps prevent the negative impact of processing a webhook more than once. Let's imagine that we need to deduct the price of a purchased item from a customer's wallet (the wallet application is external to the Shopify store). A charge is deducted from the wallet when we receive a webhook confirming the purchase. If this process is done more than once for a single purchase, the customer's wallet is now in a state that is inconsistent with their purchase history.

Situations like this negatively impact the integrity of your application's data, and in extension, the Shopify store itself.

While some activities, like updating the status of an order, are by nature idempotent, others like the one described above are not. You need a mechanism built into your webhook processing operations to ensure that no webhook is processed more than once.

Shopify helps to achieve this by sending the unique identifier for a webhook in the X-Shopify-Webhook-Id. With this identifier, you can track webhooks that have already been processed and skip them to avoid duplicating their impact on your application.

Dealing with delayed webhooks

On rare occasions, Shopify might send a webhook late (sometimes up to a day late). If the data in the payload is very critical to your application, you should verify how recent the data is. Stale data has the potential to cause state inconsistencies within your application, thereby compromising the database.

For example, think of an external inventory system where the quantity of an item is updated (i.e. deducted) after a successful purchase on Shopify. If the item is no longer available (quantity = 0 in the inventory system) and you have a late purchase webhook coming in, the current quantity in the inventory will either be negative, undefined, or default to zero based on the business logic implemented. However, this new inventory value does not reflect the actual state of the item quantity.

Shopify does its best to send the most recent data with a webhook even when the webhook delays. As an engineer, you know that assumptions can easily lead to bugs, so you want to make sure you're verifying how recent the webhook data is despite the promise by Shopify.

One way to implement this verification is by comparing the timestamp of the webhook to the current time. If the timestamp on the webhook is behind the current time, you can call the Shopify API to request the current data. This check ensures that you're getting the most up-to-date data for the Shopify resource.

Implementing reconciliation jobs

This best practice falls under the housekeeping category. Webhooks are sent to notify your external application and send real-time data about an event on a Shopify resource. However, Shopify advises that you shouldn't solely rely on webhooks because, and this is very rare, the webhook for a particular event might not be sent.

An example of a scenario where a reconciliation job is required would be if you had a Shopify store selling shirts and an external accounting application integrated with your store using webhooks.

Let's assume that you need to balance your accounts in the external accounting application at the end of each business day. A reconciliation job can be scheduled to run 30 minutes before you query your transaction history to compute your balances. This job can be used to pull all purchase information from your Shopify store and synchronize it with your accounting application so that you have all the data you need up-to-date.

Thus, to mitigate the issue of unsent Shopify webhooks, you should implement reconciliation jobs to synchronize your application data with Shopify. These reconciliation jobs are scheduled tasks that periodically fetch data from Shopify for specific resources.

Reconciliation jobs should be set up for sensitive data within your application; these are data that need to be up-to-date at certain periods while your application is running.

Conclusion

Building reliability and resilience into your infrastructure ensures that your setup can withstand the pressures of production environments. In this article, we have looked at some best practices to follow to ensure that our Shopify webhooks function without issues. This list is not set in stone, and you're encouraged to take more precautions that help prevent your application integration with Shopify from falling short of expectations.

Happy coding!

Getting Started with Shopify Webhooks

Fikayo Adepoju — Thu, 16 Dec 2021 19:04:22 +0000

Introduction

Shopify is a great e-commerce SaaS application that offers a lot of flexibility and in turn helps developers create custom integrations. By using the API, merchants can programmatically interface their store with their custom applications. On top of the API, Shopify has introduced webhooks, a way to transfer event data from stores to custom integrations in real-time.

In this article, we will take a tour of Shopify webhooks to learn what they are all about, how to set them up, and how to find out more information on them.

What is a Shopify webhook?

Shopify webhooks are a one-way communication system. When an event occurs for a subscription to a Shopify topic (e.g. orders/create), an HTTPS request is triggered. This HTTPS request is then sent to an endpoint on the receiving application.

A Shopify topic is an event that can take place on a Shopify resource. In the orders/create example stated above, the resource isorders, and create is the event.

To understand things better, let's use a simple example to describe the webhook communication process.

A Shopify merchant that sells jerseys wants to send notifications for jersey orders on their Shopify store to an external delivery service. This delivery service is responsible for processing the delivery of the jersey to the customer.

The Shopify merchant goes into their Shopify admin dashboard (or uses the API) to create a subscription for the orders/create topic by submitting an endpoint on an external application where the webhook is to be sent. A customer visits the store and places an order. Once this order is created, the event for the orders/create topic is fired. Because there is a webhook subscription for this topic, an HTTPS request is immediately triggered to the external application's endpoint.

A Shopify webhook request consists of different parts. The webhook headers contain metadata about the webhook, and the payload contains the actual Shopify event information. The payload can be sent in either JSON or XML format. The format your webhook payload is sent as can be configured on Shopify's admin interface or through the Shopify API.

It is important to note that to receive webhooks, Shopify requires a publicly accessible HTTPS endpoint. Tunneling solutions like Hookdeck's CLI and Ngrok provide a way to generate an HTTPS endpoint that channels webhook requests to your development environment.

Supported Shopify webhook events

Shopify webhooks help applications that want to stay in sync with Shopify by updating internal states or executing code based on an event that occurs in a store. Applications that are often integrated with Shopify using webhooks include invoicing systems, accounting apps, and mailing/notification services, among other things.

Merchants can integrate Shopify webhooks to monitor sales, update inventory, perform accounting operations and synchronization, and also perform many more auditory and sales management processes. Making the right integration involves knowing the right event to subscribe to, so let's look at Shopify webhook events and how we can make use of them.

An event is fired when an action for a Shopify topic (e.g. cart/update topic, which means a Shopify cart is updated) takes place on Shopify. This action can be performed by the store admin, a customer, or Shopify itself.

Shopify supports a lot of events that webhooks can subscribe to. These events are namespaced by the resource that fires the event; for example, we have the Orders/closed and Orders/cancelled events belonging to the Orders resource.

Most e-commerce integrations focus on the Orders events as they generate webhooks for events involved in the process of purchasing items from the Shopify store. There are also events for resources like Cart, Checkout, Customer, InventoryItem, etc.

Let's take a look at some of these resource events, as well as example scenarios where they can be used in our application workflow.

Cart events

These events are fired when a customer creates or updates items in their shopping cart. The two available cart events are carts/create and carts/update. One potential use case for these events is to query an external inventory to check if an item is still in stock. You can also use these events as ping signals to an external inventory application in order to sync the number of items in the inventory with the number of items on your Shopify store.

Order events

Order events are fired when a Shopify order is created, updated, paid for, or deleted. Events under this category include: orders/cancelled, orders/create, orders/delete, orders/edited, orders/fulfilled, orders/paid, orders/partially_fulfilled, and orders/updated. Below are some scenarios where you can use webhooks for these events:

Reverting a payment through an external payment platform when a customer cancels an order on your Shopify store (orders/cancelled)
Notifying a delivery service once a customer pays for an item they ordered from your Shopify store (orders/paid)
Locking items that have been ordered in an external inventory application to ensure that preceding orders do not assume the wrong availability of an item or a wrong quantity of items (orders/create)

Dispute events

These are fired when a customer creates or updates an issue (complaint or comment) relating to the service they got from your store on Shopify. The two events under this category are disputes/create and disputes/update. One use case for these events is to create and update issues on an external issue tracker like Gorgias. Services like Gorgias have more customer support features built-in than what is available on Shopify.

To effectively take advantage of what Shopify webhooks have to offer, it is important to have a good knowledge of the Shopify events that you can subscribe for. This knowledge will help you build the proper workflows for webhooks within your application or third-party integrations.

It is also important to know that not all Shopify resources generate events. For a detailed list of Shopify events, check out the documentation page.

Creating a webhook using the Shopify admin interface

Now that you have a good understanding of Shopify webhooks and how they work, and you have chosen them as a solution to a project you're working on, let's look at how to create one in the Shopify admin interface. You can also follow along if you just want to get some practical knowledge on setting up webhooks on Shopify.

The easiest way to set up a webhook on your Shopify store is through the admin interface. To locate the admin interface, navigate to https://[MY_STORE_ADDRESS].com/admin. From the admin interface, go to Settings > Notifications > Webhooks. Click the Create Webhook button and fill the webhook form by selecting an event, the payload format, and entering your application endpoint (i.e. the endpoint on the application where you will be receiving your Shopify webhooks) into the URL field as shown below:

Ensure you select one of the stable API versions then click the Save button.

You will now receive a webhook request on the endpoint you supplied whenever the selected event occurs in your Shopify store.

For a more detailed hands-on tutorial on using the Shopify admin interface to create and manage your webhooks, check out this article.

Using the Shopify webhooks API

Webhook subscriptions cannot only be created from the Shopify Admin interface — you can also create and manage your webhooks dynamically by calling the Shopify webhooks API! The Shopify API comes in two forms: the very common REST API and a GraphQL variant.

The REST API consists of endpoints that you can use to create a webhook, retrieve webhooks, modify a webhook and also remove a webhook. The GraphQL API also consists of queries and mutations that mirror the same functionalities you get from the REST API.

To use the Shopify webhooks API, you need to be authenticated. Shopify also provides a good number of client libraries for languages like PHP, Node.js, and Python that make the developer experience a lot better.

For more information and a detailed analysis of using the Shopify API to manage your webhooks, check out our tutorial on using the Shopify webhooks API.

Finding information on the Shopify webhooks documentation

The Shopify documentation contains a great deal of information on webhooks; however, it can be a bit cumbersome moving around tons of documentation pages to find the exact information you are looking for. I have made a simple list below to help you easily find enough information to start using Shopify webhooks as a professional:

Resource	Why you should check it
Overview	If you are looking for more information on Shopify webhooks, this overview will be useful in helping to deepen your understanding of Shopify webhooks.
Shopify Webhook Configuration	This page contains a simple list of steps to take when configuring your webhook workflow.
Setting up your endpoint to receive webhooks	If you're setting up a Shopify webhook for the first time, this page takes you through a step by step process showing how to get your webhook set up properly. You learn how to prepare your endpoint to receive webhooks, how to test your webhook connection, and also how you can verify your webhook source.
API documentation	If you are looking to go down the API route, this page furnishes you with all the information about available endpoints, requesting an response format, supported events and more.
API versions	Over the years, different versions of the Shopify webhooks API have been released. If you're working on a project that uses older versions of the API, this page helps you manage different API versions in your code, and also teaches you how to upgrade to the latest version.

Conclusion

A large amount of activities on the internet are e-commerce related, and Shopify plays a huge part in that space. Shopify may have taken care of a lot of the basic requirements of an e-commerce store, but the platform's capacity for extensions makes it one of the most preferred choices for setting up an online store.

In this article, you have learned about Shopify webhooks and how they enable you to build custom integrations with the platform. This way, you can complement any missing feature in Shopify or substitute it with a better or preferred alternative.

Happy coding!

Troubleshooting GitHub Webhooks

Fikayo Adepoju — Mon, 01 Nov 2021 13:43:58 +0000

Introduction

In a previous article, I demonstrated the use of GitHub webhooks in a simple tutorial using a sample API that logged webhook events. In this article, we will learn how to debug common issues with GitHub webhooks using the same sample project. This time, I have introduced some bugs into the project that we are going to learn how to debug.

Troubleshooting GitHub webhooks: Requirements checklist

To begin there are a few things you need to have or set up, including:

A GitHub repository
Node.js installed on your system to run the sample project
A publicly accessible URL to the API endpoint. We will use the Hookdeck CLI to achieve this. To install the Hookdeck CLI, check out this page.
Clear visualization of error messages. The Hookdeck CLI will help us with event pages where error details can be inspected.
A text editor for editing code

With this setup, you have an environment in which you can conveniently troubleshoot your GitHub webhooks.

Troubleshooting GitHub webhooks with Hookdeck

Cloning and running a demo API

The sample API we will use is available on the Hookdeck GitHub repository. Follow the instructions to set it up.

Clone the repository by running the following command:

git clone --single-branch --branch github-webhooks-debugging https://github.com/hookdeck/nodejs-webhook-server-example.git

Navigate to the root of the project and install the required dependencies by running the following commands:

cd nodejs-webhook-server-example
npm install

When the installation completes, run the Node.js server with the following command:

npm start

This will boot up the API application and print a message to the screen indicating that the API is now running and listening for connections on port 1337.

We are using two endpoints in this project:

/log-github-webhook: This is the endpoint that will be receiving the GitHub webhook and logging it into an in-memory database. It logs a simple object containing a subset of the information from the webhook payload.
/fetch-webhooks-logs: This endpoint can be called to retrieve a collection of the logged webhook data.

Getting a GitHub webhook URL

The next step is to use the CLI to generate a webhook URL that points to the running API application. To do this, run the following command:

hookdeck listen 1337

What source should you select? Ans: select Create new source
What should your new source label be? Ans: type the text GitHub
What path should the webhooks be forwarded to (i.e.: /webhooks)? Ans: type /log-github-webhook
What's the connection label (i.e.: My API)? Ans: type My Github Response Server

With this information, the CLI will begin the process of generating the URL and once it's done, you will see the URL printed to the screen and the CLI indicating that it is ready to receive requests.

Note: You will need to use the guest Login URL link in the console to access the dashboard. Copy and paste this into your browser to begin a guest login session.

Confirming webhook delivery

To begin receiving webhooks, you need to set up a webhook on your GitHub repository.

Go to Settings → Webhooks. On the Webhooks page, click on the Add webhook button on the top right-hand corner.

On the webhook form displayed, paste the webhook URL generated by the Hookdeck CLI into the Payload URL field.

Fill the remaining fields as follows:

Content type: Select application/json so that you can receive the payload as a JSON object.
Secret: You can leave this blank, but for this tutorial enter 1234ABCD.
SSL Verification: Leave this as the default option, Enable SSL verification.
Which events would you like to trigger this webhook: This is the point where you subscribe for a GitHub event on your repository. For this tutorial, select the Just the push event option as we are only subscribing to the push event. You can either subscribe for all events or a subset of the events using the other two options.
Active: Leave this checked to receive event details when the GitHub webhook is triggered.

See the complete webhook form below:

Click the Add webhook button to complete the process.

With this setup, anytime you push code to your repository, a webhook request will be fired to the specified webhook URL.

Immediately after you complete registration for a webhook in the step above, a ping webhook request will be fired to your webhook URL. Because we are using the Hookdeck CLI, we would already see the ping webhook request logged on the terminal where the Hookdeck CLI is running, as shown below:

This confirms that we are successfully receiving our webhook request. However, did you notice the status code indicated on the webhook request? Yeah, a 404.

Let's take a look at how to deal with this error in the next section.

Troubleshooting GitHub webhook "not found" error

Our first webhook attempt resulted in a 404 error on the destination endpoint. This tells us that even though we are successfully receiving our webhooks, it cannot locate the endpoint specified, or the endpoint does not exist. 404 errors can be fixed, most of the time, by checking for typos or misspellings in the specified route name. Worst case scenario, the route truly does not exist and needs to be created.

Use the event page link displayed on the CLI to go to the event page, where you will see a page similar to the one below:

Scroll down to the red status code badge under the Attempts section and click on it to reveal details about the error on the right-hand side of the screen, as shown below:

This helps us see the actual response from the running server, and it also confirms that the specified endpoint cannot be found.

The endpoint specified for the webhook to hit is /log-github-webhook, which can be found in the routes.js file at the root of the project directory.

router.post("/log-github-webhoo", async function(req, res) {
  //console.log(req.body);

  const payload = req.body;

  let webhook_info = {
    repo : payload.repository.name,
    author : payload.sender.login,
    time : payload.head_commit.timestamp
  }

  const save_webhook = await req.db
  .collection("webhooks")
  .insertOne(webhook_info);

  res.status(201).send({
    message: "Webhook Event successfully logged"
  });
});

Taking a close look at the route handler, you will notice that there is a typo just at the end of the endpoint name: a missing 'k'. Simply add the missing character and save the file.

You will need to restart the Node.js server to allow the changes to take effect. Shut down the server using Ctrl + C and restart it with the npm start command.

Now, fire a new webhook to your endpoint by pushing a commit to your GitHub repository. Once this is done, check your Hookdeck CLI session for the new webhook received. You will see an entry similar to the one below:

As seen from the above webhook received, we have been able to clear the 404 error. However, we now have a new error: a 401 error. Let's deal with this in the following section.

Troubleshooting GitHub webhook error 401

HTTP 401 errors indicate that an unauthorized action is being performed. Before making any assumptions, let's inspect the server response using the event's page. Copy the event page link from the Hookdeck CLI session and load it in your browser, and you will see a screen similar to the one below:

Scroll down to the red status code badge under the Attempts section and then click on it. You will see the server response details on the right-hand side as shown below:

Reading the response message, we can see that we are failing the security check on the webhook payload. Because we defined an API secret, GitHub sends the X-Hub-Signature-256 header. This header contains an encrypted version of the real payload and has to be validated against the unencrypted payload received by using the secret key.

This is a security check to prevent attackers from replacing the actual payload with a malicious one with the intention of corrupting our API.

Because we know that we are receiving the actual payload from GitHub since we set everything up and triggered the push event, we know it's not an attacker's payload causing the 401 error. Something else must be wrong with our setup.

Let's take a look at the validation logic on our API. This can be found in the server.js file, as shown below:

//Validate payload
function validatePayload(req, res, next) {

    if(req.method == "POST"){
        if (!req.rawBody) {
            return res.status(400).send({
                message: "Request Body empty"
            });
        }

        const sig = Buffer.from(req.get(sigHeaderName) || '', 'utf8')
        const hmac = crypto.createHmac(sigHashAlg, secret)
        const digest = Buffer.from(sigHashAlg + '=' + hmac.update(req.rawBody).digest('hex'), 'utf8');

        if (sig.length !== digest.length || !crypto.timingSafeEqual(digest, sig)) {
            return res.status(401).send({
                message: `Request body digest (${digest}) did not match ${sigHeaderName} (${sig})`
            });

        }
    }

    return next()

}
app.use(validatePayload);

There are a couple of variables being used here in our code that are worth checking out to confirm that they are referencing the appropriate values. These are:

sigHeaderName: This represents the signature header name sent by GitHub.
sigHashAlg: This is the algorithm used for the encryption.
secret: This is the API secret set on our GitHub webhook.

These values are set on lines 10-12 in the server.js file. Lets take a look at these definitions:

const sigHeaderName = 'X-Hub-Signature-256';
const sigHashAlg = 'sha256';
const secret = "XXXXXX";

After observing these values, have you been able to catch the error? If the secret is not the same as the one we set in our webhook form, the validation will always fail.

GitHub does not allow you to view the API secret you set in your webhooks a second time — you can only reset it. This is why you need to make sure that you remember the value or store it somewhere secure.

For this practice, we know we set the secret as a simple 1234ABCD string. In real-world applications, you want to set a more complicated secret and reference it from an environment variable in your code.

Change the value of the secret to the right value, save the file, and restart the server. Now trigger another webhook by pushing a new commit to your GitHub repository. You will see a webhook entry similar to the one below on your Hookdeck CLI session:

Finally, we have a successful operation on our API. Load the event page, which should display a page similar to the one below:

Now click on the green status badge below the Attempts section to show the server response. The server returns a successful message, as shown below:

This time, when you visit the endpoint /fetch-webhooks-logs, you will see a collection like the one below:

Troubleshooting GitHub webhook "invalid HTTP response 400" error

Another error that can occur with GitHub webhooks is a 400 HTTP error, which indicates a bad request. One thing to note when working with GitHub webhooks is that GitHub is your client. Your application, or whichever system (SaaS apps, CI/CD server, etc.) that you're integrating GitHub webhooks with, is the server.

Thus, when you see a bad request, it means that your server is not receiving what it expects from GitHub.

The first thing you want to check is the format in which you're sending your webhook payload. GitHub allows you to set the Content-type to either application/json or application/x-www-form-urlencoded .

Make sure that the value you configured in your webhook form matches the content type your server is expecting.

Sometimes, 400 errors arise from empty payloads being sent by GitHub. If your server expects a payload to be sent, make sure that the Active option on your GitHub webhook form is checked.

Conclusion

Just like any other architectural component of an application, webhooks run into errors. Debugging is considered part of the workflow when developing applications or setting up application infrastructure. As errors are inevitable, knowing what to look out for when a certain error occurs and having the right tools to debug the situation eases the pains experienced in troubleshooting.

You also save time and push out solutions faster!

GitHub Webhooks Tutorial

Fikayo Adepoju — Thu, 21 Oct 2021 18:44:27 +0000

Introduction

As one of the biggest code hosting platforms available today, GitHub provides numerous ways for developers to integrate with the platform. One of those integration options is GitHub webhooks.

In this article, we are going to explore GitHub webhooks. We will look at what a GitHub webhook is, and how to set it up on our GitHub account. Further, we will explore a sample API that receives and logs webhook events. This API will run on our local development environment, thus we will be receiving the webhooks on our local machine. However, if you have a local API you would rather use, feel free to follow along with the article.

What is GitHub Webhooks?

GitHub Webhooks gives developers the ability to integrate with the GitHub platform.

Developers can subscribe to events taking place on GitHub, such as push events that occur when code is pushed to a repository, and respond to the event with an action.

The response action can be anything: making a database entry in an external application, sending an email, or triggering a deployment process, just to name a few. Actions you can perform in response to events on GitHub are limited only by your imagination.

When a subscribed event occurs on GitHub, GitHub triggers an HTTP POST request to the defined destination. This destination is specified as an HTTP URL, which is the endpoint to be called whenever the event occurs. This is known as the webhook URL. The endpoint handler at the webhook URL handles the action to be performed in response to the event.

Along with the request, GitHub also sends the payload of data about the event that triggered the webhook. Information contained in this payload can be used for the action on the endpoint.

Tutorial: How to set up GitHub Webhooks

To begin the tutorial, let's take a look at the steps involved:

Clone the sample Node.js API for receiving GitHub webhooks on your development machine
Generate a webhook URL using the Hookdeck CLI
Register for a webhook on GitHub
Receive and inspect GitHub webhooks locally
Make some commits and view logs

Clone a demo API server for receiving GitHub webhooks

As described earlier, GitHub webhooks need to be sent to an endpoint on your API.

If you already have a server running with a POST endpoint to receive the webhook request, you can skip this section. All you need to do is to take note of the port that your local API is running and also the endpoint to receive webhooks. Substitute these values as you follow this exercise.

If you do not have a local API, you can use a Node.js sample API available on the Hookdeck GitHub repository. Follow the instructions below to set it up.

Clone the repository by running the following command:

git clone --single-branch --branch github-webhooks https://github.com/hookdeck/nodejs-webhook-server-example.git

Navigate to the root of the project and install the required dependencies by running the following commands:

cd nodejs-webhook-server-example
npm install

When the installation completes, you can then run the Node.js server with the following command:

npm start

This will boot up the API application and print a message to the screen indicating that the API is now running and listening for connections on port 1337.

We are using two endpoints in this project.

/log-github-webhook: This is the endpoint that will be receiving the GitHub webhook and logging it into an in-memory database. It logs a simple object containing a subset of the information from the webhook payload.
/fetch-webhooks-logs: This endpoint can be called to retrieve a collection of the logged webhook data.

Get the webhook URL for GitHub

Endpoints on locally running servers cannot receive GitHub webhooks; a publicly accessible endpoint is required and this is where the Hookdeck CLI comes in. The Hookdeck CLI is built specifically for working with webhooks, and helps tunnel external HTTP requests into your local development environment by providing you with a publicly accessible HTTPS URL. You can also inspect the headers and payloads of your webhooks, as you will see later on in this tutorial.

Visit Hookdeck's CLI documentation to install and set up the tool on your operating system. For macOS users, you can run the following command to install the CLI tool:

brew install hookdeck/hookdeck/hookdeck

If you're using the Windows operating system, use the following command to install the CLI tool:

scoop bucket add hookdeck https://github.com/hookdeck/scoop-hookdeck-cli.git
scoop install hookdeck

hookdeck listen 1337

What source should you select? Ans: select Create new source
What should your new source label be? Ans: type the text Github
What path should the webhooks be forwarded to (i.e.: /webhooks)? Ans: type /log-github-webhook (If you're using your own custom server, replace this value with your endpoint)
What's the connection label (i.e.: My API)? Ans: type My Github Response Server

With this information, the CLI will begin the process of generating the URL and once it's done, you will see the URL printed to the screen and the CLI indicating that it is ready to receive requests.

Note: You will need to use the guest Login URL link in the console to access the dashboard. Copy and paste this into your browser to begin a guest login session.

Set up webhook on GitHub

Next, you will need to subscribe to a GitHub webhook. Navigate to your repository of choice (this should be a repository you own) and go to Settings → Webhooks. On the Webhooks page, click on the Add webhook button on the top right-hand corner.

On the webhook form displayed, paste the webhook URL generated by the Hookdeck CLI into the Payload URL field.

Fill the remaining fields as follows:

Content type: Select application/json so that you can receive the payload as a JSON object.
Secret: You can leave this blank, but for the purpose of this tutorial enter 1234ABCD.
SSL verification: Leave this as the default option of Enable SSL verification.
Which events would you like to trigger this webhook? This is the point where you subscribe for a GitHub event on your repository. For this tutorial, select the Just the push event option, as we are only subscribing to the push event. You can either subscribe for all events or a subset of the events using the other two options.
Active: Leave this checked to receive event details when the GitHub webhook is triggered.

See the complete webhook form below:

Click the Add webhook button to complete the process.

With this setup, any time you push code to your repository, a webhook request will be fired to the specified webhook URL.

Receive and inspect GitHub webhooks and payload

Here we can see that we have received a POST request with a 201 status code indicating that a new entity was created on our server; in other words, a webhook object was logged.

To view details of the webhook request, its headers, and payload, copy the Hookdeck dashboard URL printed along with your webhook request details on the CLI and load it in your browser.

You will see a page similar to the one below:

On this page, you can go through and inspect all the data that came with your webhook request. For example, the Headers section is expanded in the image below:

You can also view all that is contained in the webhook request payload by expanding the Body section as shown below:

With this setup, you get full and clear visibility into your webhook requests and you will be able to monitor and quickly troubleshoot your webhooks when any failure occurs.

View Application webhook logs

To check that the webhook information is being logged, make a few commits to your GitHub project. You should see webhook event entries in the terminal where your Hookdeck CLI session is running. Now visit the endpoint /fetch-webhooks-logs, where you should see a couple of log entries similar to the image below:

As seen above, we are logging the repo, author, and timestamp of the commit. Ignore the first entry as this is simply a test entry so that the application database does not start empty.

GitHub webhooks verification

A publicly accessible endpoint, just like the one being used as your webhook URL, can receive requests from anyone. This situation makes your endpoint vulnerable, as ill-intentioned individuals or bots can easily spoof requests imitating GitHub to dump irrelevant data into your system or manipulate your application.

You should verify the requests hitting your webhook URL to ascertain that they are from GitHub. GitHub helps you achieve this using the token secret we set in the form in an earlier step.

GitHub uses this token to create a hash signature with each payload. This hash signature is included with the headers of each request as X-Hub-Signature-256.

With this signature, you can validate your payloads. GitHub uses the HMAC algorithm to compute the hash and it is the same algorithm you will use to implement the validation on your server.

At the moment, our project is not enforcing this check but the code can be found commented out as shown below:

//Validate payload
/* function validatePayload(req, res, next) {

    if(req.method == "POST"){
        if (!req.rawBody) {
            return next('Request body empty')
        }

        const sig = Buffer.from(req.get(sigHeaderName) || '', 'utf8')
        const hmac = crypto.createHmac(sigHashAlg, secret)
        const digest = Buffer.from(sigHashAlg + '=' + hmac.update(req.rawBody).digest('hex'), 'utf8');

        if (sig.length !== digest.length || !crypto.timingSafeEqual(digest, sig)) {
            return next(`Request body digest (${digest}) did not match ${sigHeaderName} (${sig})`)
        }
    }

    return next()

}
app.use(validatePayload); */

From lines 10 to 12, you can see the validation variables defined, as shown below:

const sigHeaderName = 'X-Hub-Signature-256';
const sigHashAlg = 'sha256';
const secret = "1234ABCD";

The secret variable is the key we set when creating our webhook. In a production environment, these values should be kept in environment variables.

You can uncomment the payload validation middleware to ensure that your webhook payload is validated against the API secret set.

For more information on securing your GitHub webhooks, you can check out the GitHub documentation here.

How do I disable a GitHub Webhook?

At the moment, there is no setting on a GitHub webhook configuration that disables a webhook. Some will suggest that you uncheck the Active parameter but this only stops the delivery of the webhook payload, it may not prevent the event itself from being fired.

The only way you can be sure you have disabled a webhook is by deleting it completely from your GitHub Webhooks page. This may seem a bit extreme if you only want to disable it temporarily.

With Hookdeck, you can use the Pause feature on the dashboard to temporarily stop your webhooks from being delivered. When you're ready to start receiving them once again, you can unpause the webhook.

Conclusion

Webhooks are a fantastic offering by GitHub to help developers extend their integrations with the platform to more custom use cases. This enables the development of more robust applications that take advantage of events occurring on GitHub.

In this tutorial, you have learned how to set up and receive GitHub webhooks on your API endpoints. The tasks you can perform in response to these events are only limited by your imagination.

Happy coding!

Top 5 Use Cases for GitHub Webhooks

Fikayo Adepoju — Wed, 06 Oct 2021 20:28:09 +0000

Introduction

In our previous article, we looked at how to get started using GitHub webhooks.

However, what are the actual real-world applications of GitHub webhooks? In what ways can this technology deliver value to our applications? These are the questions we will be trying to provide answers to in this article by taking a look at some of the most common real-world applications of GitHub webhooks.

Why are GitHub webhooks important/useful?

Before we dive into the various use cases of GitHub webhooks, let's discuss why GitHub webhooks are a useful piece of technology. For over a decade now, GitHub has served as a central hub for storing, sharing, and collaborating on code-based projects.

It can be argued that GitHub houses the code for most of the software applications, frameworks, plugins, and any code-based technology that we use today. Thus, it is important that such a platform should open doors to integrations with other external systems that rely on the code hosted on it.

GitHub webhooks provides one such integration opportunity by allowing external applications to subscribe for events taking place on GitHub. These external applications can then receive notifications and information about their subscribed events, and trigger workflows in response to these events. This way, subscribed applications can deliver more value to their users (especially through the automation of tasks). The beauty of GitHub webhooks is that they are not restricted to any use case; you are only limited by your own imagination.

In the following sections, we will take a look at some of the most common scenarios in which application engineers have employed the use of GitHub webhooks to give you an idea of the flexibility that is possible.

Triggering continuous integration build pipelines

One of the most common applications of GitHub webhooks is the triggering of continuous integration pipelines to run tests on code being pushed to a branch. Team members working on a project send pull requests from feature branches to the main branch. Before these feature branches are merged to the main branch, it's considered best practice to run the code to be merged through tests.

You can set up a continuous integration (CI) pipeline on a CI server (Jenkins, CircleCI, etc.). Then subscribe for push and pull_request events on GitHub to fire webhooks to trigger your CI server to run tests on the branch.

This will prevent defective code from being merged to your main branch when errors are detected in the CI build and the build fails.

Automatically deploying applications to destination servers

For web projects, it is important for development teams to deliver their code to application servers where users and/or testers can access it. Whether you need to deliver your web app to a staging or production environment, a continuous delivery pipeline can be set up to move your code to the hosting environment.

However, this deployment pipeline needs a trigger to tell it when to ship your code to the host. Developers can simply subscribe for a push event on the main branch (for deployment to production servers), or any deployment branch. When the push event occurs, a webhook is sent to trigger the deployment pipeline to ship the code to the hosting environment.

CI/CD platforms like CircleCI have API endpoints to trigger a deployment pipeline. You can simply point your webhook to this endpoint to trigger a deployment to your servers.

Sending notifications to team collaboration channels like Slack and Discord

GitHub tries its best to send email notifications for commits and pull requests on repositories. However, development teams spend most of their time on collaboration platforms like Slack and Discord.

Thus, the most effective way to communicate activities happening on code repositories to team members, especially code reviewers and deployment approvers, is via these collaboration channels. This ensures that anyone who needs to take immediate action gets the notification on time. This can also help facilitate discussion on commits and issues, which allows the team to stay in sync with activities taking place on the repository.

Discord allows developers to generate a webhook endpoint URL that can be submitted to GitHub when subscribing to a webhook. This allows you to get instant notifications on subscribed events in your Discord channels. You can follow this GitHub gist to see how GitHub webhooks can be used to channel event details to Discord channels.

When subscribing for webhooks to be sent to discussion channels, you can subscribe to as many events as you're interested in. There are no rules as to what you should have sent to you.

Updating external issue trackers

GitHub includes an Issues tab on each repository where users of the application and code collaborators can submit issues observed. Discussion threads can then be created on each issue to share information about resolving the issue.

However, some external applications do a better job at handling issues than GitHub. These issue trackers often possess more features for dealing with code issues and helping developers collaborate better on a project.

Some of these issue trackers, like Jira, have built-in support for GitHub to follow up on issues on a repository. Others, like Pivotal Tracker and MongoDB Realm, provide you with a webhook URL that you can add to GitHub to subscribe to the issues events, issue comment event, commit comment event, and any other event related to issues tracking.

Automating workflows on external processes

Another interesting use of GitHub webhooks is in triggering workflows on external applications and processes. This use case is custom, and always dependent on what the application engineer wants to achieve.

It can be as simple as logging activities taking place on GitHub for audit purposes, or triggering database servers to backup data before a release is made. Project management applications can subscribe to events like issues and pull_request to keep track of the project.

Customer support applications can subscribe to the issues event to collate complaints, comments, and issues that users are reporting on GitHub.

If there is a process you need to automate in your custom application, database servers, SaaS applications, or email servers in response to any activity on GitHub, you can achieve this automation with GitHub webhooks.

Conclusion

The simplicity of GitHub makes it easy to integrate into a wide range of external applications and processes. Almost every application supports HTTP, and since webhook requests are simple HTTP requests carrying a payload of event information, they can easily communicate with a lot of online applications. In this article, we have looked at some of the common scenarios where GitHub webhooks are deployed.

GitHub webhooks use cases are not limited to what we have seen in this article, but having these examples at hand will help you better understand how to use GitHub webhooks to their full potential.

If you are interested in building an integration with GitHub webhooks, we recommend reading our tutorial for GitHub webhooks to kick start your project!

Happy coding!

Getting Started with GitHub Webhooks

Fikayo Adepoju — Wed, 29 Sep 2021 16:23:06 +0000

Introduction

GitHub is arguably the largest remote code hosting service in the world. It can also be argued that it is the most-used platform by developers around the world for working on both personal and team code-based projects. One of the attributes contributing to GitHub's widespread adoption is the ability to integrate the platform into development workflows, OAuth applications, CI/CD processes, and many more use cases.

In this article, we take a look at one of the strategies for integrating GitHub into external workflows and applications, known as GitHub webhooks.

What are GitHub webhooks?

Just as webhooks are being provided by many SaaS and PaaS systems for the purpose of integration, GitHub webhooks give developers the ability to integrate with the GitHub platform.

Developers can subscribe to events taking place on GitHub, for example Push events that occur when code is pushed to a repository, and respond to the event with an action.

The response action can be anything, such as making a database entry in an external application, sending an email, triggering a deployment process, and more. Actions you can perform in response to events on GitHub are only limited by your imagination.

When a subscribed event occurs on GitHub, GitHub triggers an HTTP POST request to the defined destination. This destination is specified as an HTTP URL, which is the endpoint that is to be called whenever the event occurs. This is known as the webhook URL. The endpoint handler at the webhook URL handles the action to be performed in response to the event.

GitHub also sends the payload of data about the event that triggered the webhook along with the request. Information contained in this payload can be used for action to be taken on the endpoint.

What are the differences betweem GitHub webhooks and actions ?

GitHub Actions is a built-in CI/CD tool for GitHub and allows developers to automate the testing and deployment of code. When new code is pushed to a GitHub repository, GitHub actions can be set up to test the code, merge it to the main repository when tests pass and also deploy the application to a destination server.

GitHub webhooks, on the other hand, are just plain subscriptions to events that take place on the repository and can be used to trigger different actions asides from CI/CD workflows.

You can use GitHub webhooks to achieve the same CI/CD operations found in GitHub Actions, however, GitHub is not responsible for the CI/CD operation. You will need to connect your CI/CD triggers to the necessary event in your repository with GitHub webhooks and use an external CI/CD server.

GitHub actions also require a configuration script to set up, whereas GitHub webhooks do not require a script.

How do I use webhooks on GitHub?

Webhooks can be configured on a GitHub organization account, a specific repository, or a GitHub App. A webhook is connected to a GitHub event type and you can configure webhooks for more than one event. Once set up, webhooks will be sent each time a subscribed event is triggered.

Here are the steps to set up a webhook on GitHub:

Navigate to the repository you want to subscribe to events on
Go to Settings → Webhooks
Click on Add webhook
Fill in the webhook form specifying your webhook URL
Subscribe to all, or only the events you're interested in
Click the Add webhook button to complete the process

See below an example of a webhook setup on GitHub.

Immediately after the webhook is created, GitHub sends a ping webhook to your endpoint. This is to test that your endpoint is live, and you can also use it to verify the expected reaction on the webhook URL.

You can create up to 20 webhooks for each organization, repository, or GitHub app you're interested in.

Webhooks can also be configured via the GitHub API through authenticated sessions.

Types of GitHub webhooks events

As mentioned earlier, each webhook is tied to an event that takes place on GitHub, so let's look at the types of events that GitHub supports. GitHub supports over a dozen event types but except in very peculiar cases, your interest will be focused on a few common ones.

Some of the common events supported by GitHub are:

Push Event: This event is triggered when one or more commits are pushed to a repository branch or tag. This is one of the most common events developers subscribe to, as a lot can be done in response to a Push Event. You can trigger tests in continuous integration pipelines, send alerts to code reviewers on your team to review the new commit, and even trigger deployments to staging and/or production servers.
Pull Request Event: This event is triggered for any activity that involves Pull Requests, for example when a Pull Request is opened, assigned, labeled, or closed, etc. The type of activity is specified in the action property of the webhook request payload.
Issues Event: Any activity related to issues causes this event to be triggered. Just like the Pull Request Event, the type of activity is specified in the action property of the payload. The issue itself is contained in the issue property. One common use case of this event is in updating an external issue tracker.
Create Event: This event is fired when a repository user creates a new branch or tag in the remote repository.

For a full list of events supported by GitHub, their description and payload properties, check out the documentation here. Familiarity with the supported events will help you get the best out of GitHub webhooks.

While subscribing for events on a repository, you might be tempted to just subscribe for every single event available. This is not advisable, as your GitHub webhooks rate limits can easily be exceeded. Your endpoint will also be burdened with a large number of irrelevant events.

Finding information on the GitHub webhooks documentation

GitHub has well-documented and exhaustive material on webhooks. However, sometimes you just need to get started quickly and you want to navigate to the information you need as fast as possible.

Below is a list of the most common pages you will be interested in:

Conclusion

GitHub webhooks further expand the range of integration that developers have with the platform. Also, they are easy to set up and highly reliable. With GitHub webhooks, you can subscribe to events that trigger workflows in your external applications, like your custom application servers, Zapier, G Suite, CI/CD servers, etc.

In this article, we dipped our toes in the water that is GitHub webhooks. In subsequent articles, we are going to be taking a closer look at using GitHub webhooks, securing them, and best practices you should be aware of.

Happy coding!