Using webhooks to update a self-hosted Jekyll blog

I host the blog of a community project on my personal server. It's a static website generated with Jekyll, and the contents are versionned in a git repository.

In this blog post, I explain how I made the blog automatically update on every new commit. It's the same idea as Github Pages, but for self-hosting.

TL;DR

• create a script to clone the git repository, then build and deploy the website
• publish a webhooks endpoint on the server
• configure the GitHub repository to send a webhook on every new commit

Deploy script

The website is hosted on a Debian server, and the files are served from the /var/www/atlas.tecnologia.bo/ directory. To update it from command line, first install the Jekyll requirements, then create and launch the bash script:

$ /opt/deploy_atlas.tecnologia.bo.sh

that contains:

#!/usr/bin/env bash

# Clone git repository
rm -rf /tmp/atlas.tecnologia.bo
git clone https://github.com/RipeAtlasBolivia/atlas.tecnologia.bo.git /tmp/atlas.tecnologia.bo

# Generate blog
cd /tmp/atlas.tecnologia.bo
bundle install
bundle exec jekyll build --source /tmp/atlas.tecnologia.bo/ --destination /var/www/atlas.tecnologia.bo/

# Clean
rm -rf /tmp/atlas.tecnologia.bo

Webhook daemon

Instead of manually launching the script, let a webhook daemon do the job every time it receives an HTTP request on an endpoint.

First install the webhook package:

$ sudo apt install webhook

Verify the service is running:

$ sudo service webhook status
● webhook.service - Small server for creating HTTP endpoints (hooks)
   Loaded: loaded (/etc/systemd/system/webhook.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2019-08-12 08:25:24 UTC; 2 weeks 0 days ago
     Docs: https://github.com/adnanh/webhook/
 Main PID: 419 (webhook)
    Tasks: 6 (limit: 4697)
   Memory: 4.8M
   CGroup: /system.slice/webhook.service
           └─419 /usr/bin/webhook -verbose -hooks /etc/webhook.conf

We configure the webhook daemon, creating the file /etc/webhook.conf with the following content (see the webhook documentation for more details):

[
  {
    "id": "atlas",
    "execute-command": "/opt/deploy_atlas.tecnologia.bo.sh",
    "command-working-directory": "/tmp/",
    "trigger-rule": {
      "match": {
        "type": "payload-hash-sha1",
        "secret": "xxxxxxxxx",
        "parameter": {
          "source": "header",
          "name": "X-Hub-Signature"
        }
      }
    }
  }
]

This creates a new hook called atlas, that will be triggered when a GET or POST request is received on http://localhost:9000/hooks/atlas. When triggered, it will check that a valid X-Hub-Signature HTTP header has been provided, and then launch the /opt/deploy_atlas.tecnologia.bo.sh script from the /tmp working directory.

Try it:

• compute the X-Hub-Signature corresponding to the xxxxxxxxx secret

$ echo -n "" | openssl sha1 -hmac "xxxxxxxxx"
(stdin)= d46d87941e6f285be78ff0f1c8ea32620577b9ef

• request the endpoint:

$ curl -X POST -H "X-Hub-Signature: sha1=d46d87941e6f285be78ff0f1c8ea32620577b9ef" -v http://localhost:9000/hooks/atlas
...
> POST /hooks/atlas HTTP/1.1
> Host: localhost:9000
> User-Agent: curl/7.64.0
> Accept: */*
> X-Hub-Signature: sha1=d46d87941e6f285be78ff0f1c8ea32620577b9ef
>
< HTTP/1.1 200 OK
< Date: Mon, 26 Aug 2019 13:45:09 GMT
< Content-Length: 0
<

• check the logs:

$ grep webhook /var/log/syslog
Aug 26 14:06:40 webhook[10552]: [webhook] 2019/08/26 14:06:40 Started POST /hooks/atlas
Aug 26 14:06:40 webhook[10552]: [webhook] 2019/08/26 14:06:40 [d2ae30] incoming HTTP request from [::1]:58800
Aug 26 14:06:40 webhook[10552]: [webhook] 2019/08/26 14:06:40 [d2ae30] atlas got matched
Aug 26 14:06:40 webhook[10552]: [webhook] 2019/08/26 14:06:40 [d2ae30] atlas hook triggered successfully
Aug 26 14:06:40 webhook[10552]: [webhook] 2019/08/26 14:06:40 Completed 200 OK in 670.302µs
Aug 26 14:06:40 webhook[10552]: [webhook] 2019/08/26 14:06:40 [d2ae30] executing /opt/deploy_atlas.tecnologia.bo.sh (/opt/deploy_atlas.tecnologia.bo.sh) with arguments ["/opt/deploy_atlas.tecnologia.bo.sh"] and environment [] using /tmp/ as cwd
Aug 26 14:06:46 webhook[10552]: [webhook] 2019/08/26 14:06:46 [d2ae30] command output: Cloning into '/tmp/atlas.tecnologia.bo'...
...
Aug 26 14:06:46 webhook[10552]: Bundle complete! 4 Gemfile dependencies, 24 gems now installed.
Aug 26 14:06:46 webhook[10552]: Use `bundle info [gemname]` to see where a bundled gem is installed.
Aug 26 14:06:46 webhook[10552]: Configuration file: /tmp/atlas.tecnologia.bo/_config.yml
Aug 26 14:06:46 webhook[10552]:             Source: /tmp/atlas.tecnologia.bo/
Aug 26 14:06:46 webhook[10552]:        Destination: /var/www/atlas.tecnologia.bo/
Aug 26 14:06:46 webhook[10552]:  Incremental build: disabled. Enable with --incremental
Aug 26 14:06:46 webhook[10552]:       Generating...
Aug 26 14:06:46 webhook[10552]:        Jekyll Feed: Generating feed for posts
Aug 26 14:06:46 webhook[10552]:                     done in 0.45 seconds.
Aug 26 14:06:46 webhook[10552]:  Auto-regeneration: disabled. Use --watch to enable.
Aug 26 14:06:46 webhook[10552]: [webhook] 2019/08/26 14:06:46 [d2ae30] finished handling atlas

Note: the 200 HTTP code only means the hook has been found for the atlas identifier and the X-Hub-Signature is valid. It says nothing about the deploy script success or failure.

In order to publish the endpoint from a Apache webserver, configure a reverse proxy in your Apache configuration:

ProxyPass /webhook/ http://localhost:9000/hooks/
ProxyPassReverse /webhook/ http://localhost:9000/hooks/

GitHub configuration

Finally, in order to trigger a webhook on every new push, go to the GitHub project settings, then "Webhooks", and create a new webhook with the following parameters:

• Payload URL: https://mydomain/webhook/atlas
• Content type: application/json
• Secret: xxxxxxxxx
• Which events would you like to trigger this webhook?: Just the push event.

Alternative configuration with GitLab

If your git repository is hosted on a GitLab instance, you must change the headers check in the /etc/webhook.conf file:

[
  {
    "id": "atlas",
    "execute-command": "/opt/deploy_atlas.tecnologia.bo.sh",
    "command-working-directory": "/tmp/",
    "trigger-rule": {
      "match": {
        "type": "value",
        "value": "xxxxxxxxx",
        "parameter": {
          "source": "header",
          "name": "X-Gitlab-Token"
        }
      }
    }
  }
]

Note: the X-Gitlab-Token header provided in the request will contain the value xxxxxxxxx, not its HMAC as it occurs with GitHub. So, to simulate the request, adapt the curl command to:

$ curl -X POST -H "X-Gitlab-Token: xxxxxxxxx" -v http://localhost:9000/hooks/atlas

Then, the GitLab project configuration is similar to GitHub: go to the project "Settings", "Integrations", and add a webhook with:

• URL: https://mydomain/webhook/atlas
• Secret Token: xxxxxxxxx
• Trigger: [x] Push events

References

• https://www.rasmuslarsson.se/post/hugo.html
• https://github.com/adnanh/webhook/
• https://jekyllrb.com/

Comments

[elMor3no]

Another question ;)

How im able to use the same webhook services for diferent request

Im try something like:

[
{
"id": "atlas",
"execute-command": "/scripts/myscript1.sh",
"command-working-directory": "/tmp/",
"trigger-rule": {
"match": {
"type": "value",
"value": "secrect1",
"parameter": {
"source": "header",
"name": "X-Gitlab-Token",
}
}
}
}
]
[
{
"id": "atlas2",
"execute-command": "/scripts/myscript1.sh",
"command-working-directory": "/tmp/",
"trigger-rule": {
"match": {
"type": "value",
"value": "secrect2",
"parameter": {
"source": "header",
"name": "X-Gitlab-Token",
}
}
}
}

]

Im try in diferent ways but all the time i get:

[webhook] 2019/12/17 00:06:22 Started POST /hooks/atlas2
[webhook] 2019/12/17 00:06:22 Completed 404 Not Found in 30.368µs
[webhook] 2019/12/17 00:06:28 Started POST /hooks/atlas
[webhook] 2019/12/17 00:06:28 Completed 404 Not Found in 34.086µs

If i use only one all works fine:

[
{
"id": "atlas",
"execute-command": "/scripts/myscript1.sh",
"command-working-directory": "/tmp/",
"trigger-rule": {
"match": {
"type": "value",
"value": "secrect1",
"parameter": {
"source": "header",
"name": "X-Gitlab-Token",
}
}
}
}
]
[webhook] 2019/12/17 00:02:37 Started POST /hooks/atlas
[webhook] 2019/12/17 00:02:37 Completed 200 OK in 164.625µs

[Sylvain Lesage]

I think it's because of your configuration JSON format: instead of

[
  {...1}
]
[
  {...2}
]

you should have

[
  {...1},
  {...2}
]

You could use a linter like jsonlint.com/?json= or prettier.io/ to check your configuration file.

[elMor3no]

Thanks....

Thats solve the issue..

[hrvoj3e]

Nice. Thanks for this.

But, is this secure?
The hook scripts in execute-command run as root when using weebhook as a service (systemd).

[Sylvain Lesage]

Sure, you might want to edit /etc/systemd/system/webhook.service to specify the user, for example a dedicated webhook user created for these tasks only:

[Service]
User=webhook

[elMor3no]

Nice...

How i can debug the execution of the command...

Im not able to check why anything happends evens when i get a access sucessful...

[Sylvain Lesage]

You can access the logs produced by the command with:

grep webhook /var/log/syslog

I updated the blog post to add this information.

Note that you need access to the server to see these logs. From a client point of view, you will only see a 200 HTTP code if the command has been launched successfully, but this doesn't mean the command itself has worked as expected.