Automatically update your GitHub Pages website with posts from dev.to

Important note

It's all free. The idea here is to get a working solution without spending a penny.

Introduction

I've already talked about GitHub Pages a long while ago when I first created my personal website.

Then, for convenience, I switched to another tool, which was free at the time, and never touched my website again. I won't name it because from around 2 years ago to now, when I decided to write a few things again, the tool is just a mess and it's not working. I'd have to pay to make it work the way I want, which is basically:

• A freaking simple static website with the list of my posts in dev.to.

The reasons: I don't make any money from my website, is just a place to put some things I like to talk about; also, I forget things easily and will need to refer to these things in the future for sure.

Cut to the chase

1. Build your personal website

I won't spend time here, GitHub has lots of info about this... and I have a post on how to combine it with Google Domains (that recently got out of Beta, by the way).

2. Create a script to get the posts

Dev.to as a very nice API and, most importantly, GET requests are free! So we can create a very simple script to get your posts and then - and here's the trick, save to a file that your GitHub Pages website understands as data! Here's the script:

#!/bin/bash

#### Thing to notice 1 ####
POSTS_FILE=./_data/devPosts.json

# Clear the posts file
echo -n "" >$POSTS_FILE

#### Thing to notice 2 ####
POSTS_JSON=$(curl https://dev.to/api/articles?username=brunodrugowick&?per_page=1000)

#### Thing to notice 3 ####
echo "$POSTS_JSON" | jq >>$POSTS_FILE
date >last_updated_date

Thing to notice 1

Notice how this variable points to a file under the _data directory. This is a special directory for GitHub Actions, that's powered by Jekyll.

Files in this directory can be used as data when building your static website. You'll see in a few how this is used.

Thing to notice 2

This part of the script is the request to the dev.to API. Notice the parameters:

• username=brunodrugowick is getting the posts for my user. You should change that to your username.
• per_page=1000 is the number of posts to retrieve. I believe I won't ever get to that number, but in case I do I'll work on a loop to handle the pagination. =)

Thing to notice 3

This part of the script writes the result of the request above to the data file that we referenced earlier. The pipe (|) to jq is to format nicely, probably not required.

3. List the posts in your website

At this point, this script can already be used in your local environment to fill in a data file that you can use in your GitHub Pages website. Let's do this then.

Put this code in your index.md page (you can use .html too or put this code anywhere else that you want in your website):

{% for devPost in site.data.devPosts %}
## {{ devPost.title }}

{{ devPost.readable_publish_date }}

{{ devPost.description }} [continue to read]({{ devPost.url }})

{% endfor %}

This is a simple for loop that goes through all of the posts that you saved to a devPosts file using the script that we created earlier.

The directive site.data.devPosts is what refers to that file. Everything that we access for a post in particular, like devPost.description or devPost.url is refering to the fields returned from the dev.to API.

4. Automate the task of updating your website

What we have so far is good enough, right? Every time you write something to dev.to you could go to your personal website repository, run the script and push a commit to let GitHub build and publish your website for you.

But we can go one step further and automate the whole thing with GitHub Actions! We'll create a workflow called Update Articles in your repository. Create a file .github/workflows/update-articles.yml in your repository with the following content:

name: Update Articles

#### Thing to notice 1 ####
on:
  schedule:
    - cron:  '30 23 * * *'
  workflow_dispatch:

jobs:
  run-script:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v2

      - name: Run script to get articles from dev.to
        #### Thing to notice 2 ####
        run: |
          ./scripts/post_files_from_GET_json.sh

      - name: Commit to the repo
        run: |
          git config --global user.name 'Bruno Drugowick'
          git config --global user.email '[e-mail]'
          git add .
          git commit -am "Update articles list"
          git push

Thing to notice 1

This section ensures your workflow runs both on a schedule (every day at 11:30 PM) or whenever you want via workflow_dispatch. Here's what this will look like in your repository later:

Thing to notice 2

This section is just instructing GitHub Actions to run your workflow on the local copy of the repository that was checked out one step above.

The rest of the script is a commit to the repository with the change. This simplified version makes sure we commit anything, even if none, but we could verify if there was a change before deciding to push a commit to the repository, for example.

Conclusion

There you have it, now whatever you write on dev.to is also mirrored on your personal website.

Please let me know in the comments if you know another solution for this "problem".

Comments

[Renan Franca]

Thank you so much for sharing this! 💓

I became a Jekyll fan this year, I started to use GitHub as my blog platform.

I cross-post my blog posts here on dev.to, I want to show the discussions from dev.to on the comment section of my blog post.

Think of how to do it

Bash script

I am a noob on bash script, please, feel free to help me.
I will have to get all my posts as you did. Then I will have to grab the comments from each posts.
Help me here: is there a way to grab the comments associated with the post title?

_layout/post.html

I will compare the title from current post with the title from the _data/comments_download and if the title is the same I will print it.
to-do I will have to figure out how to show it as beautiful as dev.to does.

bad idea

I thought to use Javascript to do that, but it will destroy the performance gains from static pages.

@brunodrugowick , I will appreciate if you can advise me 🤗

[Bruno Drugowick]

@renanfranca , I have a few ideas, I can take a look.

But, basically, if the dev.to API allows you to get comments for a post, it's just a matter of creating other static data files for each post with the comments, or even combining the comments on a huge structure.

I'll read the docs and try to make it work later.

[Renan Franca]

@brunodrugowick ! Thank you very much for the feedback 😊!

I'll read the docs and try to make it work later.

That will be awesome! Take your time and have a nice week 🙂

[Bruno Drugowick]

Yeah, it's quite easy to do it. Here's the bash script to get the comments for each post saving them to a separate file. My suggestion would be to create a page for each post now. Maybe there's some things to figure out to actually generate each page now from the list of files in bash or maybe there's a way to do this in Jekyll (maybe it would require to save all the comments to the same file, I don't know.

Anyway, here's something with which you can start:

#!/bin/bash

DATA_FOLDER=./_data
POSTS_FILE=$DATA_FOLDER/devPosts.json

echo -n "" >$POSTS_FILE # Clear the posts file
POSTS_JSON=$(curl https://dev.to/api/articles?username=brunodrugowick&?per_page=1000) # GET posts from user
echo "$POSTS_JSON" | jq >>$POSTS_FILE # Create data file for posts

POST_IDS=$(cat "$POSTS_FILE" | jq -r '.[].id') # Extract post ids
for POST in $POST_IDS # Loop through posts
do
    COMMENTS_JSON=$(curl https://dev.to/api/comments?a_id="$POST") # GET comments from post
    echo $COMMENTS_JSON | jq >>$DATA_FOLDER/$POST.json # Create data file for post comments
done

date >last_updated_date

[Bruno Drugowick]

For the other readers I wanna say this is starting to defeat the purpose of my post. The stack I chose was designed to be simple because I had a simple problem to solve. I wouldn't choose the same stack if I needed this to scale.

Having said that, for you, @renanfranca , what I say is "let's do this, let's see what happens". Hahaha. You're still looking for the ideal solution to your problem and you're kinda still defining your problem (or at least that's what it looks like to me). Let's see where this will take you... doing is the best way to learn.

And let me know if you need any help... vejo que é brasileiro, podemos economizar o inglês até. Hahaha.

[Renan Franca]

Thank you very much for the bash script! That's gold 🪙 for me because I Am still learning it ☺️

I will give the feedback for sure!

I am sorry for disturbing the discussion section 😅

---pt-br-----------
Lhe adicionei no twitter, qualquer coisa eu lhe envio uma DM por lá 👍

[Bruno Drugowick]

Here's my new version of the script, closer to what you're looking for:

#!/bin/bash

DATA_FOLDER=./_data
POSTS_FOLDER=./_posts
POSTS_FILE=$DATA_FOLDER/devPosts.json
POST_HEADER=$(cat <<- 'EOF'
---
layout: post
title:  "{{TITLE}}"
date:   {{DATE}}
---
<style type="text/css" media="screen">
  .card {
    box-shadow: 0 4px 8px 0 rgba(0,0,0,0.2);
    transition: 0.3s;
    width: 100%;
  }
  .card:hover {
    box-shadow: 0 8px 16px 0 rgba(0,0,0,0.2);
  }
  .container {
    padding: 2px 16px;
  }
</style>
EOF
)

echo -n "" >$POSTS_FILE # Clear the posts file
rm -r $POSTS_FOLDER/* || mkdir $POSTS_FOLDER
POSTS_JSON=$(curl -s https://dev.to/api/articles?username=brunodrugowick&per_page=1000) # GET posts from user
echo "$POSTS_JSON" | jq >>$POSTS_FILE # Create data file for posts

POST_IDS=$(cat "$POSTS_FILE" | jq -r '.[].id') # Extract post ids
for POST in $POST_IDS # Loop through posts
do
    ARTICLE_JSON=$(curl -s https://dev.to/api/articles/"$POST") # GET full article
    POST_NAME=$(echo "$ARTICLE_JSON" | jq -r '"\(.published_at | split("T")[0])-\(.slug).markdown"')
    POST_URL=$(echo "$ARTICLE_JSON" | jq -r '.url')
    # Using HTML instead of Markdown because posts from DEV can use tags like 'post' that I cannot use here :(
    POST_BODY_HTML=$(echo "$ARTICLE_JSON" | jq -r '.body_html')
    POST_FILENAME=$POSTS_FOLDER/$POST_NAME
    POST_TITLE=$(echo "$ARTICLE_JSON" | jq -r '.title')
    POST_DATE=$(echo "$ARTICLE_JSON" | jq -r '.published_timestamp')
    POST_CATEGORIES=$(echo "$ARTICLE_JSON" | jq -r '.tag_list')

    touch "$POST_FILENAME"
    (
        printf '%s' "$POST_HEADER";
        echo "";
        echo "";
        echo "<code>[$POST_CATEGORIES]</code>";
    printf '%s' "<div class="card">
      <div class="container">
        <h4><b><br>You'll have a better experience reading in DEV</b></h4>
        <p><a href=\"$POST_URL\" target=\"_blank\">Click here to continue reading this post there >></a></p>
        <p>However, if you want to know more about the project to mirror my posts from DEV here (and why), go ahead and <a href=\"$POST_URL\" target=\"_blank\">read more</a>.</p>
        <p>You can continue to read here too, it's up to you... =]</p>
      </div>
    </div>" >>"$POST_FILENAME";
        echo "<br>";
    printf '%s' "$POST_BODY_HTML..."
    )>>"$POST_FILENAME"

    sed -i 's/{{TITLE}}/'"$POST_TITLE"'/1' "$POST_FILENAME"
    sed -i 's/{{DATE}}/'"$POST_DATE"'/1' "$POST_FILENAME"
done

date >last_updated_date

[Renan Franca]

Thank you so much! I am learning a lot about how bash is powerful because you comment on the script explain what each line does 👏

I will explain what exactly I want to do.

Let's take the following blog post on my website renanfranca.github.io/2022/03/25/w....
I want to bring only the discussion (comments) from the dev.to post dev.to/renanfranca/why-do-i-need-t... to the place highlight at the following image:

If the reader wants to join the discussion I'll put a link to my dev.to post.

My motivation

My blog post on my website isn't the same from dev.to. On my website, I use Jekyll features (ex: embed twitter posts) and I mention other authors by referencing his Twitter, Here at dev.to I use its own embedded markdown style and I reference other authors' dev.to profiles.

I want to bring the discussions to my website because I think those dev.to discussion complement my blog post.

Here is my github website repository in case you're curious: github.com/renanfranca/renanfranca...

[Jackson Kasi]

nice...

[Bruno Drugowick]

For those of you that are following this, my most recent version only commits when changes are detected.

Here's the action: github.com/brunodrugowick/brunodru...

And the referenced scripts: github.com/brunodrugowick/brunodru...

[Dhanush N]

Cool one

Checkout my GitHub, my latest dev.to posts are automatically updated on my Readme

github.com/DhanushNehru

[Bruno Drugowick]

I didn't mention, but of course you can configure whatever field that comes from the API to show up in your page. The example (and my usage) is quite simple but you can be creative and make it look good.