Discussion on: Automatically update your GitHub Pages website with posts from dev.to

[Bruno Drugowick]

@renanfranca , I have a few ideas, I can take a look.

But, basically, if the dev.to API allows you to get comments for a post, it's just a matter of creating other static data files for each post with the comments, or even combining the comments on a huge structure.

I'll read the docs and try to make it work later.

[Renan Franca]

@brunodrugowick ! Thank you very much for the feedback 😊!

I'll read the docs and try to make it work later.

That will be awesome! Take your time and have a nice week 🙂

[Bruno Drugowick]

Yeah, it's quite easy to do it. Here's the bash script to get the comments for each post saving them to a separate file. My suggestion would be to create a page for each post now. Maybe there's some things to figure out to actually generate each page now from the list of files in bash or maybe there's a way to do this in Jekyll (maybe it would require to save all the comments to the same file, I don't know.

Anyway, here's something with which you can start:

#!/bin/bash

DATA_FOLDER=./_data
POSTS_FILE=$DATA_FOLDER/devPosts.json

echo -n "" >$POSTS_FILE # Clear the posts file
POSTS_JSON=$(curl https://dev.to/api/articles?username=brunodrugowick&?per_page=1000) # GET posts from user
echo "$POSTS_JSON" | jq >>$POSTS_FILE # Create data file for posts

POST_IDS=$(cat "$POSTS_FILE" | jq -r '.[].id') # Extract post ids
for POST in $POST_IDS # Loop through posts
do
    COMMENTS_JSON=$(curl https://dev.to/api/comments?a_id="$POST") # GET comments from post
    echo $COMMENTS_JSON | jq >>$DATA_FOLDER/$POST.json # Create data file for post comments
done

date >last_updated_date

[Bruno Drugowick]

For the other readers I wanna say this is starting to defeat the purpose of my post. The stack I chose was designed to be simple because I had a simple problem to solve. I wouldn't choose the same stack if I needed this to scale.

Having said that, for you, @renanfranca , what I say is "let's do this, let's see what happens". Hahaha. You're still looking for the ideal solution to your problem and you're kinda still defining your problem (or at least that's what it looks like to me). Let's see where this will take you... doing is the best way to learn.

And let me know if you need any help... vejo que é brasileiro, podemos economizar o inglês até. Hahaha.

[Renan Franca]

Thank you very much for the bash script! That's gold 🪙 for me because I Am still learning it ☺️

I will give the feedback for sure!

I am sorry for disturbing the discussion section 😅

---pt-br-----------
Lhe adicionei no twitter, qualquer coisa eu lhe envio uma DM por lá 👍

[Bruno Drugowick]

Here's my new version of the script, closer to what you're looking for:

#!/bin/bash

DATA_FOLDER=./_data
POSTS_FOLDER=./_posts
POSTS_FILE=$DATA_FOLDER/devPosts.json
POST_HEADER=$(cat <<- 'EOF'
---
layout: post
title:  "{{TITLE}}"
date:   {{DATE}}
---
<style type="text/css" media="screen">
  .card {
    box-shadow: 0 4px 8px 0 rgba(0,0,0,0.2);
    transition: 0.3s;
    width: 100%;
  }
  .card:hover {
    box-shadow: 0 8px 16px 0 rgba(0,0,0,0.2);
  }
  .container {
    padding: 2px 16px;
  }
</style>
EOF
)

echo -n "" >$POSTS_FILE # Clear the posts file
rm -r $POSTS_FOLDER/* || mkdir $POSTS_FOLDER
POSTS_JSON=$(curl -s https://dev.to/api/articles?username=brunodrugowick&per_page=1000) # GET posts from user
echo "$POSTS_JSON" | jq >>$POSTS_FILE # Create data file for posts

POST_IDS=$(cat "$POSTS_FILE" | jq -r '.[].id') # Extract post ids
for POST in $POST_IDS # Loop through posts
do
    ARTICLE_JSON=$(curl -s https://dev.to/api/articles/"$POST") # GET full article
    POST_NAME=$(echo "$ARTICLE_JSON" | jq -r '"\(.published_at | split("T")[0])-\(.slug).markdown"')
    POST_URL=$(echo "$ARTICLE_JSON" | jq -r '.url')
    # Using HTML instead of Markdown because posts from DEV can use tags like 'post' that I cannot use here :(
    POST_BODY_HTML=$(echo "$ARTICLE_JSON" | jq -r '.body_html')
    POST_FILENAME=$POSTS_FOLDER/$POST_NAME
    POST_TITLE=$(echo "$ARTICLE_JSON" | jq -r '.title')
    POST_DATE=$(echo "$ARTICLE_JSON" | jq -r '.published_timestamp')
    POST_CATEGORIES=$(echo "$ARTICLE_JSON" | jq -r '.tag_list')

    touch "$POST_FILENAME"
    (
        printf '%s' "$POST_HEADER";
        echo "";
        echo "";
        echo "<code>[$POST_CATEGORIES]</code>";
    printf '%s' "<div class="card">
      <div class="container">
        <h4><b><br>You'll have a better experience reading in DEV</b></h4>
        <p><a href=\"$POST_URL\" target=\"_blank\">Click here to continue reading this post there >></a></p>
        <p>However, if you want to know more about the project to mirror my posts from DEV here (and why), go ahead and <a href=\"$POST_URL\" target=\"_blank\">read more</a>.</p>
        <p>You can continue to read here too, it's up to you... =]</p>
      </div>
    </div>" >>"$POST_FILENAME";
        echo "<br>";
    printf '%s' "$POST_BODY_HTML..."
    )>>"$POST_FILENAME"

    sed -i 's/{{TITLE}}/'"$POST_TITLE"'/1' "$POST_FILENAME"
    sed -i 's/{{DATE}}/'"$POST_DATE"'/1' "$POST_FILENAME"
done

date >last_updated_date

[Renan Franca]

Thank you so much! I am learning a lot about how bash is powerful because you comment on the script explain what each line does 👏

I will explain what exactly I want to do.

Let's take the following blog post on my website renanfranca.github.io/2022/03/25/w....
I want to bring only the discussion (comments) from the dev.to post dev.to/renanfranca/why-do-i-need-t... to the place highlight at the following image:

If the reader wants to join the discussion I'll put a link to my dev.to post.

My motivation

My blog post on my website isn't the same from dev.to. On my website, I use Jekyll features (ex: embed twitter posts) and I mention other authors by referencing his Twitter, Here at dev.to I use its own embedded markdown style and I reference other authors' dev.to profiles.

I want to bring the discussions to my website because I think those dev.to discussion complement my blog post.

Here is my github website repository in case you're curious: github.com/renanfranca/renanfranca...