Import DEV Posts To Jekyll With GitHub Actions

GitHub Actions are a new way to create custom workflows on GitHub. They are language agnostic, so whether you prefer to write them in Python, Node.js, Ruby or PHP, you can do so. Once the proper folder structure is created in your GitHub repository, GitHub will immediately recognize the presence of a new Action, and act accordingly.

(tl;dr You can view this Action on GitHub right now.)

There are so many possible things you can do with GitHub actions, it would take an entire blog post to just begin to describe them.

For me, I wanted to build an Action to automate the importing of my DEV blog posts into my Jekyll site on a cron schedule. I really enjoy the writing experience on DEV and want to keep that as my primary blogging platform, while also importing that data into my personal site.

In this case, the end objective will be to have a GitHub Action that:

1. Checks once a week for new DEV blog posts
2. If there are new blog posts raise a pull request with generated markdown to add the post to my Jekyll site

Furthermore, in my use case, I wanted to keep the number of blog posts on my Jekyll site limited to a specific number of recent posts as specified in an environment variable, ${NUM_OF_POSTS}. Therefore, I wanted the Action to also do the following:

• If there is a new DEV post, but there are already ${NUM_OF_POSTS} on the Jekyll site, then also delete the oldest Jekyll post as part of the new pull request.

I specifically wanted the Action to raise a pull request from a new branch, and not just commit the changes, because I try to follow the GitHub flow for new work. I wanted my Action to also follow that flow.

The first item I did in creating the Action was add an action.yml file that contained some basic information about the Action:

name: DEV Blog Posts To Jekyll Markdown
description: Search for new DEV blog posts and raise a PR with the post converted to Jekyll Markdown Post
runs:
  using: docker
  image: Dockerfile
branding:
  icon: copy
  color: white

Specifically, in the above file, I defined the name of the Action, a brief description, and that I want it to run on Docker. If I publish the Action to the GitHub Marketplace, I also specified the icon and color to use for the listing.

Once that is done, I moved on to building the actual Action. All the code for it is located inside an index.js file.

Initially, I set up my require statements. The Action requires the following dependencies:

const { Toolkit } = require('actions-toolkit');
const dotenv = require("dotenv");
dotenv.config();
const axios = require('axios').default;
const btoa = require('btoa');

I used Jason Ectovich's actions-toolkit to interact with the GitHub API. I also used dotenv to manage my environment variables, axios for my asynchronous API calls and btoa to Base64 encode the contents of the new Jekyll markdown file to be created as part of the pull request.

All the code for the Action is wrapped inside a Toolkit.run() function. Once I get some basic repository information from the GitHub API stored in variables to be used later, the next thing to do was to get my posts from the DEV API. That is accomplished using axios:

const getData = () => {
  return axios({
    method: 'get',
    url: 'https://dev.to/api/articles/me?page=1&per_page=6',
    headers: headers
  })
};

devPosts = (await getData()).data;

At this point, the devPosts variable holds my most recent DEV posts.

Next, I needed to also get the contents of my Jekyll site's _posts folder. I did that using the toolkit:

posts = (await tools.github.repos.getContents({
  owner,
  repo,
  path
})).data;

The posts variable holds now the contents of my _posts folder.

During this time I also store some data points in variables to be used later:

• Information on my last Jekyll post
• The number of posts in _posts
• The published dates for both my most recent DEV and Jekyll posts
• The titles of my most recent DEV and Jekyll posts.

I also create the file name for the newest markdown Jekyll post. I do that by manipulating the data returned from DEV into the pattern used for my Jekyll file names:

newJekyllPostFileName = `${devPostDate.split('T')[0]}-${devPostTitle.toLowerCase().split(' ').join('-')}.md`;

Now I am ready to check on the conditions necessary for making a new pull request. I need to answer the following conditions:

• Is the latest DEV post newer than the latest Jekyll post?
• Are there more or equal number of posts on Jekyll than ${NUM_OF_POSTS}?
• Does the new working branch already exist in my repository?
• If the branch already exists, does the new markdown file I want to add also already exist?
• Does the pull request already exist?

The answer to each question above will alter the course of action the code takes.

If the conditions are met, the Action will end up creating a pull request that adds a new markdown file with the following contents:

fileContents = `
---
layout: defaults
modal-id: ${postsCount+1}
date: ${devPostDate}
img: ${devPostCoverImage}
alt: Cover Image
title: ${devPostTitle}
link: ${devPostURL}

---
`.trim();

I could have also added the entire DEV post to the markdown file, or only a portion of the text, but in my use case on my personal site, I only want to create a small blog post card that links to the blog post on DEV.

Depending on the number of current posts in the Jekyll _posts folder, the pull request may also include the deletion of the oldest Jekyll post:

deletedPost = (await tools.github.repos.deleteFile({
  owner,
  repo,
  path: lastPostPath,
  message: 'Deleting oldest blog post from Jekyll site',
  sha: lastPostSHA
}));

The process of building this action was a great learning experience for me. I knew that I wanted to keep a current list of my most recent blog posts on my personal site. I also knew that I did not want to manually manage that process. The GitHub Actions tooling provided me with the opportunity to build an automated workflow to manage that process for me.

I 💙Contributions! If you have ideas for improvements and enhancements, please bring them. If you would like to use this Action in your own project, please do that as well! I'd love to hear about how you are using it.

The DEV Posts to Jekyll Markdown Action can be found on GitHub.

Comments

[Aleksandr Hovhannisyan]

Nice work! Sounds useful.

Wish there was something for the opposite—syncing posts from a Jekyll site to Dev.to. Things like custom includes and Liquid tags would make that very difficult, so I just copy-paste and reformat by hand.

One word of advice: Be sure to add a canonical link to the head of each blog post on your site, and have its href point to the corresponding Dev.to blog post. Otherwise, if Google sees your duplicate content, it may lower your site's ranking.

[Ben Greenberg]

Have you tried setting up the Publishing from RSS option in your DEV profile? I can imagine the custom includes and things like that would still mean you need to go back and edit before publishing, but at least you could skip the copy and paste step? :)

Yes, canonical URL is super important, totally agree! I only want my blog posts to live on DEV, so I'm not importing the post, just basically creating a link back to DEV from the data. But, if someone forked the action repo and modified it to import the whole post, they should definitely add the canonical link reference. Making the search engines happy is important!

[Aleksandr Hovhannisyan]

Ah, no, I'll have to look into that!

[Christian Kohler]

Interesting. Thank for sharing. I do it the other way around. I manage my markdown files on github and push them with github actions to dev.to. Have a look at: dev.to/christiankohler/introducing...

[Ben Greenberg]

That's really cool. I like how you made it independent of the platform, and users can choose where to syndicate to.