Converting a dev.to feed to markdown

dev.to has become my primary writing platform. Although personal articles are nicely grouped on one's landing page I wanted to reference them on my Jekyll-powered homepage, too. Something like this:

Being a programmer enables you to build tools that just do what you want. Nothing more. Nothing less.

Take a look.

As Jekyll is based upon markdown, mine will create *.md' files. Here's the template:

val template = """
---
layout: post
title: "!!!TITLE!!!"
date: !!!FULL_DATE_AND_TIME!!!
categories: articles
---
!!!DATE!!! on dev.to  
[📖 Read article](!!!URL!!!)
""".trimIndent()

Obviously, text starting and ending in !!! will be replaced.

val counter = 1
rssFeed.forEach {
  val title = it.title.get()
    .replace("#", "#")
    .replace(":", ":")
  val url = it.link.get()
  val date: String
  val fullDateAndTime: String
  val year: Int
  it.pubDateZonedDateTime.get().run {
    fullDateAndTime = toString()
    date = toLocalDate().toString()
    year = this.year
  }
  val dir = File("$baseDir${File.separatorChar}$year")
  dir.mkdirs()
  val file = File(dir, "${date}-${counter}.md")
  val fileContent = template.replace("!!!TITLE!!!", title)
    .replace("!!!FULL_DATE_AND_TIME!!!", fullDateAndTime)
    .replace("!!!DATE!!!", date)
    .replace("!!!URL!!!", url)
  file.writeText(fileContent)
}

So, once the values have been obtained, I simply invoke replace(). title gets a special treatment as the title is processed somewhat differently inside Jekyll, so characters like # and : must be escaped.

The markdown files are saved in folders named like the year of publication. ...just like I said, tailored to my needs. 😎

While the code for the loop is complete you are surely wondering where the feed data comes from... what this rssFeed thing is.

fun main(args: Array<String>) {
  val baseDir = if (args.isNotEmpty()) args[0] else "./_posts"
  val client = HttpClient.newBuilder().version(HttpClient.Version.HTTP_1_1).build()
  val request = HttpRequest.newBuilder()
    .uri(URI.create("https://dev.to/feed/tkuenneth/"))
    .header(
      "User-Agent",
      "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.95 Safari/537.11"
    )
    .build()
  val response = client.send(request, BodyHandlers.ofString())
  if (response.statusCode() == 200) {
    val inputStream = response.body().byteInputStream()
    val reader = RssReader()
    val rssFeed = reader.read(inputStream)

I am using Java's HttpClient. The code is straight forward, just be sure to change the url accordingly.

One thing that caused me a tiny headache...: while doing my first experiments I always got a 403. Using some C# code did work, however. And firing up the url in the browser worked, too. It turns out that to be succesful I needed to change User-Agent. Now, while one could argue the url should not be called from an app, feeds generally are processed by feed readers and the likes, aren't they?

To parse the fetched data I use rssreader by Peter Westling. The library is easy to include using Maven or Gradle. And it is published under the MIT license.

This applies to my tiny project, too. You can find it on GitHub.