Markdown files as static content

#ruby #rails #octokit #staticsite

How to use markdown files from a GitHub repository as static content in your website?

Context

We will be talking about a feature implemented on Agile Ventures that allow us to use markdown files as static content of the website. This idea was born when thinking about how to let developers contribute to the content of the site in an easier way while taking advantage of the GitHub ecosystem.

What we were using for editing the static content on the website was the mercury editor. But the problem with that editor is that has fallen out of maintenance. I have done a couple changes to that gem in order to make it work with rails 5. We are now using a custom version in a forked repository, but that is another story ;).

As the main goal, we want to get rid of that editor from our system. So, we have started from removing the functionality that involves static pages, for, later on, remove it from the projects as well. All the code related to the changes I have done in order to implement this feature can be found here.

Preparations

We need the repository from where we want to pull the content. Currently, our static pages live in this repository and we are pulling data from the root path, and working only with the markdown files.

The application is making requests to the GitHub API by using the octokit gem. So, we need to have the octokit gem in our Gemfile:

gem 'octokit', '~\> 4.8'

And the kramdown gem is being used for converting the markdown content into HTML. For adding the kramdown gem to the Gemfile:

gem 'kramdown', '~\> 1.16', '\>= 1.16.2'

Creating the service

As we will be using the kramdown gem for converting the content in a markdown file into HTML, we need to isolate the gem behavior. For doing this, we are going to create a class that will serve as a wrapper for the kramdown functionality.

	class MarkdownConverter
	attr_reader :markdown
	private :markdown

	def initialize(markdown)
	@markdown = markdown
	end

	def as_html
	converted_markdown.html_safe
	end

	private

	def converted_markdown
	Kramdown::Document.new(markdown).to_html
	end
	end

view raw markdown_converter.rb hosted with ❤ by GitHub

The path to changeable and maintainable object-oriented software begins with classes that have a single responsibility. Classes that do one thing isolate that thing from the rest of your application. This isolation allows change without consequence and reuses without duplication.

Sandi Metz.

Crafting the rake task

Now we need to create the rake task that is gonna be scheduled by using the Heroku Scheduler add-on. It will look something like this:

I have created this rake task by following the code styles that the project is already implementing in order to preserve consistency within the system. So this rake task will call the logic that resides in the GithubStaticPages module.

	namespace :fetch_github do
	desc 'Get github content for static pages'
	task :content_for_static_pages => :environment do
	GithubStaticPagesJob.run
	end
	end

view raw github_content_for_static_pages.rake hosted with ❤ by GitHub

The following code is the logic that we are using for pulling the content from markdown files, convert it to HTML and save it using the StaticPages model. This means that all HTML code related to a particular static content lives in the database.

	require 'octokit'
	require 'base64'

	module GithubStaticPagesJob
	extend self

	FILE_NAME_REGEX = /\w+[^\.md]/

	def client
	credentials = {
	client_id: Settings.github.client_id,
	client_secret: Settings.github.client_secret
	}
	@client \|\|= Octokit::Client.new(credentials)
	end

	def run
	begin
	content = get_content('agileventures/agileventures')
	process_markdown_pages(get_markdown_pages(content))
	rescue StandardError => e
	ErrorLoggingService.new(e).log("Trying to get the content from the repository may have caused the issue!")
	end
	end

	private

	def get_content(repository)
	client.contents(repository)
	end

	def get_markdown_pages(content)
	content.select{\|page\| page if page[:path] =~ /\.md$/i}
	end

	def process_markdown_pages(md_pages)
	md_pages.each do \|page\|
	begin
	filename = page[:path]
	page_content = client.contents('agileventures/agileventures', path: filename)
	markdown = Base64.decode64(page_content[:content])
	static_page = StaticPage.find_by_slug(get_slug(filename))
	static_page.nil? ? create_static_page(filename, markdown) : update_body(static_page, markdown)
	rescue Encoding::UndefinedConversionError => e
	ErrorLoggingService.new(e).log("Trying to convert this page: #{filename} caused the issue!")
	end
	end
	end

	def create_static_page(filename, markdown)
	StaticPage.create(
	title: get_title(filename),
	body: convert_markdown_to_html(markdown),
	slug: get_slug(filename)
	)
	end

	def update_body(static_page, markdown)
	static_page.body = convert_markdown_to_html(markdown)
	static_page.save!
	end

	def get_title(filename)
	get_slug(filename).tr("-", " ").titleize
	end

	def get_slug(filename)
	FILE_NAME_REGEX.match(filename)[0].downcase.tr("_", "-")
	end

	def convert_markdown_to_html(markdown)
	MarkdownConverter.new(markdown).as_html
	end
	end

view raw github_static_pages_job.rb hosted with ❤ by GitHub

Rendering the content

Since we have to change the way we CRUD static pages on the system, and we now will be using the GitHub ecosystem for doing it, there is no change to be implemented in the way we get the content from the database. The only difference is that now we have been able to remove the actions that were tightly coupled to the mercury editor.

	def show
	return false if redirect_email_blunder
	@page = StaticPage.friendly.find(get_page_id(params[:id]))
	@ancestry = @page.self_and_ancestors.map(&:title).reverse
	end

view raw static_pages_controller.rb hosted with ❤ by GitHub

Finally, we just get the HTML from the body attribute in the page object.

view raw show.html.erb hosted with ❤ by GitHub

What's next?

Instead of having this rake task running daily within our system, we could be using GitHub Webhooks for achieving the same goal. But for doing this, we need first to create the functionality within our system that will handle those requests.

Talking about priorities, I will keep working on the removal of the mercury editor from the system since that is the general goal. Once removed, with the AV (Agile Ventures) team, we will decide what to do next. I'll keep you posted;)!

Notes

Something that it worth to mention is that this feature only creates and updates the static pages on the database. It doesn't remove pages that no longer exist in the repository from where we pull the content. This was a decision that we discussed within the team to be implemented in this way.

All code shared here is Open Source and you will find it in the WebsiteOne repository. If you want to contribute to the code, even if it is a small change where you are thinking "this could have been done in this other way". I invite you to create a pull request. It doesn't matter how small the change is, it will help us to improve and make our code better, while you will be contributing to an Open Source project ;).