tflookup - Developer Diary Part 2

Dear Diary!

Hmmm... A lookup app. How could that work?

• I need a frontend obviously. And a backend, that does the heavy work.
• Frontend and backend should communicate through REST. What else?
• The backend should index the Terraform documentation
• Everything should be well tested of course

First things first. The absolute basis for the whole project is the documentation index. Without that, I could trash the whole thing.

So, how is the terraform documentation built actually?

The Terraform developers do that using a massive repository with multiple submodules for all providers:

hashicorp / terraform-website

Build configuration and partial content for terraform.io

Terraform Website

This repository contains the build infrastructure and some of the content for terraform.io. Pull requests from the community are welcomed!

Table of Contents

• How the Site Works
• Where the Docs Live
• Deploying Changes to terraform.io
• Running the Site Locally
• Previewing Changes from Terraform Core
• Writing Normal Docs Content
• Screenshots
• Navigation Sidebars
• Using Submodules
• Finding Broken Links
• More about stable-website

How the Site Works

↥ back to top

terraform.io is in transition at the moment, and the production site is kind of hybrid:

• Fastly handles all the traffic.

• The following paths (all marketing content, at the moment) are proxied to a Next.js app running on Vercel:

  • / (the front page)
  • /community
  • /cloud (and all sub-paths under /cloud)

  For help with these pages, talk to the Web Platform team!

• The rest of the site falls through to static pages built with Middleman. That's what this repository manages!

…

View on GitHub

They write all documentation in Markdown with Frontmatter included and cramp everything together with Middleman

The Frontmatter includes titles, layouts, description and the information where the documentation lives in the documentation tree on the website.

So that's what I need.

I "submoduled" the terraform-website repository as well and wrote an Indexer, that walks through every datasource and resource documentation available for all providers in the terraform website.

That worked quite well (aside from some minor errors)[https://github.com/dploeger/tflookup/blob/master/indexErrors.txt], where people forgot how to write proper Frontmatter.

However, I needed to make some assumptions:

• Every vendor has a "website/docs" subfolder where its documentation lives
• There's only a datasource or a resource
• The datasources are living in a "d" directory, the resources in an "r" directory
• The titles are designed as ": "

I didn't go through all documentation files, but the documentation seems to be quite structured like this.

Using all this information, I could create my documentation index, which included the title, the name, wether it's a datasource or a resource, the description and a link to the real documentation page.

I included ways to store the indexed documentation, so the future frontend server wouldn't need to initialize all the required submodules for that. I also wrote a script which updates the documentation every night, which is running on my private server.

Okay, index done.

Yours
Dennis

This post is one of five posts from the tflookup developer diary series

Cover Image: "diary writing" by Fredrik Rubensson