Attempting to Learn Go - Ghost to Hugo 2

Intro

Welcome back! We are continuing on our journey to make a prototype program that converts an exported Ghost database to Markdown. With the end goal being that we can get shindakun.net up and running with Hugo. Last time, we took it pretty easy and focused mostly on reading the file into memory and converting the JSON to a Go struct. From there we printed out the first post.

Post Data

As a recap here is what one of the post fields contain.

{
    "id": "60710b90705967038fe662d6",
    "uuid": "71ba3d71-ac18-4f33-82f7-1962baa83a07",
    "title": "db test",
    "slug": "db-test",
    "mobiledoc": "{\"version\":\"0.3.1\",\"markups\":[],\"atoms\":[],\"cards\":[[\"markdown\",{\"cardName\":\"card-markdown\",\"markdown\":\"\\n<strike>This is a db test</strike>.\\n\"}]],\"sections\":[[10,0]],\"ghostVersion\":\"3.0\"}",
    "html": "<!--kg-card-begin: markdown--><p><strike>This is a db test</strike>.</p>\n<!--kg-card-end: markdown-->",
    "comment_id": "2",
    "plaintext": "This is a db test.",
    "feature_image": null,
    "featured": 0,
    "type": "post",
    "status": "published",
    "locale": null,
    "visibility": "public",
    "email_recipient_filter": "none",
    "author_id": "60710b8d705967038fe66214",
    "created_at": "2004-08-09T19:11:20.000Z",
    "updated_at": "2004-08-09T19:11:20.000Z",
    "published_at": "2004-08-09T19:11:20.000Z",
    "custom_excerpt": null,
    "codeinjection_head": null,
    "codeinjection_foot": null,
    "custom_template": null,
    "canonical_url": null
},

The majority of the fields we need to craft a post with appropriate frontmatter exist within this the object we’re getting back. We can see the title, slug, published date, etc. The section that contains the Markdown is in a format known as Mobiledoc.

Mobiledoc

According to the Ghost documentation Mobiledoc is

…a standardised JSON-based document storage format, which forms the heart of publishing with Ghost.

When extracted from the JSON object and cleaned up we’ll have another bit of JSON we can work with.

{
    "version": "0.3.1",
    "markups": [],
    "atoms": [],
    "cards": [
        [
            "markdown",
            {
                "cardName": "card-markdown",
                "markdown": "\n<strike>This is a db test</strike>.\n"
            }
        ]
    ],
    "sections": [
        [
            10,
            0
        ]
    ],
    "ghostVersion": "3.0"
}

Sounds Easy

First we’ll use our favorite site, the JSON to Go converter to convert the JSON object to a struct we can work with.

type Mobiledoc struct {
    Version string `json:"version"`
    Markups []interface{} `json:"markups"`
    Atoms []interface{} `json:"atoms"`
    Cards [][]interface{} `json:"cards"`
    Sections [][]int `json:"sections"`
    GhostVersion string `json:"ghostVersion"`
}

Our code is pretty long now so I’m going to leave out the other struct GhostDatabase struct, it’ll be in the complete code listing below though. We’re still going to be dumping code to the screen since we’re still working on our decoding logic.

package main

import (
    "encoding/json"
    "fmt"
    "io"
    "os"
    "strconv"
    "time"
)

type GhostDatabase struct {...}

type Mobiledoc struct {
    Version string `json:"version"`
    Markups []interface{} `json:"markups"`
    Atoms []interface{} `json:"atoms"`
    Cards [][]interface{} `json:"cards"`
    Sections [][]int `json:"sections"`
    GhostVersion string `json:"ghostVersion"`
}

func main() {
    fmt.Println("ghost2hugo")

    file, err := os.Open("shindakun-dot-net.ghost.2022-03-18-22-02-58.json")
    if err != nil {
        fmt.Println(err)
    }

    defer file.Close()

    b, err := io.ReadAll(file)
    if err != nil {
        fmt.Println(err)
    }

    var db GhostDatabase

    err = json.Unmarshal(b, &db)
    if err != nil {
        fmt.Println(err)
    }

Let’s continue to focus on the first post for now since once we have that working it should just be a matter of looping through the “database”. This is where it gets a little tricky. We’re working with a couple of nested arrays so to work our way down to the appropriate section we use db.Db[0].Data.Posts[0].Mobiledoc. This will give use the escaped version of our JSON object.

"{\"version\":\"0.3.1\",\"markups\":[],\"atoms\":[],\"cards\":[[\"markdown\",{\"cardName\":\"card-markdown\",\"markdown\":\"\\n<strike>This is a db test</strike>.\\n\"}]],\"sections\":[[10,0]],\"ghostVersion\":\"3.0\"}"

Let’s Go On A Trip

I knew that there would be a way to unescape the string that we get, checking the Go documentation led me to strconv.Unquote.

Unquote interprets s as a single-quoted, double-quoted, or backquoted Go string literal, returning the string value that s quotes. (If s is single-quoted, it would be a Go character literal; Unquote returns the corresponding one-character string.)

Exactly what we need! Except that when I tried to unquote the string I kept receiving an invalid syntax error. This had me confused for a little bit. After puzzling over it I realized if I prepended and appended a back tick to the string it seems to be treated as a raw string literal. This leads to code that looks like the following.

    c := "`" + db.Db[0].Data.Posts[0].Mobiledoc + "`"

    un, err := strconv.Unquote(c)
    if err != nil {
        fmt.Println(err)
    }
    fmt.Printf("%v\n", un)

Finally! We have the JSON!

{"version":"0.3.1","markups":[],"atoms":[],"cards":[["markdown",{"cardName":"card-markdown","markdown":"\n<strike>This is a db test</strike>.\n"}]],"sections":[[10,0]],"ghostVersion":"3.0"}

Now, we can unmarshal that into the Mobiledoc struct we set up earlier!

    var md Mobiledoc

    err = json.Unmarshal([]byte(un), &md)
    if err != nil {
        fmt.Println(err)
    }

    fmt.Printf("%#v", md)

Checking our dumped result we can see that we’ve got the expected data.

main.Mobiledoc{Version:"0.3.1", Markups:[]interface {}{}, Atoms:[]interface {}{}, Cards:[][]interface {}{[]interface {}{"markdown", map[string]interface {}{"cardName":"card-markdown", "markdown":"\n<strike>This is a db test</strike>.\n"}}}, Sections:[][]int{[]int{10, 0}}, GhostVersion:"3.0"}

Right now we’re concerned with the section called Cards which if we naively want to access we can use the following.

    card := md.Cards[0][1]
    fmt.Printf("\n\ncard: %#v\n", card)

card: map[string]interface {}{"cardName":"card-markdown", "markdown":"\n<strike>This is a db test</strike>.\n"}

Let’s convert that to something a bit easier to access.

    bbb := card.(map[string]interface{})
    fmt.Println(bbb["markdown"])

<strike>This is a db test</strike>.

Next Time

So far so good, we’ve extracted the Markdown from the first post as expected. Next time around we’ll be writing a small loop to start trying to extract all the posts. I have a feeling that’s where the fun will begin.

---

Enjoy this post?
How about buying me a coffee?

---

shindakun / atlg

Source repo for the "Attempting to Learn Go" posts I've been putting up over on dev.to

Attempting to Learn Go

Here you can find the code I've been writing for my Attempting to Learn Go posts that I've been writing and posting over on Dev.to.

Post Index

Post	Code
Building a Downloader Part 01	-
Building a Downloader Part 02	-
Building a Downloader Part 03	-
Building a Downloader Part 04	-
Building a Downloader Part 05	-
Consuming a REST API	src
Continuing REST Adventures	src
Now Sending REST Requests	src
REST API and A Bit On Templates	src
Sending Email Via API Again	src
Let's Get Modular!	src
Let's Get Modular - Again!	src
Building DevLog Part 01	src
Building DevLog Part 02	src
Building DevLog Part 03	src
Building DevLog Part 04	src
Building DevLog Part 05	src
Listing Files By Extension 01	src
Listing Files By Extension 02	src
Dev.to API 01	src
Dev.to API 02	see above code
Issuer

…

View on GitHub

Code Listing

package main

import (
    "encoding/json"
    "fmt"
    "io"
    "os"
    "strconv"
    "time"
)

type GhostDatabase struct {
    Db []struct {
        Meta struct {
            ExportedOn int64 `json:"exported_on"`
            Version string `json:"version"`
        } `json:"meta"`
        Data struct {
            Posts []struct {
                ID string `json:"id"`
                UUID string `json:"uuid"`
                Title string `json:"title"`
                Slug string `json:"slug"`
                Mobiledoc string `json:"mobiledoc"`
                HTML string `json:"html"`
                CommentID string `json:"comment_id"`
                Plaintext string `json:"plaintext"`
                FeatureImage interface{} `json:"feature_image"`
                Featured int `json:"featured"`
                Type string `json:"type"`
                Status string `json:"status"`
                Locale interface{} `json:"locale"`
                Visibility string `json:"visibility"`
                EmailRecipientFilter string `json:"email_recipient_filter"`
                AuthorID string `json:"author_id"`
                CreatedAt time.Time `json:"created_at"`
                UpdatedAt time.Time `json:"updated_at"`
                PublishedAt time.Time `json:"published_at"`
                CustomExcerpt interface{} `json:"custom_excerpt"`
                CodeinjectionHead interface{} `json:"codeinjection_head"`
                CodeinjectionFoot interface{} `json:"codeinjection_foot"`
                CustomTemplate interface{} `json:"custom_template"`
                CanonicalURL interface{} `json:"canonical_url"`
            } `json:"posts"`
            PostsAuthors []struct {
                ID string `json:"id"`
                PostID string `json:"post_id"`
                AuthorID string `json:"author_id"`
                SortOrder int `json:"sort_order"`
            } `json:"posts_authors"`
            PostsMeta []interface{} `json:"posts_meta"`
            PostsTags []struct {
                ID string `json:"id"`
                PostID string `json:"post_id"`
                TagID string `json:"tag_id"`
                SortOrder int `json:"sort_order"`
            } `json:"posts_tags"`
            Roles []struct {
                ID string `json:"id"`
                Name string `json:"name"`
                Description string `json:"description"`
                CreatedAt time.Time `json:"created_at"`
                UpdatedAt time.Time `json:"updated_at"`
            } `json:"roles"`
            RolesUsers []struct {
                ID string `json:"id"`
                RoleID string `json:"role_id"`
                UserID string `json:"user_id"`
            } `json:"roles_users"`
            Settings []struct {
                ID string `json:"id"`
                Group string `json:"group"`
                Key string `json:"key"`
                Value string `json:"value"`
                Type string `json:"type"`
                Flags interface{} `json:"flags"`
                CreatedAt time.Time `json:"created_at"`
                UpdatedAt time.Time `json:"updated_at"`
            } `json:"settings"`
            Tags []struct {
                ID string `json:"id"`
                Name string `json:"name"`
                Slug string `json:"slug"`
                Description interface{} `json:"description"`
                FeatureImage interface{} `json:"feature_image"`
                ParentID interface{} `json:"parent_id"`
                Visibility string `json:"visibility"`
                OgImage interface{} `json:"og_image"`
                OgTitle interface{} `json:"og_title"`
                OgDescription interface{} `json:"og_description"`
                TwitterImage interface{} `json:"twitter_image"`
                TwitterTitle interface{} `json:"twitter_title"`
                TwitterDescription interface{} `json:"twitter_description"`
                MetaTitle interface{} `json:"meta_title"`
                MetaDescription interface{} `json:"meta_description"`
                CodeinjectionHead interface{} `json:"codeinjection_head"`
                CodeinjectionFoot interface{} `json:"codeinjection_foot"`
                CanonicalURL interface{} `json:"canonical_url"`
                AccentColor interface{} `json:"accent_color"`
                CreatedAt time.Time `json:"created_at"`
                UpdatedAt time.Time `json:"updated_at"`
            } `json:"tags"`
            Users []struct {
                ID string `json:"id"`
                Name string `json:"name"`
                Slug string `json:"slug"`
                Password string `json:"password"`
                Email string `json:"email"`
                ProfileImage string `json:"profile_image"`
                CoverImage interface{} `json:"cover_image"`
                Bio interface{} `json:"bio"`
                Website interface{} `json:"website"`
                Location interface{} `json:"location"`
                Facebook interface{} `json:"facebook"`
                Twitter interface{} `json:"twitter"`
                Accessibility string `json:"accessibility"`
                Status string `json:"status"`
                Locale interface{} `json:"locale"`
                Visibility string `json:"visibility"`
                MetaTitle interface{} `json:"meta_title"`
                MetaDescription interface{} `json:"meta_description"`
                Tour interface{} `json:"tour"`
                LastSeen time.Time `json:"last_seen"`
                CreatedAt time.Time `json:"created_at"`
                UpdatedAt time.Time `json:"updated_at"`
            } `json:"users"`
        } `json:"data"`
    } `json:"db"`
}

type Mobiledoc struct {
    Version string `json:"version"`
    Markups []interface{} `json:"markups"`
    Atoms []interface{} `json:"atoms"`
    Cards [][]interface{} `json:"cards"`
    Sections [][]int `json:"sections"`
    GhostVersion string `json:"ghostVersion"`
}

func main() {
    fmt.Println("ghost2hugo")

    file, err := os.Open("shindakun-dot-net.ghost.2022-03-18-22-02-58.json")
    if err != nil {
        fmt.Println(err)
    }

    defer file.Close()

    b, err := io.ReadAll(file)
    if err != nil {
        fmt.Println(err)
    }

    var db GhostDatabase

    err = json.Unmarshal(b, &db)
    if err != nil {
        fmt.Println(err)
    }

    c := "`" + db.Db[0].Data.Posts[0].Mobiledoc + "`"

    un, err := strconv.Unquote(c)
    if err != nil {
        fmt.Println(err)
    }
    fmt.Printf("%v", un)

    var md Mobiledoc

    err = json.Unmarshal([]byte(un), &md)
    if err != nil {
        fmt.Println(err)
    }

    fmt.Printf("%#v", md)

    card := md.Cards[0][1]
    fmt.Printf("\n\ncard: %#v\n", card)

    bbb := card.(map[string]interface{})
    fmt.Println(bbb["markdown"])
}