DEV Community: Steve Layton

Attempting to Learn Go - Ghost to Hugo 4

Steve Layton — Sat, 01 Oct 2022 20:54:29 +0000

Intro

Here we are once again! If you missed the last couple of posts I suggest reading them first to get a feeling for how our little prototype has evolved. This time around we’re going to be doing some refactoring to clean up a bit. The majority of what we’ll be doing is adding methods to our “database” struct.

The Method

What is a method? Well, you can think of it as a function that is attached to a struct. Or as A Tour of Go puts it

A method is a function with a special receiver argument.

In our code we have a struct GhostDatabase. We can add methods with the following signature where the receiver is our struct.

func (gd *GhostDatabase) getLength() int

We could then call the method with the following

var db GhostDatabase

...load JSON code here...

length := db.getLength()

OK so what’s our complete method look like? In the case of most of the methods we’ll be putting in place is pretty easy to reason about.

func (gd *GhostDatabase) getLength() int {
    return len(gd.Db[0].Data.Posts)
}

Many Methods

Now lets quickly knock out some useful methods. For the most part they do what they say on the tin, so I won’t go into detail on each one.

func (gd *GhostDatabase) getMobiledoc(i int) string {
    return gd.Db[0].Data.Posts[i].Mobiledoc
}

func (gd *GhostDatabase) getPostId(i int) string {
    return gd.Db[0].Data.Posts[i].ID
}

func (gd *GhostDatabase) getPostTitle(i int) string {
    return gd.Db[0].Data.Posts[i].Title
}

func (gd *GhostDatabase) getPostSlug(i int) string {
    return gd.Db[0].Data.Posts[i].Slug
}

func (gd *GhostDatabase) getPostStatus(i int) string {
    return gd.Db[0].Data.Posts[i].Status
}

func (gd *GhostDatabase) getPostCreatedAt(i int) time.Time {
    return gd.Db[0].Data.Posts[i].CreatedAt
}

func (gd *GhostDatabase) getPostUpdatedAt(i int) time.Time {
    return gd.Db[0].Data.Posts[i].UpdatedAt
}

func (gd *GhostDatabase) getPostPublishedAt(i int) time.Time {
    return gd.Db[0].Data.Posts[i].PublishedAt
}

func (gd *GhostDatabase) getPostFeatureImage(i int) string {
    return gd.Db[0].Data.Posts[i].FeatureImage
}

Next Time

That’s all of our methods for now. With them out of the way I believe we have all the parts need to actually start writing our Markdown files!

Enjoy this post?
How about buying me a coffee?

shindakun / atlg

Source repo for the "Attempting to Learn Go" posts I've been putting up over on dev.to

Attempting to Learn Go

Here you can find the code I've been writing for my Attempting to Learn Go posts that I've been writing and posting over on Dev.to.

Post Index

Post	Code
Building a Downloader Part 01	-
Building a Downloader Part 02	-
Building a Downloader Part 03	-
Building a Downloader Part 04	-
Building a Downloader Part 05	-
Consuming a REST API	src
Continuing REST Adventures	src
Now Sending REST Requests	src
REST API and A Bit On Templates	src
Sending Email Via API Again	src
Let's Get Modular!	src
Let's Get Modular - Again!	src
Building DevLog Part 01	src
Building DevLog Part 02	src
Building DevLog Part 03	src
Building DevLog Part 04	src
Building DevLog Part 05	src
Listing Files By Extension 01	src
Listing Files By Extension 02	src
Dev.to API 01	src
Dev.to API 02	see above code
Issuer

…

View on GitHub

Code Listing

package main

import (
    "encoding/json"
    "fmt"
    "io"
    "os"
    "reflect"
    "strconv"
    "strings"
    "time"
)

type GhostDatabase struct {
    Db []struct {
        Meta struct {
            ExportedOn int64 `json:"exported_on"`
            Version string `json:"version"`
        } `json:"meta"`
        Data struct {
            Posts []struct {
                ID string `json:"id"`
                UUID string `json:"uuid"`
                Title string `json:"title"`
                Slug string `json:"slug"`
                Mobiledoc string `json:"mobiledoc"`
                HTML string `json:"html"`
                CommentID string `json:"comment_id"`
                Plaintext string `json:"plaintext"`
                FeatureImage string `json:"feature_image"`
                Featured int `json:"featured"`
                Type string `json:"type"`
                Status string `json:"status"`
                Locale interface{} `json:"locale"`
                Visibility string `json:"visibility"`
                EmailRecipientFilter string `json:"email_recipient_filter"`
                AuthorID string `json:"author_id"`
                CreatedAt time.Time `json:"created_at"`
                UpdatedAt time.Time `json:"updated_at"`
                PublishedAt time.Time `json:"published_at"`
                CustomExcerpt interface{} `json:"custom_excerpt"`
                CodeinjectionHead interface{} `json:"codeinjection_head"`
                CodeinjectionFoot interface{} `json:"codeinjection_foot"`
                CustomTemplate interface{} `json:"custom_template"`
                CanonicalURL interface{} `json:"canonical_url"`
            } `json:"posts"`
            PostsAuthors []struct {
                ID string `json:"id"`
                PostID string `json:"post_id"`
                AuthorID string `json:"author_id"`
                SortOrder int `json:"sort_order"`
            } `json:"posts_authors"`
            PostsMeta []interface{} `json:"posts_meta"`
            PostsTags []struct {
                ID string `json:"id"`
                PostID string `json:"post_id"`
                TagID string `json:"tag_id"`
                SortOrder int `json:"sort_order"`
            } `json:"posts_tags"`
            Roles []struct {
                ID string `json:"id"`
                Name string `json:"name"`
                Description string `json:"description"`
                CreatedAt time.Time `json:"created_at"`
                UpdatedAt time.Time `json:"updated_at"`
            } `json:"roles"`
            RolesUsers []struct {
                ID string `json:"id"`
                RoleID string `json:"role_id"`
                UserID string `json:"user_id"`
            } `json:"roles_users"`
            Settings []struct {
                ID string `json:"id"`
                Group string `json:"group"`
                Key string `json:"key"`
                Value string `json:"value"`
                Type string `json:"type"`
                Flags interface{} `json:"flags"`
                CreatedAt time.Time `json:"created_at"`
                UpdatedAt time.Time `json:"updated_at"`
            } `json:"settings"`
            Tags []struct {
                ID string `json:"id"`
                Name string `json:"name"`
                Slug string `json:"slug"`
                Description interface{} `json:"description"`
                FeatureImage interface{} `json:"feature_image"`
                ParentID interface{} `json:"parent_id"`
                Visibility string `json:"visibility"`
                OgImage interface{} `json:"og_image"`
                OgTitle interface{} `json:"og_title"`
                OgDescription interface{} `json:"og_description"`
                TwitterImage interface{} `json:"twitter_image"`
                TwitterTitle interface{} `json:"twitter_title"`
                TwitterDescription interface{} `json:"twitter_description"`
                MetaTitle interface{} `json:"meta_title"`
                MetaDescription interface{} `json:"meta_description"`
                CodeinjectionHead interface{} `json:"codeinjection_head"`
                CodeinjectionFoot interface{} `json:"codeinjection_foot"`
                CanonicalURL interface{} `json:"canonical_url"`
                AccentColor interface{} `json:"accent_color"`
                CreatedAt time.Time `json:"created_at"`
                UpdatedAt time.Time `json:"updated_at"`
            } `json:"tags"`
            Users []struct {
                ID string `json:"id"`
                Name string `json:"name"`
                Slug string `json:"slug"`
                Password string `json:"password"`
                Email string `json:"email"`
                ProfileImage string `json:"profile_image"`
                CoverImage interface{} `json:"cover_image"`
                Bio interface{} `json:"bio"`
                Website interface{} `json:"website"`
                Location interface{} `json:"location"`
                Facebook interface{} `json:"facebook"`
                Twitter interface{} `json:"twitter"`
                Accessibility string `json:"accessibility"`
                Status string `json:"status"`
                Locale interface{} `json:"locale"`
                Visibility string `json:"visibility"`
                MetaTitle interface{} `json:"meta_title"`
                MetaDescription interface{} `json:"meta_description"`
                Tour interface{} `json:"tour"`
                LastSeen time.Time `json:"last_seen"`
                CreatedAt time.Time `json:"created_at"`
                UpdatedAt time.Time `json:"updated_at"`
            } `json:"users"`
        } `json:"data"`
    } `json:"db"`
}

type Mobiledoc struct {
    Version string `json:"version"`
    Markups []interface{} `json:"markups"`
    Atoms []interface{} `json:"atoms"`
    Cards [][]interface{} `json:"cards"`
    Sections [][]interface{} `json:"sections"`
    GhostVersion string `json:"ghostVersion"`
}

func (gd *GhostDatabase) getLength() int {
    return len(gd.Db[0].Data.Posts)
}

func (gd *GhostDatabase) getMobiledoc(i int) string {
    return gd.Db[0].Data.Posts[i].Mobiledoc
}

func (gd *GhostDatabase) getPostId(i int) string {
    return gd.Db[0].Data.Posts[i].ID
}

func (gd *GhostDatabase) getPostTitle(i int) string {
    return gd.Db[0].Data.Posts[i].Title
}

func (gd *GhostDatabase) getPostSlug(i int) string {
    return gd.Db[0].Data.Posts[i].Slug
}

func (gd *GhostDatabase) getPostStatus(i int) string {
    return gd.Db[0].Data.Posts[i].Status
}

func (gd *GhostDatabase) getPostCreatedAt(i int) time.Time {
    return gd.Db[0].Data.Posts[i].CreatedAt
}

func (gd *GhostDatabase) getPostUpdatedAt(i int) time.Time {
    return gd.Db[0].Data.Posts[i].UpdatedAt
}

func (gd *GhostDatabase) getPostPublishedAt(i int) time.Time {
    return gd.Db[0].Data.Posts[i].PublishedAt
}

func (gd *GhostDatabase) getPostFeatureImage(i int) string {
    return gd.Db[0].Data.Posts[i].FeatureImage
}

func (gd *GhostDatabase) getTags(pid string) []string {
    var r []string
    for i := 0; i < len(gd.Db[0].Data.PostsTags); i++ {
        if gd.Db[0].Data.PostsTags[i].PostID == pid {
            tagId := gd.Db[0].Data.PostsTags[i].TagID
            for j := 0; j < len(gd.Db[0].Data.Tags); j++ {
                if gd.Db[0].Data.Tags[j].ID == tagId {
                    r = append(r, gd.Db[0].Data.Tags[j].Name)
                }
            }
        }
    }
    return r
}

func main() {
    fmt.Println("ghost2hugo")

    file, err := os.Open("shindakun-dot-net.ghost.2022-03-18-22-02-58.json")
    if err != nil {
        fmt.Println(err)
    }

    defer file.Close()

    b, err := io.ReadAll(file)
    if err != nil {
        fmt.Println(err)
    }

    var db GhostDatabase

    err = json.Unmarshal(b, &db)
    if err != nil {
        fmt.Println(err)
    }

    for i := 0; i < db.getLength(); i++ {
        fmt.Println(db.getPostTitle(i))
        fmt.Println(db.getPostSlug(i))
        fmt.Println(db.getPostStatus(i))
        fmt.Println(db.getPostCreatedAt(i))
        fmt.Println(db.getPostUpdatedAt(i))
        fmt.Println(db.getPostPublishedAt(i))
        fmt.Println(db.getPostFeatureImage(i))

        id := db.getPostId(i)
        tags := db.getTags(id)
        fmt.Println(tags)

        c := strings.ReplaceAll(db.getMobiledoc(i), "`", "%'")
        cc := "`" + c + "`"

        ucn, err := strconv.Unquote(cc)
        if err != nil {
            fmt.Println(err, ucn)
        }

        ucn = strings.ReplaceAll(ucn, "%'", "`")

        var md Mobiledoc

        err = json.Unmarshal([]byte(ucn), &md)
        if err != nil {
            fmt.Println(err)
        }

        if reflect.ValueOf(md.Cards).Len() > 0 {
            card := md.Cards[0][1]
            bbb := card.(map[string]interface{})

            fmt.Println(bbb["markdown"])
        }

        fmt.Println("---")
    }
}

Attempting to Learn Go - Ghost to Hugo 3

Steve Layton — Wed, 28 Sep 2022 00:00:05 +0000

Intro

We’re back once again to continue our work on the ghost2hugo prototype. So far we can open the JSON file, load the data into memory, and print out the Markdown for the first post. The next step is to make sure we can read and process every post included in the backup.

Looping

We’re going to replace all the code that prints a single article with a loop that prints out each article. For a refresher, here is the code that prints the first article.

    c := "`" + db.Db[0].Data.Posts[0].Mobiledoc + "`"

    un, err := strconv.Unquote(c)
    if err != nil {
        fmt.Println(err)
    }
    fmt.Printf("%v", un)

    var md Mobiledoc

    err = json.Unmarshal([]byte(un), &md)
    if err != nil {
        fmt.Println(err)
    }

    fmt.Printf("%#v", md)

    card := md.Cards[0][1]
    fmt.Printf("\n\ncard: %#v\n", card)

    bbb := card.(map[string]interface{})
    fmt.Println(bbb["markdown"])

Most of this will live inside of our new loop.

    for i := 0; i < len(db.Db[0].Data.Posts); i++ {
        fmt.Println(db.Db[0].Data.Posts[i].Title)
        fmt.Println(db.Db[0].Data.Posts[i].Slug)
        fmt.Println(db.Db[0].Data.Posts[i].Status)
        fmt.Println(db.Db[0].Data.Posts[i].CreatedAt)
        fmt.Println(db.Db[0].Data.Posts[i].UpdatedAt)

        cc := "`" + db.Db[0].Data.Posts[i].Mobiledoc + "`"

        ucn, err := strconv.Unquote(cc)
        if err != nil {
            fmt.Println(err, ucn)
        }
        fmt.Printf("\n\n\n%v\n\n\n", ucn)

        var md Mobiledoc

        err = json.Unmarshal([]byte(ucn), &md)
        if err != nil {
            fmt.Println(err)
        }

        card := md.Cards[0][1]
        bbb := card.(map[string]interface{})

        fmt.Println(bbb["markdown"])
        }
    }

And when we run the code this time we see a bunch of articles fly by! Until, that is, we hit an error! The invalid syntax error indicates this is a problem with strconv.Unquote().

invalid syntax 

unexpected end of JSON input
panic: runtime error: index out of range [0] with length 0

goroutine 1 [running]:
main.main()
        /Users/steve/Code/ghost2hugo/main.go:214 +0x769
exit status 2

Roadblocks

Well that’s no good, we can’t just have our code crash out like that! Let’s take a closer look at the problem and see if we can work our way through it. OK, now for the master of all debugging techniques! We add a fmt.Println to print out the chunk of text.

        fmt.Println(cc)

Running the code again we crash out as expected but this time we’re presented with the problem text:

`{"version":"0.3.1","markups":[],"atoms":[],"cards":[["markdown",{"cardName":"card-markdown","markdown":"\n* Incresed title to 64 characters. Should be more then enough. \n * ics and other code work over at angstridden dot net tonight.\n\nALTER TABLE `posts` CHANGE `posttitle` `posttitle` VARCHAR( 64 ) NOT NULL\n\n\n"}]],"sections":[[10,0]],"ghostVersion":"3.0"}`

See the problem? Yeah, it looks like there are back ticks in the Markdown that are causing the string convert to not work as expected. To get around this we’re going to do a simple strings.ReplaceAll before the conversion to change the back tick to something else. We can then reverse that process to get the normal text. We’ll replace the back tick with “%'”.

        c := strings.ReplaceAll(db.Db[0].Data.Posts[i].Mobiledoc, "`", "%'")
        cc := "`" + c + "`"

After we “unquote” we can run another strings.ReplaceAll to convert back.

        ucn = strings.ReplaceAll(ucn, "%'", "`")

Running again we run into another problem!

---
Archives
archives-post
published
2007-07-01 09:46:18 +0000 UTC
2007-08-12 20:50:02 +0000 UTC
`{"version":"0.3.1","markups":[],"atoms":[],"cards":[],"sections":[[1,"p",[[0,[],0,""]]]],"ghostVersion":"3.0"}`
panic: runtime error: index out of range [0] with length 0

goroutine 1 [running]:
main.main()
        /Users/steve/Code/ghost2hugo/main.go:214 +0x7a6
exit status 2

Argh!

More Roadblocks

Our naive implementation from the previous post has returned to bite us! We were assuming that cards exist.

        card := md.Cards[0][1]
        bbb := card.(map[string]interface{})

Looking at the returned JSON confirms this.

{"version":"0.3.1","markups":[],"atoms":[],"cards":[],"sections":[[1,"p",[[0,[],0,""]]]],"ghostVersion":"3.0"}

Now, how do we check to see if the cards field is empty?! I noodled on it a little bit and figured the best way would be to use reflect.ValueOf().Len(). This will check the length of md.Cards to ensure we actually have some data.

        if reflect.ValueOf(md.Cards).Len() > 0 {
            card := md.Cards[0][1]
            bbb := card.(map[string]interface{})

            fmt.Println(bbb["markdown"])
        }

On rerunning we get what looks to be all the posts. Except the final draft which looks odd! It turns out this one has no Markdown card. Is this the only one? I certainly hope so since it looks like it may be a pain to convert to Markdown.

Questing
questing
draft
2019-08-23 21:26:44 +0000 UTC
2021-04-12 04:53:26 +0000 UTC
`{"version":"0.3.1","atoms":[],"cards":[["paywall",{}]],"markups":[["a",["href","https://shindakun.dev"]],["a",["href","https://dev.to/shindakun"]]],"sections":[[1,"p",[[0,[],0,"It's been quite sometime since I've posted anything here. I've done a bit of posting over on "],[0,[0],1,"shindakun.dev"],[0,[],0," and over on "],[0,[1],1,"DEV"],[0,[],0," which has been kind of nice. But, those don't really cover gaming at all. But, I updated the site recently and everyone once in a while I use the site to try something out for work and seeing an old post over and over was no good."]]],[1,"p",[[0,[],0,"I only seem to have a small sliver of time for gaming now. With everything else I want/need to get done there is only so much time. Heed this warning - don't get older! Just kidding, it's not so bad."]]],[1,"p",[[0,[],0,"My 5 year old hasn't been introduced into gaming much outside of some basic games she can play on her tablet. I have discovered that she enjoys watching me play the digital version of Warhammer Quest. Which is good since she's going to be getting a crash course in some more difficult board games sooner or later. I have a copy of the Gloomhaven digital board game and the boxed game (and expansion) but haven't actually event opened it yet. Maybe I should fix that this weekend..."]]],[10,0],[1,"p",[]]],"ghostVersion":"3.0"}`
json: cannot unmarshal string into Go struct field Mobiledoc.sections of type int
<nil>

Looks like we also have an issue unmarshaling, on that last line there. This is easy enough to fix for now. We just need to update sections in our Mobiledoc struct to use [][]interface{} and not [][]int.

type Mobiledoc struct {
    Version string `json:"version"`
    Markups []interface{} `json:"markups"`
    Atoms []interface{} `json:"atoms"`
    Cards [][]interface{} `json:"cards"`
    Sections [][]interface{} `json:"sections"`
    GhostVersion string `json:"ghostVersion"`
}

Next Time

We’ve made some good progress over the last few posts. On the first look it seems we’re extracting what we need. I think we’re ready to do a little refactoring, remove our current debug print statements, and get ready for the next part of our converter.

Enjoy this post?
How about buying me a coffee?

shindakun / atlg

Source repo for the "Attempting to Learn Go" posts I've been putting up over on dev.to

Attempting to Learn Go

Here you can find the code I've been writing for my Attempting to Learn Go posts that I've been writing and posting over on Dev.to.

Post Index

Post	Code
Building a Downloader Part 01	-
Building a Downloader Part 02	-
Building a Downloader Part 03	-
Building a Downloader Part 04	-
Building a Downloader Part 05	-
Consuming a REST API	src
Continuing REST Adventures	src
Now Sending REST Requests	src
REST API and A Bit On Templates	src
Sending Email Via API Again	src
Let's Get Modular!	src
Let's Get Modular - Again!	src
Building DevLog Part 01	src
Building DevLog Part 02	src
Building DevLog Part 03	src
Building DevLog Part 04	src
Building DevLog Part 05	src
Listing Files By Extension 01	src
Listing Files By Extension 02	src
Dev.to API 01	src
Dev.to API 02	see above code
Issuer

…

View on GitHub

Code Listing

package main

import (
    "encoding/json"
    "fmt"
    "io"
    "os"
    "reflect"
    "strconv"
    "strings"
    "time"
)

type GhostDatabase struct {
    Db []struct {
        Meta struct {
            ExportedOn int64 `json:"exported_on"`
            Version string `json:"version"`
        } `json:"meta"`
        Data struct {
            Posts []struct {
                ID string `json:"id"`
                UUID string `json:"uuid"`
                Title string `json:"title"`
                Slug string `json:"slug"`
                Mobiledoc string `json:"mobiledoc"`
                HTML string `json:"html"`
                CommentID string `json:"comment_id"`
                Plaintext string `json:"plaintext"`
                FeatureImage interface{} `json:"feature_image"`
                Featured int `json:"featured"`
                Type string `json:"type"`
                Status string `json:"status"`
                Locale interface{} `json:"locale"`
                Visibility string `json:"visibility"`
                EmailRecipientFilter string `json:"email_recipient_filter"`
                AuthorID string `json:"author_id"`
                CreatedAt time.Time `json:"created_at"`
                UpdatedAt time.Time `json:"updated_at"`
                PublishedAt time.Time `json:"published_at"`
                CustomExcerpt interface{} `json:"custom_excerpt"`
                CodeinjectionHead interface{} `json:"codeinjection_head"`
                CodeinjectionFoot interface{} `json:"codeinjection_foot"`
                CustomTemplate interface{} `json:"custom_template"`
                CanonicalURL interface{} `json:"canonical_url"`
            } `json:"posts"`
            PostsAuthors []struct {
                ID string `json:"id"`
                PostID string `json:"post_id"`
                AuthorID string `json:"author_id"`
                SortOrder int `json:"sort_order"`
            } `json:"posts_authors"`
            PostsMeta []interface{} `json:"posts_meta"`
            PostsTags []struct {
                ID string `json:"id"`
                PostID string `json:"post_id"`
                TagID string `json:"tag_id"`
                SortOrder int `json:"sort_order"`
            } `json:"posts_tags"`
            Roles []struct {
                ID string `json:"id"`
                Name string `json:"name"`
                Description string `json:"description"`
                CreatedAt time.Time `json:"created_at"`
                UpdatedAt time.Time `json:"updated_at"`
            } `json:"roles"`
            RolesUsers []struct {
                ID string `json:"id"`
                RoleID string `json:"role_id"`
                UserID string `json:"user_id"`
            } `json:"roles_users"`
            Settings []struct {
                ID string `json:"id"`
                Group string `json:"group"`
                Key string `json:"key"`
                Value string `json:"value"`
                Type string `json:"type"`
                Flags interface{} `json:"flags"`
                CreatedAt time.Time `json:"created_at"`
                UpdatedAt time.Time `json:"updated_at"`
            } `json:"settings"`
            Tags []struct {
                ID string `json:"id"`
                Name string `json:"name"`
                Slug string `json:"slug"`
                Description interface{} `json:"description"`
                FeatureImage interface{} `json:"feature_image"`
                ParentID interface{} `json:"parent_id"`
                Visibility string `json:"visibility"`
                OgImage interface{} `json:"og_image"`
                OgTitle interface{} `json:"og_title"`
                OgDescription interface{} `json:"og_description"`
                TwitterImage interface{} `json:"twitter_image"`
                TwitterTitle interface{} `json:"twitter_title"`
                TwitterDescription interface{} `json:"twitter_description"`
                MetaTitle interface{} `json:"meta_title"`
                MetaDescription interface{} `json:"meta_description"`
                CodeinjectionHead interface{} `json:"codeinjection_head"`
                CodeinjectionFoot interface{} `json:"codeinjection_foot"`
                CanonicalURL interface{} `json:"canonical_url"`
                AccentColor interface{} `json:"accent_color"`
                CreatedAt time.Time `json:"created_at"`
                UpdatedAt time.Time `json:"updated_at"`
            } `json:"tags"`
            Users []struct {
                ID string `json:"id"`
                Name string `json:"name"`
                Slug string `json:"slug"`
                Password string `json:"password"`
                Email string `json:"email"`
                ProfileImage string `json:"profile_image"`
                CoverImage interface{} `json:"cover_image"`
                Bio interface{} `json:"bio"`
                Website interface{} `json:"website"`
                Location interface{} `json:"location"`
                Facebook interface{} `json:"facebook"`
                Twitter interface{} `json:"twitter"`
                Accessibility string `json:"accessibility"`
                Status string `json:"status"`
                Locale interface{} `json:"locale"`
                Visibility string `json:"visibility"`
                MetaTitle interface{} `json:"meta_title"`
                MetaDescription interface{} `json:"meta_description"`
                Tour interface{} `json:"tour"`
                LastSeen time.Time `json:"last_seen"`
                CreatedAt time.Time `json:"created_at"`
                UpdatedAt time.Time `json:"updated_at"`
            } `json:"users"`
        } `json:"data"`
    } `json:"db"`
}

type Mobiledoc struct {
    Version string `json:"version"`
    Markups []interface{} `json:"markups"`
    Atoms []interface{} `json:"atoms"`
    Cards [][]interface{} `json:"cards"`
    Sections [][]interface{} `json:"sections"`
    GhostVersion string `json:"ghostVersion"`
}

func main() {
    fmt.Println("ghost2hugo")

    file, err := os.Open("shindakun-dot-net.ghost.2022-03-18-22-02-58.json")
    if err != nil {
        fmt.Println(err)
    }

    defer file.Close()

    b, err := io.ReadAll(file)
    if err != nil {
        fmt.Println(err)
    }

    var db GhostDatabase

    err = json.Unmarshal(b, &db)
    if err != nil {
        fmt.Println(err)
    }

    for i := 0; i < len(db.Db[0].Data.Posts); i++ {
        fmt.Println(db.Db[0].Data.Posts[i].Title)
        fmt.Println(db.Db[0].Data.Posts[i].Slug)
        fmt.Println(db.Db[0].Data.Posts[i].Status)
        fmt.Println(db.Db[0].Data.Posts[i].CreatedAt)
        fmt.Println(db.Db[0].Data.Posts[i].UpdatedAt)

        c := strings.ReplaceAll(db.Db[0].Data.Posts[i].Mobiledoc, "`", "%'")
        cc := "`" + c + "`"

        fmt.Println(cc)

        ucn, err := strconv.Unquote(cc)
        if err != nil {
            fmt.Println(err, ucn)
        }

        ucn = strings.ReplaceAll(ucn, "%'", "`")

        var md Mobiledoc

        err = json.Unmarshal([]byte(ucn), &md)
        if err != nil {
            fmt.Println(err)
        }

        if reflect.ValueOf(md.Cards).Len() > 0 {
            card := md.Cards[0][1]
            bbb := card.(map[string]interface{})

            fmt.Println(bbb["markdown"])
        }

        fmt.Println("---")
    }
}

Attempting to Learn Go - Ghost to Hugo 2

Steve Layton — Fri, 23 Sep 2022 02:27:29 +0000

Intro

Welcome back! We are continuing on our journey to make a prototype program that converts an exported Ghost database to Markdown. With the end goal being that we can get shindakun.net up and running with Hugo. Last time, we took it pretty easy and focused mostly on reading the file into memory and converting the JSON to a Go struct. From there we printed out the first post.

Post Data

As a recap here is what one of the post fields contain.

{
    "id": "60710b90705967038fe662d6",
    "uuid": "71ba3d71-ac18-4f33-82f7-1962baa83a07",
    "title": "db test",
    "slug": "db-test",
    "mobiledoc": "{\"version\":\"0.3.1\",\"markups\":[],\"atoms\":[],\"cards\":[[\"markdown\",{\"cardName\":\"card-markdown\",\"markdown\":\"\\n<strike>This is a db test</strike>.\\n\"}]],\"sections\":[[10,0]],\"ghostVersion\":\"3.0\"}",
    "html": "<!--kg-card-begin: markdown--><p><strike>This is a db test</strike>.</p>\n<!--kg-card-end: markdown-->",
    "comment_id": "2",
    "plaintext": "This is a db test.",
    "feature_image": null,
    "featured": 0,
    "type": "post",
    "status": "published",
    "locale": null,
    "visibility": "public",
    "email_recipient_filter": "none",
    "author_id": "60710b8d705967038fe66214",
    "created_at": "2004-08-09T19:11:20.000Z",
    "updated_at": "2004-08-09T19:11:20.000Z",
    "published_at": "2004-08-09T19:11:20.000Z",
    "custom_excerpt": null,
    "codeinjection_head": null,
    "codeinjection_foot": null,
    "custom_template": null,
    "canonical_url": null
},

The majority of the fields we need to craft a post with appropriate frontmatter exist within this the object we’re getting back. We can see the title, slug, published date, etc. The section that contains the Markdown is in a format known as Mobiledoc.

Mobiledoc

According to the Ghost documentation Mobiledoc is

…a standardised JSON-based document storage format, which forms the heart of publishing with Ghost.

When extracted from the JSON object and cleaned up we’ll have another bit of JSON we can work with.

{
    "version": "0.3.1",
    "markups": [],
    "atoms": [],
    "cards": [
        [
            "markdown",
            {
                "cardName": "card-markdown",
                "markdown": "\n<strike>This is a db test</strike>.\n"
            }
        ]
    ],
    "sections": [
        [
            10,
            0
        ]
    ],
    "ghostVersion": "3.0"
}

Sounds Easy

First we’ll use our favorite site, the JSON to Go converter to convert the JSON object to a struct we can work with.

type Mobiledoc struct {
    Version string `json:"version"`
    Markups []interface{} `json:"markups"`
    Atoms []interface{} `json:"atoms"`
    Cards [][]interface{} `json:"cards"`
    Sections [][]int `json:"sections"`
    GhostVersion string `json:"ghostVersion"`
}

Our code is pretty long now so I’m going to leave out the other struct GhostDatabase struct, it’ll be in the complete code listing below though. We’re still going to be dumping code to the screen since we’re still working on our decoding logic.

package main

import (
    "encoding/json"
    "fmt"
    "io"
    "os"
    "strconv"
    "time"
)

type GhostDatabase struct {...}

type Mobiledoc struct {
    Version string `json:"version"`
    Markups []interface{} `json:"markups"`
    Atoms []interface{} `json:"atoms"`
    Cards [][]interface{} `json:"cards"`
    Sections [][]int `json:"sections"`
    GhostVersion string `json:"ghostVersion"`
}

func main() {
    fmt.Println("ghost2hugo")

    file, err := os.Open("shindakun-dot-net.ghost.2022-03-18-22-02-58.json")
    if err != nil {
        fmt.Println(err)
    }

    defer file.Close()

    b, err := io.ReadAll(file)
    if err != nil {
        fmt.Println(err)
    }

    var db GhostDatabase

    err = json.Unmarshal(b, &db)
    if err != nil {
        fmt.Println(err)
    }

Let’s continue to focus on the first post for now since once we have that working it should just be a matter of looping through the “database”. This is where it gets a little tricky. We’re working with a couple of nested arrays so to work our way down to the appropriate section we use db.Db[0].Data.Posts[0].Mobiledoc. This will give use the escaped version of our JSON object.

"{\"version\":\"0.3.1\",\"markups\":[],\"atoms\":[],\"cards\":[[\"markdown\",{\"cardName\":\"card-markdown\",\"markdown\":\"\\n<strike>This is a db test</strike>.\\n\"}]],\"sections\":[[10,0]],\"ghostVersion\":\"3.0\"}"

Let’s Go On A Trip

I knew that there would be a way to unescape the string that we get, checking the Go documentation led me to strconv.Unquote.

Unquote interprets s as a single-quoted, double-quoted, or backquoted Go string literal, returning the string value that s quotes. (If s is single-quoted, it would be a Go character literal; Unquote returns the corresponding one-character string.)

Exactly what we need! Except that when I tried to unquote the string I kept receiving an invalid syntax error. This had me confused for a little bit. After puzzling over it I realized if I prepended and appended a back tick to the string it seems to be treated as a raw string literal. This leads to code that looks like the following.

    c := "`" + db.Db[0].Data.Posts[0].Mobiledoc + "`"

    un, err := strconv.Unquote(c)
    if err != nil {
        fmt.Println(err)
    }
    fmt.Printf("%v\n", un)

Finally! We have the JSON!

{"version":"0.3.1","markups":[],"atoms":[],"cards":[["markdown",{"cardName":"card-markdown","markdown":"\n<strike>This is a db test</strike>.\n"}]],"sections":[[10,0]],"ghostVersion":"3.0"}

Now, we can unmarshal that into the Mobiledoc struct we set up earlier!

    var md Mobiledoc

    err = json.Unmarshal([]byte(un), &md)
    if err != nil {
        fmt.Println(err)
    }

    fmt.Printf("%#v", md)

Checking our dumped result we can see that we’ve got the expected data.

main.Mobiledoc{Version:"0.3.1", Markups:[]interface {}{}, Atoms:[]interface {}{}, Cards:[][]interface {}{[]interface {}{"markdown", map[string]interface {}{"cardName":"card-markdown", "markdown":"\n<strike>This is a db test</strike>.\n"}}}, Sections:[][]int{[]int{10, 0}}, GhostVersion:"3.0"}

Right now we’re concerned with the section called Cards which if we naively want to access we can use the following.

    card := md.Cards[0][1]
    fmt.Printf("\n\ncard: %#v\n", card)

card: map[string]interface {}{"cardName":"card-markdown", "markdown":"\n<strike>This is a db test</strike>.\n"}

Let’s convert that to something a bit easier to access.

    bbb := card.(map[string]interface{})
    fmt.Println(bbb["markdown"])

<strike>This is a db test</strike>.

Next Time

So far so good, we’ve extracted the Markdown from the first post as expected. Next time around we’ll be writing a small loop to start trying to extract all the posts. I have a feeling that’s where the fun will begin.

Enjoy this post?
How about buying me a coffee?

shindakun / atlg

Source repo for the "Attempting to Learn Go" posts I've been putting up over on dev.to

Attempting to Learn Go

Here you can find the code I've been writing for my Attempting to Learn Go posts that I've been writing and posting over on Dev.to.

Post Index

Post	Code
Building a Downloader Part 01	-
Building a Downloader Part 02	-
Building a Downloader Part 03	-
Building a Downloader Part 04	-
Building a Downloader Part 05	-
Consuming a REST API	src
Continuing REST Adventures	src
Now Sending REST Requests	src
REST API and A Bit On Templates	src
Sending Email Via API Again	src
Let's Get Modular!	src
Let's Get Modular - Again!	src
Building DevLog Part 01	src
Building DevLog Part 02	src
Building DevLog Part 03	src
Building DevLog Part 04	src
Building DevLog Part 05	src
Listing Files By Extension 01	src
Listing Files By Extension 02	src
Dev.to API 01	src
Dev.to API 02	see above code
Issuer

…

View on GitHub

Code Listing

package main

import (
    "encoding/json"
    "fmt"
    "io"
    "os"
    "strconv"
    "time"
)

type GhostDatabase struct {
    Db []struct {
        Meta struct {
            ExportedOn int64 `json:"exported_on"`
            Version string `json:"version"`
        } `json:"meta"`
        Data struct {
            Posts []struct {
                ID string `json:"id"`
                UUID string `json:"uuid"`
                Title string `json:"title"`
                Slug string `json:"slug"`
                Mobiledoc string `json:"mobiledoc"`
                HTML string `json:"html"`
                CommentID string `json:"comment_id"`
                Plaintext string `json:"plaintext"`
                FeatureImage interface{} `json:"feature_image"`
                Featured int `json:"featured"`
                Type string `json:"type"`
                Status string `json:"status"`
                Locale interface{} `json:"locale"`
                Visibility string `json:"visibility"`
                EmailRecipientFilter string `json:"email_recipient_filter"`
                AuthorID string `json:"author_id"`
                CreatedAt time.Time `json:"created_at"`
                UpdatedAt time.Time `json:"updated_at"`
                PublishedAt time.Time `json:"published_at"`
                CustomExcerpt interface{} `json:"custom_excerpt"`
                CodeinjectionHead interface{} `json:"codeinjection_head"`
                CodeinjectionFoot interface{} `json:"codeinjection_foot"`
                CustomTemplate interface{} `json:"custom_template"`
                CanonicalURL interface{} `json:"canonical_url"`
            } `json:"posts"`
            PostsAuthors []struct {
                ID string `json:"id"`
                PostID string `json:"post_id"`
                AuthorID string `json:"author_id"`
                SortOrder int `json:"sort_order"`
            } `json:"posts_authors"`
            PostsMeta []interface{} `json:"posts_meta"`
            PostsTags []struct {
                ID string `json:"id"`
                PostID string `json:"post_id"`
                TagID string `json:"tag_id"`
                SortOrder int `json:"sort_order"`
            } `json:"posts_tags"`
            Roles []struct {
                ID string `json:"id"`
                Name string `json:"name"`
                Description string `json:"description"`
                CreatedAt time.Time `json:"created_at"`
                UpdatedAt time.Time `json:"updated_at"`
            } `json:"roles"`
            RolesUsers []struct {
                ID string `json:"id"`
                RoleID string `json:"role_id"`
                UserID string `json:"user_id"`
            } `json:"roles_users"`
            Settings []struct {
                ID string `json:"id"`
                Group string `json:"group"`
                Key string `json:"key"`
                Value string `json:"value"`
                Type string `json:"type"`
                Flags interface{} `json:"flags"`
                CreatedAt time.Time `json:"created_at"`
                UpdatedAt time.Time `json:"updated_at"`
            } `json:"settings"`
            Tags []struct {
                ID string `json:"id"`
                Name string `json:"name"`
                Slug string `json:"slug"`
                Description interface{} `json:"description"`
                FeatureImage interface{} `json:"feature_image"`
                ParentID interface{} `json:"parent_id"`
                Visibility string `json:"visibility"`
                OgImage interface{} `json:"og_image"`
                OgTitle interface{} `json:"og_title"`
                OgDescription interface{} `json:"og_description"`
                TwitterImage interface{} `json:"twitter_image"`
                TwitterTitle interface{} `json:"twitter_title"`
                TwitterDescription interface{} `json:"twitter_description"`
                MetaTitle interface{} `json:"meta_title"`
                MetaDescription interface{} `json:"meta_description"`
                CodeinjectionHead interface{} `json:"codeinjection_head"`
                CodeinjectionFoot interface{} `json:"codeinjection_foot"`
                CanonicalURL interface{} `json:"canonical_url"`
                AccentColor interface{} `json:"accent_color"`
                CreatedAt time.Time `json:"created_at"`
                UpdatedAt time.Time `json:"updated_at"`
            } `json:"tags"`
            Users []struct {
                ID string `json:"id"`
                Name string `json:"name"`
                Slug string `json:"slug"`
                Password string `json:"password"`
                Email string `json:"email"`
                ProfileImage string `json:"profile_image"`
                CoverImage interface{} `json:"cover_image"`
                Bio interface{} `json:"bio"`
                Website interface{} `json:"website"`
                Location interface{} `json:"location"`
                Facebook interface{} `json:"facebook"`
                Twitter interface{} `json:"twitter"`
                Accessibility string `json:"accessibility"`
                Status string `json:"status"`
                Locale interface{} `json:"locale"`
                Visibility string `json:"visibility"`
                MetaTitle interface{} `json:"meta_title"`
                MetaDescription interface{} `json:"meta_description"`
                Tour interface{} `json:"tour"`
                LastSeen time.Time `json:"last_seen"`
                CreatedAt time.Time `json:"created_at"`
                UpdatedAt time.Time `json:"updated_at"`
            } `json:"users"`
        } `json:"data"`
    } `json:"db"`
}

type Mobiledoc struct {
    Version string `json:"version"`
    Markups []interface{} `json:"markups"`
    Atoms []interface{} `json:"atoms"`
    Cards [][]interface{} `json:"cards"`
    Sections [][]int `json:"sections"`
    GhostVersion string `json:"ghostVersion"`
}

func main() {
    fmt.Println("ghost2hugo")

    file, err := os.Open("shindakun-dot-net.ghost.2022-03-18-22-02-58.json")
    if err != nil {
        fmt.Println(err)
    }

    defer file.Close()

    b, err := io.ReadAll(file)
    if err != nil {
        fmt.Println(err)
    }

    var db GhostDatabase

    err = json.Unmarshal(b, &db)
    if err != nil {
        fmt.Println(err)
    }

    c := "`" + db.Db[0].Data.Posts[0].Mobiledoc + "`"

    un, err := strconv.Unquote(c)
    if err != nil {
        fmt.Println(err)
    }
    fmt.Printf("%v", un)

    var md Mobiledoc

    err = json.Unmarshal([]byte(un), &md)
    if err != nil {
        fmt.Println(err)
    }

    fmt.Printf("%#v", md)

    card := md.Cards[0][1]
    fmt.Printf("\n\ncard: %#v\n", card)

    bbb := card.(map[string]interface{})
    fmt.Println(bbb["markdown"])
}

Attempting to Learn Go - Ghost to Hugo 1

Steve Layton — Mon, 19 Sep 2022 03:54:29 +0000

Intro

You probably don’t know this but shindakun.net has been effectively offline now for months. Currently, visiting it will only show a white page and a pop-up, no content is accessible. This is a side effect to my upgrading to a new server and moving away from Ghost as my platform of choice.

I love(d) Ghost but it just got too large to effectively run multiple sites on a small instance on Digital Ocean. I want to get the site back up and running but this time using Hugo. This seems like a great opportunity to make a new series of ATLG posts… plus it’s been a while since I’ve written any Go.

Post Data

Before shutting down the original server I exported a copy of the Ghost database as JSON. I should be able to use this to build out the needed Markdown files to make up the content of the site. Here’s the first post.

{
    "id": "60710b90705967038fe662d6",
    "uuid": "71ba3d71-ac18-4f33-82f7-1962baa83a07",
    "title": "db test",
    "slug": "db-test",
    "mobiledoc": "{\"version\":\"0.3.1\",\"markups\":[],\"atoms\":[],\"cards\":[[\"markdown\",{\"cardName\":\"card-markdown\",\"markdown\":\"\\n<strike>This is a db test</strike>.\\n\"}]],\"sections\":[[10,0]],\"ghostVersion\":\"3.0\"}",
    "html": "<!--kg-card-begin: markdown--><p><strike>This is a db test</strike>.</p>\n<!--kg-card-end: markdown-->",
    "comment_id": "2",
    "plaintext": "This is a db test.",
    "feature_image": null,
    "featured": 0,
    "type": "post",
    "status": "published",
    "locale": null,
    "visibility": "public",
    "email_recipient_filter": "none",
    "author_id": "60710b8d705967038fe66214",
    "created_at": "2004-08-09T19:11:20.000Z",
    "updated_at": "2004-08-09T19:11:20.000Z",
    "published_at": "2004-08-09T19:11:20.000Z",
    "custom_excerpt": null,
    "codeinjection_head": null,
    "codeinjection_foot": null,
    "custom_template": null,
    "canonical_url": null
},

Yes, I have some posts dating back to 2004! Back then I wrote my own custom blog software in PHP. It was a neat time, too bad none of the code survived. It almost makes me want to try and do it again only in Go… but I’ll resist that urge.

Quick and Dirty

Alright let’s write a program which will read our JSON and spit out a single post. First things first we need a struct that maps to our “database”. We get this by simply slapping the JSON into the awesome JSON to Go converter.

type GhostDatabase struct {
    Db []struct {
        Meta struct {
            ExportedOn int64 `json:"exported_on"`
            Version string `json:"version"`
        } `json:"meta"`
        Data struct {
            Posts []struct {
                ID string `json:"id"`
                UUID string `json:"uuid"`
                Title string `json:"title"`
                Slug string `json:"slug"`
                Mobiledoc string `json:"mobiledoc"`
                HTML string `json:"html"`
                CommentID string `json:"comment_id"`
                Plaintext string `json:"plaintext"`
                FeatureImage interface{} `json:"feature_image"`
                Featured int `json:"featured"`
                Type string `json:"type"`
                Status string `json:"status"`
                Locale interface{} `json:"locale"`
                Visibility string `json:"visibility"`
                EmailRecipientFilter string `json:"email_recipient_filter"`
                AuthorID string `json:"author_id"`
                CreatedAt time.Time `json:"created_at"`
                UpdatedAt time.Time `json:"updated_at"`
                PublishedAt time.Time `json:"published_at"`
                CustomExcerpt interface{} `json:"custom_excerpt"`
                CodeinjectionHead interface{} `json:"codeinjection_head"`
                CodeinjectionFoot interface{} `json:"codeinjection_foot"`
                CustomTemplate interface{} `json:"custom_template"`
                CanonicalURL interface{} `json:"canonical_url"`
            } `json:"posts"`
            PostsAuthors []struct {
                ID string `json:"id"`
                PostID string `json:"post_id"`
                AuthorID string `json:"author_id"`
                SortOrder int `json:"sort_order"`
            } `json:"posts_authors"`
            PostsMeta []interface{} `json:"posts_meta"`
            PostsTags []struct {
                ID string `json:"id"`
                PostID string `json:"post_id"`
                TagID string `json:"tag_id"`
                SortOrder int `json:"sort_order"`
            } `json:"posts_tags"`
            Roles []struct {
                ID string `json:"id"`
                Name string `json:"name"`
                Description string `json:"description"`
                CreatedAt time.Time `json:"created_at"`
                UpdatedAt time.Time `json:"updated_at"`
            } `json:"roles"`
            RolesUsers []struct {
                ID string `json:"id"`
                RoleID string `json:"role_id"`
                UserID string `json:"user_id"`
            } `json:"roles_users"`
            Settings []struct {
                ID string `json:"id"`
                Group string `json:"group"`
                Key string `json:"key"`
                Value string `json:"value"`
                Type string `json:"type"`
                Flags interface{} `json:"flags"`
                CreatedAt time.Time `json:"created_at"`
                UpdatedAt time.Time `json:"updated_at"`
            } `json:"settings"`
            Tags []struct {
                ID string `json:"id"`
                Name string `json:"name"`
                Slug string `json:"slug"`
                Description interface{} `json:"description"`
                FeatureImage interface{} `json:"feature_image"`
                ParentID interface{} `json:"parent_id"`
                Visibility string `json:"visibility"`
                OgImage interface{} `json:"og_image"`
                OgTitle interface{} `json:"og_title"`
                OgDescription interface{} `json:"og_description"`
                TwitterImage interface{} `json:"twitter_image"`
                TwitterTitle interface{} `json:"twitter_title"`
                TwitterDescription interface{} `json:"twitter_description"`
                MetaTitle interface{} `json:"meta_title"`
                MetaDescription interface{} `json:"meta_description"`
                CodeinjectionHead interface{} `json:"codeinjection_head"`
                CodeinjectionFoot interface{} `json:"codeinjection_foot"`
                CanonicalURL interface{} `json:"canonical_url"`
                AccentColor interface{} `json:"accent_color"`
                CreatedAt time.Time `json:"created_at"`
                UpdatedAt time.Time `json:"updated_at"`
            } `json:"tags"`
            Users []struct {
                ID string `json:"id"`
                Name string `json:"name"`
                Slug string `json:"slug"`
                Password string `json:"password"`
                Email string `json:"email"`
                ProfileImage string `json:"profile_image"`
                CoverImage interface{} `json:"cover_image"`
                Bio interface{} `json:"bio"`
                Website interface{} `json:"website"`
                Location interface{} `json:"location"`
                Facebook interface{} `json:"facebook"`
                Twitter interface{} `json:"twitter"`
                Accessibility string `json:"accessibility"`
                Status string `json:"status"`
                Locale interface{} `json:"locale"`
                Visibility string `json:"visibility"`
                MetaTitle interface{} `json:"meta_title"`
                MetaDescription interface{} `json:"meta_description"`
                Tour interface{} `json:"tour"`
                LastSeen time.Time `json:"last_seen"`
                CreatedAt time.Time `json:"created_at"`
                UpdatedAt time.Time `json:"updated_at"`
            } `json:"users"`
        } `json:"data"`
    } `json:"db"`
}

Perfect. Note that not all of the struct is strictly required but I’ll leave it in for now. Who knows I might write something to pull the tags for posts too.

Reading the File

To start with we’re going to open our file. Note that this is indeed a quick and dirty version, maybe we’ll add some proper error handling later.

    file, err := os.Open("shindakun-dot-net.ghost.2022-03-18-22-02-58.json")
    if err != nil {
        fmt.Println(err)
    }

Once opened we need to read the file into memory as a []byte.

    b, err := io.ReadAll(file)
    if err != nil {
        fmt.Println(err)
    }

Now we’ll declare our database variable, db. Using json.Unmarshal() we convert the JSON into the struct we will be using.

    var db GhostDatabase

    err = json.Unmarshal(b, &db)
    if err != nil {
        fmt.Println(err)
    }

Finally, we print out the text of the first post.

    fmt.Printf("%#v", db.Db[0].Data.Posts[0].HTML)

This leaves us with the output

<!--kg-card-begin: markdown--><p><strike>This is a db test</strike>.</p>\n<!--kg-card-end: markdown-->

Next Time

This part of our program is relatively easy. We don’t want the HTML though, what I’d rather do is pull the Markdown out of the mobiledoc value.

"mobiledoc": "{\"version\":\"0.3.1\",\"markups\":[],\"atoms\":[],\"cards\":[[\"markdown\",{\"cardName\":\"card-markdown\",\"markdown\":\"\\n<strike>This is a db test</strike>.\\n\"}]],\"sections\":[[10,0]],\"ghostVersion\":\"3.0\"}",

Which, as luck would have it, is just an embedded JSON object. It looks like we’ll need to do some cleaning up of the object removing escaped quotes. Once cleaned up it should look similar to

{
    "version": "0.3.1",
    "markups": [],
    "atoms": [],
    "cards": [
        [
            "markdown",
            {
                "cardName": "card-markdown",
                "markdown": "\n<strike>This is a db test</strike>.\n"
            }
        ]
    ],
    "sections": [
        [
            10,
            0
        ]
    ],
    "ghostVersion": "3.0"
}

I haven’t looked to closely at the database but I’m hoping when I wrote my posts in Ghost they all were in a single card-markdown. Oh well, we’ll cross that bridge eventually.

Until next time!

Enjoy this post?
How about buying me a coffee?

shindakun / atlg

Source repo for the "Attempting to Learn Go" posts I've been putting up over on dev.to

Attempting to Learn Go

Here you can find the code I've been writing for my Attempting to Learn Go posts that I've been writing and posting over on Dev.to.

Post Index

Post	Code
Building a Downloader Part 01	-
Building a Downloader Part 02	-
Building a Downloader Part 03	-
Building a Downloader Part 04	-
Building a Downloader Part 05	-
Consuming a REST API	src
Continuing REST Adventures	src
Now Sending REST Requests	src
REST API and A Bit On Templates	src
Sending Email Via API Again	src
Let's Get Modular!	src
Let's Get Modular - Again!	src
Building DevLog Part 01	src
Building DevLog Part 02	src
Building DevLog Part 03	src
Building DevLog Part 04	src
Building DevLog Part 05	src
Listing Files By Extension 01	src
Listing Files By Extension 02	src
Dev.to API 01	src
Dev.to API 02	see above code
Issuer

…

View on GitHub

Code Listing

package main

import (
    "encoding/json"
    "fmt"
    "io"
    "os"
    "time"
)

type GhostDatabase struct {
    Db []struct {
        Meta struct {
            ExportedOn int64 `json:"exported_on"`
            Version string `json:"version"`
        } `json:"meta"`
        Data struct {
            Posts []struct {
                ID string `json:"id"`
                UUID string `json:"uuid"`
                Title string `json:"title"`
                Slug string `json:"slug"`
                Mobiledoc string `json:"mobiledoc"`
                HTML string `json:"html"`
                CommentID string `json:"comment_id"`
                Plaintext string `json:"plaintext"`
                FeatureImage interface{} `json:"feature_image"`
                Featured int `json:"featured"`
                Type string `json:"type"`
                Status string `json:"status"`
                Locale interface{} `json:"locale"`
                Visibility string `json:"visibility"`
                EmailRecipientFilter string `json:"email_recipient_filter"`
                AuthorID string `json:"author_id"`
                CreatedAt time.Time `json:"created_at"`
                UpdatedAt time.Time `json:"updated_at"`
                PublishedAt time.Time `json:"published_at"`
                CustomExcerpt interface{} `json:"custom_excerpt"`
                CodeinjectionHead interface{} `json:"codeinjection_head"`
                CodeinjectionFoot interface{} `json:"codeinjection_foot"`
                CustomTemplate interface{} `json:"custom_template"`
                CanonicalURL interface{} `json:"canonical_url"`
            } `json:"posts"`
            PostsAuthors []struct {
                ID string `json:"id"`
                PostID string `json:"post_id"`
                AuthorID string `json:"author_id"`
                SortOrder int `json:"sort_order"`
            } `json:"posts_authors"`
            PostsMeta []interface{} `json:"posts_meta"`
            PostsTags []struct {
                ID string `json:"id"`
                PostID string `json:"post_id"`
                TagID string `json:"tag_id"`
                SortOrder int `json:"sort_order"`
            } `json:"posts_tags"`
            Roles []struct {
                ID string `json:"id"`
                Name string `json:"name"`
                Description string `json:"description"`
                CreatedAt time.Time `json:"created_at"`
                UpdatedAt time.Time `json:"updated_at"`
            } `json:"roles"`
            RolesUsers []struct {
                ID string `json:"id"`
                RoleID string `json:"role_id"`
                UserID string `json:"user_id"`
            } `json:"roles_users"`
            Settings []struct {
                ID string `json:"id"`
                Group string `json:"group"`
                Key string `json:"key"`
                Value string `json:"value"`
                Type string `json:"type"`
                Flags interface{} `json:"flags"`
                CreatedAt time.Time `json:"created_at"`
                UpdatedAt time.Time `json:"updated_at"`
            } `json:"settings"`
            Tags []struct {
                ID string `json:"id"`
                Name string `json:"name"`
                Slug string `json:"slug"`
                Description interface{} `json:"description"`
                FeatureImage interface{} `json:"feature_image"`
                ParentID interface{} `json:"parent_id"`
                Visibility string `json:"visibility"`
                OgImage interface{} `json:"og_image"`
                OgTitle interface{} `json:"og_title"`
                OgDescription interface{} `json:"og_description"`
                TwitterImage interface{} `json:"twitter_image"`
                TwitterTitle interface{} `json:"twitter_title"`
                TwitterDescription interface{} `json:"twitter_description"`
                MetaTitle interface{} `json:"meta_title"`
                MetaDescription interface{} `json:"meta_description"`
                CodeinjectionHead interface{} `json:"codeinjection_head"`
                CodeinjectionFoot interface{} `json:"codeinjection_foot"`
                CanonicalURL interface{} `json:"canonical_url"`
                AccentColor interface{} `json:"accent_color"`
                CreatedAt time.Time `json:"created_at"`
                UpdatedAt time.Time `json:"updated_at"`
            } `json:"tags"`
            Users []struct {
                ID string `json:"id"`
                Name string `json:"name"`
                Slug string `json:"slug"`
                Password string `json:"password"`
                Email string `json:"email"`
                ProfileImage string `json:"profile_image"`
                CoverImage interface{} `json:"cover_image"`
                Bio interface{} `json:"bio"`
                Website interface{} `json:"website"`
                Location interface{} `json:"location"`
                Facebook interface{} `json:"facebook"`
                Twitter interface{} `json:"twitter"`
                Accessibility string `json:"accessibility"`
                Status string `json:"status"`
                Locale interface{} `json:"locale"`
                Visibility string `json:"visibility"`
                MetaTitle interface{} `json:"meta_title"`
                MetaDescription interface{} `json:"meta_description"`
                Tour interface{} `json:"tour"`
                LastSeen time.Time `json:"last_seen"`
                CreatedAt time.Time `json:"created_at"`
                UpdatedAt time.Time `json:"updated_at"`
            } `json:"users"`
        } `json:"data"`
    } `json:"db"`
}

func main() {
    fmt.Println("ghost2hugo")

    file, err := os.Open("shindakun-dot-net.ghost.2022-03-18-22-02-58.json")
    if err != nil {
        fmt.Println(err)
    }

    b, err := io.ReadAll(file)
    if err != nil {
        fmt.Println(err)
    }

    var db GhostDatabase

    err = json.Unmarshal(b, &db)
    if err != nil {
        fmt.Println(err)
    }

    fmt.Printf("%#v", db.Db[0].Data.Posts[0].HTML)
}

Adding gcc to git bash

Steve Layton — Sat, 03 Sep 2022 19:43:41 +0000

So today I was working on my “main” computer which didn’t have much set up on it. Basically only git bash. Problem is I wanted to work on my Hugo site and theme. To do that I needed to install the version of Hugo with extended support for SCSS. This requires gcc. The quickest solution I found was to just install msys2. Once installed it was just a matter of changing to the Hugo directory and running go install --tags extended.

$ cd /d/Code/hugo/
$ go install --tags extended
$ hugo version
hugo v0.103.0-DEV-06c3ac67411e2faaf1545427d4df4e326fffd3c0+extended windows/amd64 BuildDate=2022-09-03T09:38:03Z

Enjoy this post?
How about buying me a coffee?

Polywork Poster in Go

Steve Layton — Wed, 12 May 2021 03:14:50 +0000

A couple of days ago I finally got access to Polywork. Which, for those who don't know, is like a new-aged Linkedin. It allows you to create a timeline of your work and accomplishments. The idea is that people will be able to contact you through the site to offer you work or opportunities.Take a look at meta.shindakun.dev for a simple example timeline.

One of the things that struck me was that there is no public API at the moment. Which bummed me out as I didn't want to start entering blog posts by hand. So, I decided to poke around a bit and see what the flow is for creating a new post. With the objected being crafting some Go code to create posts.

This would dovetail nicely into our data connector series.

Getting Started

First, we need to look to see how the site lets us put up a new post. This can be done via the network tab in Chromes Dev Tools.

We can see that clicking on the "Post" button brings up a form that we can use to enter our details. Looking at the details of the edit network call shows the entire Turbo Frame that makes up. new in this case seems to just redirect to edit.

Perfect! This will allow us to grab the details we need from the form. We need both the update ID and the authenticity token. Though at present it seems the authenticity token isn't actually required when posting a new update. 🤷‍♂️ I won't show the complete posting flow via the website since it includes cookies but you get the idea.

Our Plan

So here is what we are going to do:

make a request to https://www.polywork.com/shindakun/highlights/new/
extract the update ID and authenticity token
create a multipart form with information we want to post
POST to https://www.polywork.com/shindakun/highlights/{{update_id}}

Enjoy this post?
How about buying me a coffee?

Getting Polywork to Go

As always you can find the full code listing at the bottom of this post. Also, this code is pretty sloppy and a bit brittle since it is just a proof of concept.

package main

import (
    "bytes"
    "fmt"
    "io"
    "log"
    "mime/multipart"
    "net/http"
    "regexp"
    "strings"

    "github.com/andybalholm/cascadia"
    "golang.org/x/net/html"
)

const cookie = "_polywork_session=...snip..."
const username = "shindakun"

We start up by setting our cookie value. This is necessary to be able to POST to the website. I simply grabbed this value from the network tab in Chrome. We could start with a "magic" link login but this seemed easier. We'll also set our username.

func main() {
    req, err := http.NewRequest(http.MethodGet, "https://www.polywork.com/"+username+"/highlights/new/", nil)
    req.Header.Add("Cookie", cookie)
    req.Header.Add("Turbo-Frame", "navbar-post-highlight")
    if err != nil {
        log.Fatal(err)
    }

    res, err := http.DefaultClient.Do(req)
    if err != nil {
        log.Fatal(err)
    }
    defer res.Body.Close()

If you've worked with Go before this probably looks very familiar. What we are doing is using http.NewRequest to request to /{{username}}/highlights/new. Note how we are setting the cookie directly in the header and setting a header for the Turbo-Frame.

Calling http.DefaultClient.Do(req) makes the request to the remote server and if successful will return that data in res.

    doc, err := html.Parse(res.Body)
    if err != nil {
        log.Fatal(err)
    }

    // inputs := cascadia.MustCompile("input").MatchAll(doc)

    buttons := cascadia.MustCompile("button").MatchAll(doc)

    re := regexp.MustCompile(`\/shindakun\/highlights\/(.*)\/save_draft`)
    r := re.FindStringSubmatch(buttons[1].Attr[5].Val)

Here we're going to get a bit tricky. The first thing we want to do is use the parse our incoming form as HTML. This then allows us to use the cascadia library and CSS selectors to grab all the buttons. We don't really need all of them but for this first version what the heck.

Using the regular expression \/shindakun\/highlights\/(.*)\/save_draft we extract the ID for the post we are working with.

This is the main section of the code that is very brittle since I'm not doing anything to validate we are looking at the right button key and instead just hard coding the values that I saw during testing.

    // _method: patch
    // authenticity_token: ...snip...
    // draft_highlight[end_date]: 01-01-2021
    // draft_highlight[content]: <div>Testing</div>
    // draft_highlight[activity_ids][]: 76199
    // commit: Post

During testing, I made a copy of the form data POSTed from Chrome for reference. We'll use this as our base form data. As you'll see below not every field is required. I think I can leave out the _method field but have not tried as of yet. ~~I mentioned before but you should also note that I'm not using the authenticity_token token as it seems it is not required.~~ Update: the latest site update now properly requires the token.

    body := &bytes.Buffer{}
    writer := multipart.NewWriter(body)

    fw, _ := writer.CreateFormField("_method")
    _, err = io.Copy(fw, strings.NewReader("patch"))
    if err != nil {
        log.Fatal(err)
    }

    // fw, _ = writer.CreateFormField("authenticity_token")
    // _, err = io.Copy(fw, strings.NewReader(inputs[1].Attr[2].Val))
    // if err != nil {
    // log.Fatal(err)
    // }

    fw, _ = writer.CreateFormField("draft_highlight[content]")
    _, err = io.Copy(fw, strings.NewReader("<div>Posting via Golang</div> :) <a href=\"https://shindakun.dev\">shindakun</a>"))
    if err != nil {
        log.Fatal(err)
    }

    fw, _ = writer.CreateFormField("commit")
    _, err = io.Copy(fw, strings.NewReader("Post"))
    if err != nil {
        log.Fatal(err)
    }

    writer.Close()

It's a bit awkward to create a form in Go but hey, it works. We make a form field with writer.CreateFormField and then io.Copy our data into that field. Right now our payload is hardcoded. We could easily adapt my data connector post to automatically add my Ghost blog posts to Polywork.

    post, err := http.NewRequest("POST", "https://www.polywork.com/shindakun/highlights/"+r[1], bytes.NewReader(body.Bytes()))
    if err != nil {
        log.Fatal(err)
    }
    post.Header.Set("Content-Type", writer.FormDataContentType())
    post.Header.Add("Cookie", cookie)
    rsp, _ := http.DefaultClient.Do(post)
    if rsp.StatusCode != http.StatusOK {
        log.Printf("Request failed with response code: %d", rsp.StatusCode)
    }
}

And now we do the actual form submit. And that's all there is to it.

What's Next?

Next, I think I am going to adapt this code to use with the data connector we are working on. I'll probably try to make it pretty generic so I can use it as its own module.

Code Listing

package main

import (
    "bytes"
    "fmt"
    "io"
    "log"
    "mime/multipart"
    "net/http"
    "regexp"
    "strings"

    "github.com/andybalholm/cascadia"
    "golang.org/x/net/html"
)

const cookie = "_polywork_session=...snip..."
const username = "shindakun"

func main() {
    req, err := http.NewRequest(http.MethodGet, "https://www.polywork.com/"+username+"/highlights/new/", nil)
    req.Header.Add("Cookie", cookie)
    req.Header.Add("Turbo-Frame", "navbar-post-highlight")
    if err != nil {
        log.Fatal(err)
    }

    res, err := http.DefaultClient.Do(req)
    if err != nil {
        log.Fatal(err)
    }
    defer res.Body.Close()

    doc, err := html.Parse(res.Body)
    if err != nil {
        log.Fatal(err)
    }

    // inputs := cascadia.MustCompile("input").MatchAll(doc)

    buttons := cascadia.MustCompile("button").MatchAll(doc)

    re := regexp.MustCompile(`\/shindakun\/highlights\/(.*)\/save_draft`)
    r := re.FindStringSubmatch(buttons[1].Attr[5].Val)

    // _method: patch
    // authenticity_token: ...snip...
    // draft_highlight[end_date]: 01-01-2021
    // draft_highlight[content]: <div>Testing</div>
    // draft_highlight[activity_ids][]: 76199
    // commit: Post

    body := &bytes.Buffer{}
    writer := multipart.NewWriter(body)
    fw, _ := writer.CreateFormField("_method")
    _, err = io.Copy(fw, strings.NewReader("patch"))
    if err != nil {
        log.Fatal(err)
    }
    // fw, _ = writer.CreateFormField("authenticity_token")
    // _, err = io.Copy(fw, strings.NewReader(inputs[1].Attr[2].Val))
    // if err != nil {
    // log.Fatal(err)
    // }
    fw, _ = writer.CreateFormField("draft_highlight[content]")
    _, err = io.Copy(fw, strings.NewReader("<div>Posting via Golang</div> :) <a href=\"https://shindakun.dev\">shindakun</a>"))
    if err != nil {
        log.Fatal(err)
    }
    fw, _ = writer.CreateFormField("commit")
    _, err = io.Copy(fw, strings.NewReader("Post"))
    if err != nil {
        log.Fatal(err)
    }
    writer.Close()

    post, err := http.NewRequest("POST", "https://www.polywork.com/shindakun/highlights/"+r[1], bytes.NewReader(body.Bytes()))
    if err != nil {
        log.Fatal(err)
    }
    post.Header.Set("Content-Type", writer.FormDataContentType())
    post.Header.Add("Cookie", cookie)
    rsp, _ := http.DefaultClient.Do(post)
    if rsp.StatusCode != http.StatusOK {
        log.Printf("Request failed with response code: %d", rsp.StatusCode)
    }
}

Golang Data Connector Part 2

Steve Layton — Sun, 02 May 2021 17:39:54 +0000

Intro

Last time we started our small data collector. The idea is we want to move data from a Ghost blog over to something else. This time we're going to extend our starting point to actually make use of the data that comes back from the API call. That means we need to take a closer look at the incoming data... and with that in mind let's get started.

Adding Struct-ure

Currently, we are receiving our data and converting it from []byte to string so we can print it to the screen. That's cool and all but not very useful. Instead, we need to see if we can parse the data and make it usable for the next stage of this project.

To do this we'll need to convert the JSON into a struct. OK, so how do we do that? There are a couple of different ways but the easiest I've found is to use the site JSON to Go. This site converts a JSON object into what is a matching Golang struct.

We can simply paste the raw output from the call to https://demo.ghost.io/ghost/api/v4/content/posts/?key=22444f78447824223cefc48062 into the form. It will spit out the exact struct we need:

type AutoGenerated struct {
    Posts []struct {
        ID string `json:"id"`
        UUID string `json:"uuid"`
        Title string `json:"title"`
        Slug string `json:"slug"`
        HTML string `json:"html"`
        CommentID string `json:"comment_id"`
        FeatureImage string `json:"feature_image"`
        Featured bool `json:"featured"`
        Visibility string `json:"visibility"`
        CreatedAt time.Time `json:"created_at"`
        UpdatedAt time.Time `json:"updated_at"`
        PublishedAt time.Time `json:"published_at"`
        CustomExcerpt string `json:"custom_excerpt"`
        CodeinjectionHead interface{} `json:"codeinjection_head"`
        CodeinjectionFoot interface{} `json:"codeinjection_foot"`
        CustomTemplate interface{} `json:"custom_template"`
        CanonicalURL interface{} `json:"canonical_url"`
        EmailRecipientFilter string `json:"email_recipient_filter"`
        URL string `json:"url"`
        Excerpt string `json:"excerpt"`
        ReadingTime int `json:"reading_time"`
        Access bool `json:"access"`
        OgImage interface{} `json:"og_image"`
        OgTitle interface{} `json:"og_title"`
        OgDescription interface{} `json:"og_description"`
        TwitterImage interface{} `json:"twitter_image"`
        TwitterTitle interface{} `json:"twitter_title"`
        TwitterDescription interface{} `json:"twitter_description"`
        MetaTitle interface{} `json:"meta_title"`
        MetaDescription interface{} `json:"meta_description"`
        EmailSubject interface{} `json:"email_subject"`
        Frontmatter interface{} `json:"frontmatter"`
        Plaintext string `json:"plaintext,omitempty"`
    } `json:"posts"`
    Meta struct {
        Pagination struct {
            Page int `json:"page"`
            Limit int `json:"limit"`
            Pages int `json:"pages"`
            Total int `json:"total"`
            Next interface{} `json:"next"`
            Prev interface{} `json:"prev"`
        } `json:"pagination"`
    } `json:"meta"`
}

Let's start by updating AutoGenerated to say Ghost. From there we can add the entire struct to the top of our previous code - right above main(). That is if we want to just added. In our imagined receiving system the entire struct isn't necessary so I've left out a bit. I've also updated several of the interface{} entries to be strings since that's what we'll actually be getting if the value is set.

package main

import (
    "encoding/json"
    "fmt"
    "io"
    "log"
    "net/http"
    "time"
)

type Ghost struct {
    Posts []struct {
        ID string `json:"id"`
        UUID string `json:"uuid"`
        Title string `json:"title"`
        Slug string `json:"slug"`
        HTML string `json:"html"`
        FeatureImage string `json:"feature_image"`
        Featured bool `json:"featured"`
        Visibility string `json:"visibility"`
        CreatedAt time.Time `json:"created_at"`
        UpdatedAt time.Time `json:"updated_at"`
        PublishedAt time.Time `json:"published_at"`
        CustomExcerpt string `json:"custom_excerpt"`
        URL string `json:"url"`
        Excerpt string `json:"excerpt"`
        ReadingTime int `json:"reading_time"`
        Access bool `json:"access"`
        OgImage string `json:"og_image"`
        OgTitle string `json:"og_title"`
        OgDescription string `json:"og_description"`
        TwitterImage string `json:"twitter_image"`
        TwitterTitle string `json:"twitter_title"`
        TwitterDescription string `json:"twitter_description"`
        MetaTitle string `json:"meta_title"`
        MetaDescription string `json:"meta_description"`
        Plaintext string `json:"plaintext,omitempty"`
    } `json:"posts"`
    Meta struct {
        Pagination struct {
            Page int `json:"page"`
            Limit int `json:"limit"`
            Pages int `json:"pages"`
            Total int `json:"total"`
            Next interface{} `json:"next"`
            Prev interface{} `json:"prev"`
        } `json:"pagination"`
    } `json:"meta"`
}

func main() {

    ...snip...

Enjoy this post?
How about buying me a coffee?

More Printing

Alright, we're slowly getting somewhere! Now, we are going to replace our fmt.Println() with code to actually use our struct. We'll read the body in using the same manner as last time.

    body, err := io.ReadAll(resp.Body)
    if err != nil {
        log.Fatal(err)
    }

But this time instead of printing we'll create a variable p of the type Ghost. Using p we then json.Unmarshal our received body.

    var p Ghost
    err = json.Unmarshal(body, &p)
    if err != nil {
        log.Fatal(err)
    }

From here we now have access to range over the entirety of p.Posts and once again we print them out to the screen.

    for i := range p.Posts {
        fmt.Printf("%v", p.Posts[i])
    }

Next Time

That's all for this quick post, you can see the full code listing below. Next time we'll be taking one more step forward and actually use the http package to send our posts to another service.

Code Listing

package main

import (
    "encoding/json"
    "fmt"
    "io"
    "log"
    "net/http"
    "time"
)

type Ghost struct {
    Posts []struct {
        ID string `json:"id"`
        UUID string `json:"uuid"`
        Title string `json:"title"`
        Slug string `json:"slug"`
        HTML string `json:"html"`
        FeatureImage string `json:"feature_image"`
        Featured bool `json:"featured"`
        Visibility string `json:"visibility"`
        CreatedAt time.Time `json:"created_at"`
        UpdatedAt time.Time `json:"updated_at"`
        PublishedAt time.Time `json:"published_at"`
        CustomExcerpt string `json:"custom_excerpt"`
        URL string `json:"url"`
        Excerpt string `json:"excerpt"`
        ReadingTime int `json:"reading_time"`
        Access bool `json:"access"`
        OgImage string `json:"og_image"`
        OgTitle string `json:"og_title"`
        OgDescription string `json:"og_description"`
        TwitterImage string `json:"twitter_image"`
        TwitterTitle string `json:"twitter_title"`
        TwitterDescription string `json:"twitter_description"`
        MetaTitle string `json:"meta_title"`
        MetaDescription string `json:"meta_description"`
        Plaintext string `json:"plaintext,omitempty"`
    } `json:"posts"`
    Meta struct {
        Pagination struct {
            Page int `json:"page"`
            Limit int `json:"limit"`
            Pages int `json:"pages"`
            Total int `json:"total"`
            Next interface{} `json:"next"`
            Prev interface{} `json:"prev"`
        } `json:"pagination"`
    } `json:"meta"`
}

func main() {
    req, err := http.NewRequest(http.MethodGet, "https://demo.ghost.io/ghost/api/v4/content/posts/?key=22444f78447824223cefc48062", nil)
    if err != nil {
        log.Fatal(err)
    }

    resp, err := http.DefaultClient.Do(req)
    if err != nil {
        log.Fatal(err)
    }

    defer resp.Body.Close()

    body, err := io.ReadAll(resp.Body)
    if err != nil {
        log.Fatal(err)
    }

    var p Ghost
    err = json.Unmarshal(body, &p)
    if err != nil {
        log.Fatal(err)
    }

    for i := range p.Posts {
        fmt.Printf("%v", p.Posts[i])
    }
}

Golang Data Connector Part 1

Steve Layton — Sat, 17 Apr 2021 23:51:21 +0000

Intro

There are a handful of tools out there like Zapier (which just hit a $5 billion valuation) which connect disparate systems together. It's an interesting idea but, how does one actually create a connector (or integration) without Zapier? In this series of posts, we'll step through how I might go about building something.

Getting Spooky

Let's pretend that we want to move data from a Ghost blog over to another service. That sounds simple enough, for this post we're going to not worry about a continued connection we're just going to get our content from Point A. We'll worry about Point B next time.

OK, that's well and good but where do we start? The best place to start is the Ghost content API documentation at https://ghost.org/docs/content-api/. This will give us all the information we need to start planning. There is even a live content/posts endpoint we can call at https://demo.ghost.io/ghost/api/v4/content/posts/?key=22444f78447824223cefc48062, which will show us what we can expect to receive back.

Go Get It

Phase one of this simple connector is relatively easy thanks to Go's http package. We use it to reach out and request the first page of post content. We'll start by creating a new request. The full code listing is at the bottom of this post.

    req, err := http.NewRequest(http.MethodGet, "https://demo.ghost.io/ghost/api/v4/content/posts/?key=22444f78447824223cefc48062", nil)

http.NewRequest takes in the method of the HTTP request, in this case, it is a GET, the URL of the request, and a body. This time we don't have a body so we'll just use nil. OK, great we have our request all set up but how do we use it? Again we're back to http, we can use http.DefaultClient method Do() to actually make our request. We simply pass in our request and capture the response in a new variable.

    resp, err := http.DefaultClient.Do(req)

And that's it! We have our data, well as long as the call was OK that is. We aren't really doing anything with errors instead we'll just bail out. I imagine that we can have some sort of retry instead of just exiting.

From here we'll log the response body to the console.

    body, err := io.ReadAll(resp.Body)
    if err != nil {
        log.Fatal(err)
    }

    fmt.Println(string(body))

Next Time

In our next post, we'll modify our project to allow us to actually do something with the data we have received

Enjoying this post?
How about buying me a coffee?

Code Listing

package main

import (
    "encoding/json"
    "fmt"
    "io"
    "log"
    "net/http"
)

func main() {
    req, err := http.NewRequest(http.MethodGet, "https://demo.ghost.io/ghost/api/v4/content/posts/?key=22444f78447824223cefc48062", nil)
    if err != nil {
        log.Fatal(err)
    }

    resp, err := http.DefaultClient.Do(req)
    if err != nil {
        log.Fatal(err)
    }

    defer resp.Body.Close()

    body, err := io.ReadAll(resp.Body)
    if err != nil {
        log.Fatal(err)
    }

    fmt.Println(string(body))
}

Webmentions are Great (Mostly)

Steve Layton — Fri, 27 Dec 2019 06:54:29 +0000

Oh No

Why are there two author icons?! We’ll get back to that in a moment.

Webmetion Bliss

After setting up and documenting how I deployed webmentions on micro.blog it got picked up by a few people on the site! That was really nice - especially since it worked for the most part. I attempted to make it as clear as possible so others could follow in my footsteps and I succeeded - a few people got set up and were running with mentions. There is a great example over on Miraz Jordan’s site, which you can see a snippet of below.

The Troubles

All is not perfect though. So, I thought I’d touch on some of that here. First, if you were to look at the interactions on Miraz’s site you would see some “likes” down near the bottom that rendered with just an icon and a data. Those posts came from Mathew Packer testing out the site to site replies.

For some reason, those are coming out of webmention.io with what looks to be incorrectly parsed values. Note the '' where you would expect a comment or, more likely, a title to appear.

sentence: "https://www.mathewpacker.com/2019/12/2523/ posted '' linking to https://miraz.me/2019/12/21/thanks-to-shindakun.html"

Though in this case, it doesn’t matter since we are not rendering the sentence (or sentence_html). I believe the JSON issue might be why the original implementation for webmention.js didn’t show “shares” or “likes” with text and just showed an author picture and name. I don’t use WordPress so I am not sure just how far I may troubleshoot that - it seems like the “Post Kinds” plugin might not be formatting the post 100% correctly.

Problems at Home

That brings us back to the header image. A few days after the post went up I backported my changes to my local Hugo theme and pushed them live. Within a day or so I had a few interactions. Chris Aldrich posted a reply to a post by Jan-Lukas Else which linked back to the implementation post. However, it was showing two author photos.

At first, I figured that the JavaScript parsing the webmention data was incorrect. However, nothing jumped out as an issue. I then used a microformat2 parser to see if I could pin down the cause from the original mention.

Behold! It turns out that within the reply post Jetpack Lazy Image loading is causing the reply-to author block to end up with an array of images. This seems to bubble back up through the content that webmentions.io is sending to my page.

In this case I could render the sentence_html and not the content but then I’d lose out some nice conversation context.

I noticed one other thing that I’ll need to keep an eye out for and see if I can work around. Most replies aren’t going to have a title so you may end up with something like the following, where what we are replying to is blank.

I have some off looking replies on the webmention post mostly due to bad CSS. And the fact that I appear to have messed up the CSS I was using for content with no author image. Which makes it difficult to see if the issue is something I’ve done.

I should get those fixed up… and then I need to take a much closer look at the webmention.js and it’s associated CSS.

Enjoy this post?
How about buying me a coffee?

Adding an Image Lightbox To micro.blog

Steve Layton — Sat, 07 Dec 2019 06:15:31 +0000

Background

A day or so ago someone posted in one of the micro.blog slack channels asking how to add a lightbox. I had been meaning to update my setup here on the blog and add a lightbox since my current images weren’t linked. Mostly though, I wanted to get rid of jQuery. So with that in mind, we’re going to go through the first steps of adding a lightbox. We’ll first add it so images in individual posts are linked. This also lays some of the groundwork for us to extend the “photos” page to include a lightbox.

Finding A Lightbox

The first step was to find a “modern” lightbox solution. As I mentioned the majority of older solutions used jQuery and I don’t want another dependancy. jQuery is great and all but, I’m trying to watch my overall page weight. After some searching around I settled on GLightbox.

Hugo Themes

micro.blog, like this site, uses Hugo for hosted blogs. This setup allows having control over the theme so we can easily pick a custom theme and extend it with our gallery script. One thing to take note of - I am using the Hello and while this should work on any Hugo there will be differences if you are using another theme.

GLightbox JS and CSS

GLightbox is very easy to install which makes it perfect for this. We’ll need to edit our theme so I recommend first going through the steps in the help.micro.blog the section on custom themes.

CSS

Assuming you are using the Hello theme we now need to edit layouts/partials/head.html and include our CSS and the actually Glightbox JavaScript. This is relatively easy though, just follow the instructions on the Micro.blog help (Manton has a great YouTube video which covers the basics).

Once in the editor, we’ll want to add the following two lines:

<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/glightbox/dist/css/glightbox.min.css">
<script src="https://cdn.jsdelivr.net/gh/mcstudios/glightbox/dist/js/glightbox.min.js"></script>

JavaScript

So far so good! I have a feeling the next part is going to be different with every theme, so it might not be as easy. When adding an image to a post through the micro.blog website it is simply included as an <img> tag. The last thing I want to do is have to go in and add all the tags to images as I post them.

With that in mind, I crafted a passible bit of JavaScript which will pull out any images that appear within post content and automatically link them so they will be shown in the lightbox.

Neato.

To add the JavaScript we’ll again need to edit part of the theme. In this case we’ll want to open static/assets/main.js for editing. Once open we can added the following JavaScript to the top. Feel free to take out the comments, I’ve just added them for this post.

// First we'll grab all the images inside a post paragraph
let ims = document.querySelectorAll('.post-content p img')

// If ims === 0 lets not do anything
if (ims.length > 0) {

  // Now we'll loop through all of the images that were picked up
  // Note that I'm doing no real error handling, you might want to clean this up
  for (let i = 0; i < ims.length; i++) {

    // For each image we get the parent element, in this case it will be <p>
    let parentElm = ims[i].parentElement;

    // Using the paraent element and the image object we replace the innerHTML
    // with our image with the class "glightbox" and a link to the image
    parentElm.innerHTML = '<a href="' + ims[i].src + '"><img src="' + ims[i].src + '" class="glightbox"></a>'
  }
}

// Finally we call GLightbox and if all went well...
const lightbox = GLightbox({});

Caveats

“This code works on my computer!” Hopefully, if everything went smoothly then the image above should be clickable and should appear in the lightbox. The JavaScript image selection code is likely extremely brittle and probably won’t work for many themes. There should be enough information here for others to follow. For best results, you will probably want to keep to only one image per post (though it should work with most).

Enjoy this post?
How about buying me a coffee?

Attempting to Learn Go - Issuer 04 - Addendum

Steve Layton — Sat, 13 Jul 2019 17:52:32 +0000

Whoops

Our last Issuer post left out one small bit of code. Let’s fix that and wrap this up. Also, let’s cover how to test the webhook before moving to Google Cloud Functions.

Missed `if`

As I published Issuer to Google Cloud Functions I meant to update the code. The idea is to only create new issues when the originating payload action was opened. The final revision of the post left that out though so here we go. Let’s look at our updated HandleWebhook().

func HandleWebhook(res http.ResponseWriter, req *http.Request) {
  var Payload Payload
  defer req.Body.Close()

  p, err := github.ValidatePayload(req, []byte(Secret))
  if err != nil {
    http.Error(res, "bad request: "+err.Error(), 400)
    log.Printf("bad request: %v", err.Error())
    return
  }

  err = json.Unmarshal(p, &Payload)
  if err != nil {
    http.Error(res, "bad request: "+err.Error(), 400)
    log.Printf("bad request: %v", err.Error())
    return
  }

Here we are - we check Payload.Action to ensure it’s opened and if so then we create our new issue.

  if Payload.Action == "opened" {
    err = createNewIssue(&Payload)
    if err != nil {
      log.Printf("bad request: %v", err.Error())
      return
    }
  }
}

Testing Webhooks

Before pushing to GCF it was necessary to test the webhooks to ensure I was receiving what I needed. This is actually pretty easy thanks to ngrok. ngrok is a command line tool which allows us to access our internal program from an external network. No need to worry about setting up your firewall or tunneling.

`ngrok`

To do this we’ll need to make sure we have ngrok installed. On a Mac, which is where I tend to do most of my development lately, it’s a simple matter of installing via brew.

brew install ngrok

If you happen to be using Windows, you can download from the ngrok website, https://ngrok.com/download. Follow the instructions to install on the website. Once installed you should be able to open a command prompt or terminal window and run ngrok.

➜ ~ ngrok
NAME:
   ngrok - tunnel local ports to public URLs and inspect traffic

DESCRIPTION:
    ngrok exposes local networked services behinds NATs and firewalls to the
    public internet over a secure tunnel. Share local websites, build/test
    webhook consumers and self-host personal services.
    Detailed help for each command is available with 'ngrok help <command>'.
    Open http://localhost:4040 for ngrok's web interface to inspect traffic.
...

Now What

Now what? That’s easy! Let’s run…

ngrok http 3000

Assuming you are running the code we wrote last time, that runs on port 3000. Now open a new terminal window and lets run our code.

go run main.go

Now we’re up and running, let's enable some webhooks on GitHub and do some actual testing.

GitHub Webhooks

With both ngrok and our code running we go to GitHub! Pick a repo, or create a test one. Under the settings tab, chose “Webhooks” on the left. Click “Add webhook” on the upper right. Enter our ngrok URL, ex: https://9ba7d0f6.ngrok.io and add /webhook. In “Content type” make sure you select application/json. For this project I chose “Let me select individual events” and then selected only “Issues”. Then click “Add webhook” at the bottom.

GitHub will send off a test post, we should see something right away.

And there we have it! It’s mostly blank because it’s just a test but it’s enough to show that we’re working as expected.

You can find the code for this and most of the other Attempting to Learn Go posts in the repo on GitHub.

shindakun / atlg

Source repo for the "Attempting to Learn Go" posts I've been putting up over on dev.to

Attempting to Learn Go

Here you can find the code I've been writing for my Attempting to Learn Go posts that I've been writing and posting over on Dev.to.

Post Index

Post	Code
Building a Downloader Part 01	-
Building a Downloader Part 02	-
Building a Downloader Part 03	-
Building a Downloader Part 04	-
Building a Downloader Part 05	-
Consuming a REST API	src
Continuing REST Adventures	src
Now Sending REST Requests	src
REST API and A Bit On Templates	src
Sending Email Via API Again	src
Let's Get Modular!	src
Let's Get Modular - Again!	src
Building DevLog Part 01	src
Building DevLog Part 02	src
Building DevLog Part 03	src
Building DevLog Part 04	src
Building DevLog Part 05	src
Listing Files By Extension 01	src
Listing Files By Extension 02	src
Dev.to API 01	src
Dev.to API 02	see above code
Issuer

…

View on GitHub

Enjoy this post?
How about buying me a coffee?

Attempting to Learn Go - Issuer 03 - Cloud Function Go!

Steve Layton — Sat, 22 Jun 2019 23:39:49 +0000

Functioning In The Cloud

Welcome back! We are just done with our GitHub TODO issue creator thing. This time around we’re going to go through the steps to deploy it as a Google Cloud Function. To do this we’ll need to alter the code. But before we dive into the code itself we’ll need to get set up to use Go modules.

Go Mod

First, we’ll need to get our mod file ready. You may or may not already have modules enabled. If you do you should be able to leave off the GO111MODULE=on part of the command.

GO111MODULE=on go mod issuer

Now, we need to make sure we’ve got all the modules we need.

➜ issuer GO111MODULE=on go mod tidy
go: finding github.com/google/go-github/v25/github latest
go: finding golang.org/x/oauth2 latest
go: downloading google.golang.org/appengine v1.4.0
go: extracting google.golang.org/appengine v1.4.0
go: downloading github.com/golang/protobuf v1.2.0
go: extracting github.com/golang/protobuf v1.2.0
go: downloading golang.org/x/sync v0.0.0-20190227155943-e225da77a7e6
go: extracting golang.org/x/sync v0.0.0-20190227155943-e225da77a7e6

Looking good so far! Next, we want to vendor our dependencies. This copies the modules we downloaded in the previous step into our code directory. You don't really need to do this step, I'm including it anyway.

GO111MODULE=on go mod vendor

We should now see the vendor directory.

➜ issuer ls
go.mod go.sum main.go vendor

Alterations

Now let’s make a few changes to our code to get it ready for deploying as a Cloud Function. We’ll be starting off the same as last time. You may notice that I’ve removed the ability to query /status, it is no longer needed.

package issuer

import (
  "context"
  "encoding/json"
  "fmt"
  "log"
  "net/http"
  "os"
  "strconv"

  "github.com/google/go-github/v25/github"
  "github.com/shindakun/envy"
  "golang.org/x/oauth2"
)

Note that I’ve pulled out our constants and replaced them will variables. This allows us to make the code a bit more generic and so others can use it. We’ll cover how we are setting up these variables a bit later on. Note, some people may not like the use of global variables here. Since our function is so small and we have a good idea what’s happening in each step it’s OK to use them for now.

Of course, if you’d rather not have globals feel free to remove them. You can hard code as constants or load them inside our main function call.

// RepoOwner is the owner of the repo we want to open an issue in
var RepoOwner string

// IssueRepo is the repo we want to open this new issue in
var IssueRepo string

// ProjectColumn is the TODO column number of the project we want to add the issue to
var ProjectColumn int64

// Token is the GitHub Personal Access Token
var Token string

// Secret is used to validate payloads
var Secret string

// Payload of GitHub webhook
type Payload struct {
  Action string `json:"action"`
  Issue struct {
    URL string `json:"url"`
    RepositoryURL string `json:"repository_url"`
    Number int `json:"number"`
    Title string `json:"title"`
    Body string `json:"body"`
  } `json:"issue"`
  Repository struct {
    Name string `json:"name"`
  } `json:"repository"`
}

I’ve also changed handleWebhook() to HandleWebhook(). Exporting the function is what allows us to call it as the cloud function.

func HandleWebhook(res http.ResponseWriter, req *http.Request) {
  var Payload Payload
  defer req.Body.Close()

  p, err := github.ValidatePayload(req, []byte(Secret))
  if err != nil {
    http.Error(res, "bad request: "+err.Error(), 400)
    log.Printf("bad request: %v", err.Error())
    return
  }

As @kunde21 pointed out last week we are better off using json.Unmarshal() here. This also allows us to remove the imports for bytes and io/ioutil.

err = json.Unmarshal(p, &Payload)
  if err != nil {
    http.Error(res, "bad request: "+err.Error(), 400)
    log.Printf("bad request: %v", err.Error())
    return
  }

  err = createNewIssue(&Payload)
  if err != nil {
    log.Printf("bad request: %v", err.Error())
    return
  }
}

func createNewIssue(p *Payload) error {
  log.Printf("Creating New Issue.\n")
  log.Printf(" Name: %#v\n", p.Repository.Name)
  log.Printf(" Title: %#v\n", p.Issue.Title)
  log.Printf(" Body: %#v\n", p.Issue.Body)
  log.Printf(" URL: %#v\n", p.Issue.URL)

  ctx := context.Background()
  ts := oauth2.StaticTokenSource(
    &oauth2.Token{AccessToken: Token},
  )
  tc := oauth2.NewClient(ctx, ts)

  client := github.NewClient(tc)

  title := fmt.Sprintf("[%s] %s", p.Repository.Name, p.Issue.Title)
  body := fmt.Sprintf("%s\n%s/%s#%d", p.Issue.Body, RepoOwner, p.Repository.Name, p.Issue.Number)

  issue := &github.IssueRequest{
    Title: &title,
    Body: &body,
  }

  ish, _, err := client.Issues.Create(ctx, RepoOwner, IssueRepo, issue)
  if err != nil {
    log.Printf("error: %v", err)
    return err
  }

  id := *ish.ID
  card := &github.ProjectCardOptions{
    ContentID: id,
    ContentType: "Issue",
  }

  _, _, err = client.Projects.CreateProjectCard(ctx, ProjectColumn, card)
  if err != nil {
    log.Printf("error: %v", err)
    return err
  }

  return nil
}

Now we get to one of the biggest changes. Since the cloud function is going to call HandleWebhook(), we no longer need our main(). But that presents an issue, we have some environment variables we want to use. We could load them in the HandleWebhook() call it’s more appropriate to make use of Go’s init().

init() runs before main (or our handler in this case), which allows us to load our variables as normal.

func init() {
  log.Println("Issuer")
  var err error
  Token, err = envy.Get("GITHUBTOKEN")
  if err != nil || Token == "" {
    log.Printf("error: %v", err)
    os.Exit(1)
  }

  Secret, err = envy.Get("SECRET")
  if err != nil || Secret == "" {
    log.Printf("error: %v", err)
    os.Exit(1)
  }

  RepoOwner, err = envy.Get("REPOOWNER")
  if err != nil || RepoOwner == "" {
    log.Printf("error: %v", err)
    os.Exit(1)
  }

  IssueRepo, err = envy.Get("ISSUEREPO")
  if err != nil || IssueRepo == "" {
    log.Printf("error: %v", err)
    os.Exit(1)
  }

ProjectColumn requires a bit more setting up since I never extended envy to return ints. So we need to convert to an int64 before we can use the column numeric ID to create our TODO card on the kanban board.

n, err := envy.Get("PROJECTCOLUMN")
  if err != nil || n == "" {
    log.Printf("error: %v", err)
    os.Exit(1)
  }
    ProjectColumn, err = strconv.ParseInt(n, 10, 64)
      if err != nil || ProjectColumn == 0 {
    log.Printf("error: %v", err)
    os.Exit(1)
  }
}

Go To The Cloud

I am going to assume that you have the Google Cloud command line tools installed and a project set up. If you do not Google has some very good tutorials. Check out https://cloud.google.com/functions/docs/quickstart for more

Note: You wouldn’t want to have the secret and the token deployed like this in production. You would instead want to use Cloud KMS or Runtime Configurator. I’m living dangerously.

➜ issuer gcloud functions deploy issuer --entry-point HandleWebhook --runtime go111 --trigger-http --memory=128MB --set-env-vars SECRET=secret,GITHUBTOKEN=token,REPOOWNER=shindakun,ISSUEREPO=to,PROJECTCOLUMN=5647145

Deploying function (may take a while - up to 2 minutes)...done.
availableMemoryMb: 128
entryPoint: HandleWebhook
environmentVariables:
  GITHUBTOKEN: token
  ISSUEREPO: to
  PROJECTCOLUMN: '5647145'
  REPOOWNER: shindakun
  SECRET: secret
httpsTrigger:
  url: https://us-east1-golang-slackbot.cloudfunctions.net/issuer
labels:
  deployment-tool: cli-gcloud
name: projects/golang-slackbot/locations/us-east1/functions/issuer
runtime: go111
serviceAccountEmail: golang-slackbot@appspot.gserviceaccount.com
sourceUploadUrl: https://storage.googleapis.com/gcf-upload-us-central1-eeb5af0e-fd09-4fe7-30851592ebba/bcc11c6f-55fc-4d73-864a-6a89813206a6.zip?GoogleAccessId=service-84452958925@gcf-gserviceaccount.com&Expires=1561244032&Signature=QX%2BKy5j6YTA6%D%3D
status: ACTIVE
timeout: 60s
updateTime: '2019-06-22T22:24:37Z'
versionId: '1'

Next time

And there we have it! We’re now live and update any source repos to use our trigger URL. That makes this stage of our issuer complete. New issues will appear in our target repo and on the TODO section of the kanban board!

What should we tackle next time? I’ll have to take a look at the kanban board and see if any ideas jump out at me. Until then feel free to let me know if you spot something to refactor.

You can find the code for this and most of the other Attempting to Learn Go posts in the repo on GitHub.

shindakun / atlg

Source repo for the "Attempting to Learn Go" posts I've been putting up over on dev.to

Attempting to Learn Go

Here you can find the code I've been writing for my Attempting to Learn Go posts that I've been writing and posting over on Dev.to.

Post Index

Post	Code
Building a Downloader Part 01	-
Building a Downloader Part 02	-
Building a Downloader Part 03	-
Building a Downloader Part 04	-
Building a Downloader Part 05	-
Consuming a REST API	src
Continuing REST Adventures	src
Now Sending REST Requests	src
REST API and A Bit On Templates	src
Sending Email Via API Again	src
Let's Get Modular!	src
Let's Get Modular - Again!	src
Building DevLog Part 01	src
Building DevLog Part 02	src
Building DevLog Part 03	src
Building DevLog Part 04	src
Building DevLog Part 05	src
Listing Files By Extension 01	src
Listing Files By Extension 02	src
Dev.to API 01	src
Dev.to API 02	see above code
Issuer

…

View on GitHub

Enjoy this post?
How about buying me a coffee?