Pages are like notebooks

Mads Cordes — Fri, 11 Mar 2022 23:05:39 +0000

The Idea

I've for a while have been tinkering with what I think JavaScript'ers have been missing - namely a way to make Jupyter-like notebooks!

So I started building. Actually been on this since November 12th - so about a month. Lot's of stuff is going to happen still!

The past

Back on the 12th of November it couldn't do much, in fact you were only able to build a page using the source code!

This couldn't spark much joy to any people, so that brings us to the present.

The present

Now you are able to build pages as your heart desires! still missing some documentation on how to use the page-builder, but it's definitely a step in the right direction!

The future

No one knows what the future will bring, not even me for this project! What I do know, though, is that there wil have to be some docs upon how to use the scripting of the pages. Hopefully also a more intuitive way to use the page builder and so on.

I'd love to get some way to work on a saved notebook again, as for now it is not possible to get working on an old saved notebook, which is quite a shame!

The page itself

The notebooks are easily managed by a notion-esque idea-inspired way to build a page - in this case, a notebook.

You start with mostly nothing, then add along all the fields you'd possibly need for a long coding experiment.

It all starts with a box - a magic box one might say. This box allows you to add any element you'd like.

For now though, there are four elements to choose from:

Header - A header element
Text - A text-area element
Code - The snippets to run and global to run from
Comment - A super simple comment - for less text that Text, but not quite a heading

Checkout the /page to get a taste off all the elements. It'll show off all of the elements!

The End

Thanks for reading!

By now you must be super excited to get the pages - notbooks - a try! You without further ado, hasty.dev/page and a demo is also available on hasty.dev/page/demo 🥰

Pst. There's also a shortcut to get a compelely clean slate by navigating to hasty.dev/page/new 💡

Batch importing 6.8k entries in go

Mads Cordes — Fri, 16 Jul 2021 00:28:53 +0000

What it's all about

I've recently decided to import 6.8k entries of crypto currencies into a database - namely Rethink.

I have an old side-hustle, that never got into business, but I still have fun making it. I started on this project four years ago, and just yet decided to pick it back up. Back then I were just getting started with Go, and wasn't a particularly seasoned go-dev.

In this project I had this one function that bothered the heck out of me, as it took forever to run, alas I only had to run it once every time I re-created the database. But still too much waiting time for little moi.

Read along in the next chapter.

The Structs

All of the scructs used in the below code are as follows:

type Coin struct {
    ID          string  `json:"id" gorethink:"id,omitempty"`
    Name        string  `json:"name" gorethink:"name" mapstructure:"CoinName"`
    Symbol      string  `json:"symbol" gorethink:"symbol"`
    Algorithm   string  `json:"algorithm" gorethink:"algorithm"`
    Price       float64 `json:"price" gorethink:"price"`
}

type Data struct {
    Data map[string]Coin
}

The Usual Suspect

This one function, I decided to time using time. This would be done like so:

time go run . -fetch

The fetch-flag were to tell it to populate my database, stupid name now that I think about it, but it's not the one to be questioned as for now.

The timing returned 2.5m, that's a lot of waiting time a busy little bee like me. The total response is below:

go run . -fetch  2.24s user 1.12s system 2% cpu 2:25.31 total

As you see it didn't take much CPU, but a hecka long time! So I decided to re-write it as in the next chapter.

Code

func FetchCrypto(d *db.DB) {
    dat, _ := ioutil.ReadFile("coins.json")

    var data Data
    json.Unmarshal(dat, &data)

    for _, c := range data.Data {
        c.ID = ""
        if c.Algorithm == "N/A" {
            c.Algorithm = ""
        }

        r.Table("coin").
            Insert(c).
            Exec(d.S)
    }
}

The Batch Answer

Now, how would one go along and rewrite such a beautifully old function? The answer might surprise you!

Batching! Why, just because I wanted to see if I could. And success! I done it! It takes batches of one hundred currencies for every go-routine and process that, and waits for the rest to complete.

The timing of this one is as follows:

go run . -fetch  1.89s user 0.81s system 47% cpu 5.683 total

Easy peasy, right? It was - A tad more complicated though, but I love how it turned out.

Code

func FetchCryptoV2(d *db.DB) {
    dat, _ := ioutil.ReadFile("coins.json")

    var data Data
    json.Unmarshal(dat, &data)

    var wg sync.WaitGroup
    batch := 100

    coins := []coin.Coin{}
    for _, c := range data.Data {
        coins = append(coins, c)
    }

    length := len(coins)
    for i := 0; i < length; i += batch {
        wg.Add(1)
        go func(i int) {
            b := coins[i:]
            if len(b) > batch {
                b = b[:batch]
            }

            for _, c := range b {
                c.ID = ""
                if c.Algorithm == "N/A" {
                    c.Algorithm = ""
                }

                r.Table("coin").
                    Insert(c).
                    Exec(d.S)
            }

            wg.Done()
        }(i)
    }

    wg.Wait()
}

Bonus Chapter

I also decided to make use of the slice insertion of Rethink. This is super fast!

Even faster than mine, I haven't tweaked mine to use anything other than one hundred in batch, but I suspect that lowering the batch-size will speed it up a bit.

The timing of this is:

go run . -fetch  1.02s user 0.26s system 28% cpu 4.488 total

Code

func FetchCryptoV3(d *db.DB) {
    dat, _ := ioutil.ReadFile("coins.json")

    var data Data
    json.Unmarshal(dat, &data)

    coins := []coin.Coin{}
    for _, c := range data.Data {
        c.ID = ""
        if c.Algorithm == "N/A" {
            c.Algorithm = ""
        }

        coins = append(coins, c)
    }

    r.Table("coin").
        Insert(coins).
        Exec(d.S)
}

Final thoughts

No error checking has been taken into account, but if you use any of these functions, you probably should, especially in production environments.

Again, you should play around with the batching sizes too, if you want to use anything long the lines of the second function.

Edit

I just did a new test with a batching size of 10, this more than halved the time it took to insert into the database, thus making it faster than the standard one now.

Making it a meer 2.9s.

go run . -fetch  1.67s user 0.49s system 74% cpu 2.917 total

Best,
Mads Cordes

DEV Community: Mads Cordes

Pages are like notebooks

The Idea

The past

The present

The future

The page itself

The End

Batch importing 6.8k entries in go

What it's all about

The Structs

The Usual Suspect

Code

The Batch Answer

Code

Bonus Chapter

Code

Final thoughts

Edit