DEV Community: Caleb Schoepp

Highest impact reading for intermediate level programmers?

Caleb Schoepp — Mon, 17 Aug 2020 20:12:52 +0000

In your opinion, what is the highest impact content an intermediate level programmer can read?

How to Organize Your Code

Caleb Schoepp — Sun, 31 May 2020 23:11:00 +0000

When I first started programming, I struggled a lot with how I should organize all my code. I needed an orderly way to keep track of the tutorials, code snippets, and side-projects I was rapidly accumulating. It's definitely a tough problem.

Thank you to Dan Fletcher for his blog post on this very topic. He helped me figure this out when I was starting out. The following is a summary and rehashing of his fantastic ideas.

Don't

Make heavy use of nesting: At first, it seems like nesting is the key to good organization. This is wrong for two main reasons. First, nesting makes long filepaths which makes navigation with the command-line inconvenient. Second, deeply nested directories impose too brittle of a hierarchy. Eventually you will have files that can live in multiple places within the hierarchy. Then the hierarchy becomes inconsistent and loses its value. Keep things simple and avoid nesting too deeply.

nesting
└── 2020
    ├── feb
    │   ├── big-projects
    │   │   └── project-1
    │   └── small-projects
    │       └── project-2
    └── mar
        ├── big-projects
        │   └── project-3
        └── small-projects
            └── project-4

Organize by language: For simpler projects, this is a logical conclusion. But, as projects grow in complexity they will start to make use of many languages. It is not uncommon for my projects to involve 3-4 different languages. This makes it impossible to know which directory a project should live under.

by-language
├── go
│   └── project-1
├── javascript
│   ├── project-2
│   └── project-3
└── python
    └── project-4

Do

Keep everything in one place: I have a single directory on my computer that all my programming work lives under. Get creative with the name! I call mine the batcave. What would you call yours?

Use a flat structure: Use a single directory for every project. Each directory contains everything that the project needs. These project directories are the level at which I like to track things with Git i.e. every project directory would be its own GitHub repository.

Keep the odds and ends together: I like to group the really small stuff together. I'm talking about the code snippets and the tiny test scripts. For example, when I started learning JavaScript I made a directory called javascript-snippets. Anytime I wanted to quickly try something out I could make a file and toss it in that directory.

batcave
├── javascript-snippets
├── project-1
├── project-2
├── project-3
└── project-4

Sticking to these do's and don'ts will go a long way to keeping your code organized. Remember though, these are only guidelines — sometimes it will make sense to break them. Now go forth and code!

If you enjoyed this article, check out my blog for similar content.

Storing Empty Interfaces in BigCache

Caleb Schoepp — Mon, 25 May 2020 22:30:08 +0000

This was also posted on my personal blog

Recently at work, I was tasked with adding some caching to one of our Golang services. The particular service required incoming requests to provide an API key for authentication. So on every request, the service was making an extra round trip to the database to verify the API key even though it was usually the same key. Not great. Implementing the cache ended up being much harder than I thought it would be.

After some research and exhaustive debate among the engineers, we decided that BigCache fit our needs best.

Here’s the catch. The signature for the set method in BigCache is Set(key string, entry []byte) error. It expects you to store a byte slice. But, we wanted to store a struct with multiple fields that represented the API key. We might have been able to just store the bytes of the actual key this time. But this would just be putting off solving the problem. We needed a signature more like those found in other Golang cache implementations, Set(key, entry interface{}). This would let us store anything we wanted.

The obvious solution to this problem is serialization. If we could serialize an arbitrary struct into a byte slice, then we could store anything. To actually use the struct we stored, we could just deserialize the byte slice we get from the cache. Serializing the struct is as easy as importing any number of available encoding libraries in Golang. But now comes the headache. When we deserialize the bytes, how will the language know the shape of the struct to put the data in? Turns out encoding/gob, the Golang specific serialization library, has this capability.

I highly recommend you go read the blog post by Rob Pike that talks all about Gob; it’s a great read. In short, Gob is a Go native way of serializing data and it also comes with the ability to serialize interface types. To enable this you need to register your type with the aptly named register function before serializing it. I ended up getting stuck here, because any code sample for register() I found would always register a singular concrete struct or interface; I needed it to register an arbitrary interface{} type. With a little messing around in the Go playground I discovered it can do that too.

// What most code samples did
type foo struct {
    bar string
}

gob.register(foo{})

// What I needed and discovered is possible
var type interface{} // Could be anything

gob.register(type)

Putting It All Together

With the problem of storing an arbitrary struct as bytes solved, I'll show you how I put it all together. First we want an interface for the cache that the rest of the system can interact with. For a simple cache we don’t need much more than get and set methods.

type Cache interface {
    Set(key, value interface{}) error
    Get(key interface{}) (interface{}, error)
}

Now let’s define the BigCache implementation that will fulfill the above interface. First we need a struct that holds the cache and can have methods added to it. You could also build other things like metrics into this struct.

type bigCache struct {
    cache *bigcache.BigCache
}

Next are the implementations of the get and set methods. Both methods assert the key is a string. From there, get and set are contrapositives of each other. One serializes a value and stores it. The other retrieves a value and deserializes it.

func (c *bigCache) Set(key, value interface{}) error {
    // Assert the key is of string type
    keyString, ok := key.(string)
    if !ok {
        return errors.New("a cache key must be a string")
    }

    // Serialize the value into bytes
    valueBytes, err := serialize(value)
    if err != nil {
        return err
    }

    return c.cache.Set(keyString, valueBytes)
}

func (c *bigCache) Get(key interface{}) (interface{}, error) {
    // Assert the key is of string type
    keyString, ok := key.(string)
    if !ok {
        return nil, errors.New("a cache key must be a string")
    }

    // Get the value in the byte format it is stored in
    valueBytes, err := c.cache.Get(keyString)
    if err != nil {
        return nil, err
    }

    // Deserialize the bytes of the value
    value, err := deserialize(valueBytes)
    if err != nil {
        return nil, err
    }

    return value, nil
}

Finally the encoding/gob serialization logic. Beyond the use of register() this is the fairly standard way to serialize things in Go.

func serialize(value interface{}) ([]byte, error) {
    buf := bytes.Buffer{}
    enc := gob.NewEncoder(&buf)
    gob.Register(value)

    err := enc.Encode(&value)
    if err != nil {
        return nil, err
    }

    return buf.Bytes(), nil
}

func deserialize(valueBytes []byte) (interface{}, error) {
    var value interface{}
    buf := bytes.NewBuffer(valueBytes)
    dec := gob.NewDecoder(buf)

    err := dec.Decode(&value)
    if err != nil {
        return nil, err
    }

    return value, nil
}

And with all of that we’ve managed to store interface{} values in BigCache. Now my team's service is a bit more efficient. Pretty cool! If you are looking for a more comprehensive implementation, checkout my gist.

If you enjoyed this article, check out my blog for similar content.

The Importance of Maintaining Dev/Prod Parity

Caleb Schoepp — Wed, 13 May 2020 05:49:42 +0000

This was also posted on my personal blog

One of my side projects recently died on the vine and it didn’t have to.

The project was a web service to convert an email newsletter into a PDF for offline reading where all the links would still work. This was done by pre-rendering the content of the links into the PDF.

Generating the PDF required a plethora of tools. I used Puppeteer to render links as HTML documents and convert them to PDFs; a mix of the Python PDF libraries PyMuPDF and PyPDF4 to merge and link the generated PDFs; and GhostScript to compress the final result.

A World Without

At the start, I hacked together a mess of Python, Node.js, and Bash scripts to prove I could get it to work. When it came time to try and deploy it in a more robust fashion, I had three obvious choices for hosting — a VPS like Digital Ocean, a PaaS like Heroku, or something serverless like AWS Lambda.

Right out of the gate I knew Heroku wasn’t going to be an option. The process I had developed required using Puppeteer, which is really just a JavaScript wrapper around a Chromium binary. PaaS offerings are generally too sandboxed to allow you to run arbitrary binaries alongside your code. I learned this the hard way at a previous internship where I spent an entire week trying to run Google Lighthouse on Azure’s PaaS. Suffice to say I was not successful.

The remaining options were to use a VPS or serverless. I knew a VPS would have been more simple to deploy to. A task queue and worker model would have let me use what I already had. It could even run on one machine. Dirt simple.

Instead, the allure of the sub-second billing and “infinite scale” tempted me to use a serverless offering. Both these reasons shouldn’t have mattered to my zero-user side project. Regardless, I chose to use the Serverless Framework hosted on AWS Lambda. I came to regret it.

Things went okay at first. Spinning up a few Lambdas with the Serverless Framework is a breeze. But the deployment rapidly grew in complexity. First, my build step needed a bespoke docker plugin so that I could use one of my PDF libraries. Next, I had to find a way to package in binaries with my deployment so that my Lambdas could use them. Once I had the individual functions working I realized I would need to coordinate them somehow so that I could robustly handle failures. No trouble, just add a plugin to start using AWS Step Functions.

I added all these tools/plugins/shims to be able to solve my problem, but they were starting to become a problem. My deployment times had grown to be as long as 10-15 minutes because the build step was so bloated. This alone isn’t terrible, but the straw that broke the camels back was that I couldn’t test my changes locally.

Without the ability to test locally, development ground to a halt. Every syntax error, typo, and silly accident was locked behind 15 minutes of waiting. So much for rapid iteration. Over time, I lost interest and the project died.

Dev/Prod Parity

If I had been able to test my project locally it wouldn’t have died. Testing changes locally would have shortened my feedback loop. 30 seconds is much more manageable than 15 minutes. In other words, what killed my project was a failure to maintain Dev/Prod parity.

Dev/Prod parity is the idea that your local testing environment should be as close to identical to your production environment as possible. I like the description of it from the Twelve-Factor App. If you aren’t already familiar with the idea of a Twelve-Factor App, you should go check it out - it’s great.

My failed side project is an extreme example of losing environment parity; in fact, by the end, there was no parity. The only working environment was prod. But even in less extreme cases, differences in environments can start to cause big problems.

Key Takeaways

It’s clear to me now why Dev/Prod parity is so valuable. There are two ways it will change how I work on future projects.

First, I’m going to be more cognizant of how new features break Dev/Prod parity. When differences arise, I shouldn’t rush ahead with the feature anyway. My story is a clear example of how pushing the fix off can snowball and cause real problems. Instead, I should find a way to implement the feature without breaking parity. If this isn’t possible, it’s time to reassess how important the feature is. I want to work in a world where saying, “It runs on my machine”, also means that it runs on the production machine.

Second, I need to start thinking about Dev/Prod parity from the very start of the project. Even before I write the first line of code. In hindsight, choosing to use AWS Lambda was a mistake for my particular situation. The tooling for doing advanced workflows with AWS Lambda is still very nascent and set me up for failure. Sure it’s easy to locally emulate a simple CRUD app with only a few functions. But local emulation is untenable once you need binaries, Step Functions, and Docker build steps.

My side project died on the vine, but yours doesn’t have to. With the discipline to maintain Dev/Prod parity, you can avoid the issues I had.

Thanks for reading! If you liked what I had to say you can find more of my writing on my personal blog.