DEV Community: Charles Clinton Pustejovsky III

Snapshots and Projections

Charles Clinton Pustejovsky III — Sun, 26 Feb 2023 01:44:07 +0000

In Part 2, we built Append and Query functionality for our event store which left us with this list of events:

store.Event{Id:"58f02691-78ed-4ca5-8e59-8f4deb44e063", Version:0, CharacterName:"cpustejovsky", CharacterHitPoints:8, Note:"Init"}
store.Event{Id:"58f02691-78ed-4ca5-8e59-8f4deb44e063", Version:1, CharacterName:"cpustejovsky", CharacterHitPoints:-2, Note:"Slashing damage from goblin"}
store.Event{Id:"58f02691-78ed-4ca5-8e59-8f4deb44e063", Version:2, CharacterName:"cpustejovsky", CharacterHitPoints:-3, Note:"bludgeoning damage from bugbear"}

To make use of the long list of events returned from a Query and to manage the increasingly long list of events, we'll be building functionality to Project and Snapshot our events.

Projection

We could manually take the CharacterHitPoints for each event in the list and add them together to get the current hit points for our character. (8 + -2 + -3 = 3).

To make this easier to use, we can create a projection. A projection takes the stream of events we have and projects it to be read. In this case, you can imagine the value being projected to a UI to show the current hit points of the player character. Project will take the Event id and return an Event with a reconstituted state. Let's first write our test:

t.Run("Project Events from Event Store", func(t *testing.T) {
    aggEvent, err := es.Project(ctx, id)
    require.Nil(t, err)
    assert.Equal(t, 8, aggEvent.CharacterHitPoints)
    assert.Equal(t, "cpustejovsky", aggEvent.CharacterName)
})

With a failing test, we can start on making it pass. We'll need to query our event store and range through the events, reconstituting them into a single state.

// Project takes an id, queries events since the last snapshot, and returns a reconstituted Event
func (es *EventStore) Project(ctx context.Context, id string) (*Event, error) {
    var agg Event
    //Query events
    events, err := es.QueryAll(ctx, id)
    if err != nil {
        return nil, err
    }
    //Reconstitute the event
    for i, event := range events {
        if i == len(events)-1 {
            agg.Id = event.Id
            agg.CharacterName = event.CharacterName
            agg.Version = event.Version + 1
        }
        agg.CharacterHitPoints += event.CharacterHitPoints
    }
    return &agg, nil
}

Now our tests should be passing. As a refactor, we can split up the reconstitution logic from the Project method:

// Project takes an id, queries events since the last snapshot, and returns an aggregated Event
func (es *EventStore) Project(ctx context.Context, id string) (*Event, error) {
    events, err := es.QueryAll(ctx, id)
    if err != nil {
        return nil, err
    }
    agg := reconstituteEvent(events)
    return &agg, nil
}

func reconstituteEvent(events []Event) Event {
    var agg Event
    for i, event := range events {
        if i == len(events)-1 {
            agg.Id = event.Id
            agg.CharacterName = event.CharacterName
            agg.Version = event.Version + 1
        }
        agg.CharacterHitPoints += event.CharacterHitPoints
    }
    return agg
}

Snapshot

As we have more DnD sessions with more and more changes to the hit points of our player character, we will run into trouble handling the growing volume of events. To help with this, we can take snapshots of the state at points in our event log. This will mean the Event Store won't need to query every single event whenever we project it.

To begin with, let's write our test. Our Snapshot method will take a context and a reconstituted event e and return an error:

t.Run("Snapshot should return no error", func(t *testing.T) {  
   e, err := es.Project(ctx, id)  
   assert.Nil(t, err)  
   err = es.Snapshot(ctx, e)  
   assert.Nil(t, err)  
})

There are a variety of ways we could implement this functionality, but for now I'm sticking with DynamoDB's single table design and using the same table for the snapshots as for the standard events. As a result, our Snapshot method is using the Append method to add the snapshot event to:

// using a constant instead of a hardcoded value for it to be easily used in multiple places 
const SnapshotValue string = "SNAPSHOT"
//...
func (es *EventStore) Snapshot(ctx context.Context, agg *Event) error {  
   e := &Event{  
      Id:                 agg.Id,  
      Version:            agg.Version,  
      CharacterName:      agg.CharacterName,  
      CharacterHitPoints: agg.CharacterHitPoints,  
      Note:               SnapshotValue,  
   }  
   return es.Append(ctx, e)  
}

With the Snapshot test passing, our next step should be to see if our QueryAll still works. Let's write the test:

t.Run("QueryAll should not return the snapshot", func(t *testing.T) {  
   queriedEvents, err := es.QueryAll(ctx, id)  
   assert.Nil(t, err)  
   assert.Equal(t, len(events), len(queriedEvents))  
   for _, event := range events {  
      assert.NotEqual(t, store.SnapshotValue, event.Note)  
   }  
})

This test is failing because we haven't modified our QueryAll to skip over snapshots. We can make that modification to the DynamoDB parameters to get the test working:

func (es *EventStore) QueryAll(ctx context.Context, id string) ([]Event, error) {
params := dynamodb.QueryInput{  
   TableName:              aws.String(es.Table),  
   KeyConditionExpression: aws.String("Id = :uuid"),  
   FilterExpression:       aws.String("Note <> :note"),  
   ExpressionAttributeValues: map[string]types.AttributeValue{  
      ":uuid": &types.AttributeValueMemberS{Value: id},  
      ":note": &types.AttributeValueMemberS{Value: SnapshotValue},
   },  
}
//...

Assessing

You can take a look at our current code as of Part 3 here. There is plenty of functionality to add, but I want to address two key problems over the next two parts.

First, the event store can only keep track of a Player Character's hit points. We need to abstract the Event to be able to represent a myriad of different events.

Second, we've tied our Event Store to a DynamoDB client. This is fine for our integration tests but will burden any unit tests for code relying on our Event Store but not focusing on the Event Store's behavior. To fix that, we'll be using interfaces and mocks.

Have any questions or comments? Is there anything I missed or got wrong about event projections or snapshots? Let me know in the comments or reach out to me on Twitter or the Gopher's Slack.

Append and Query

Charles Clinton Pustejovsky III — Thu, 02 Feb 2023 18:08:04 +0000

Setup
Append
Query

In Part One, we went over the basics of event stores, DynamoDB, and Test-Driven Development. Now, we'll build our event store and its most basic functionality.

Setup

First, we'll need to create an AWS Account and get your access key (For better security, follow AWS's advice and set up AWS credentials in your local environment).

Store your access keys in a .env file that is included in your .gitignore.

Next we'll set up our AWS configuration and create a test for our Event Store.

(NOTE: I am using the require and assert packages from the testify repo.)

var EventStoreTable = "event-store"

func TestEventStore(t *testing.T) {
    //Create Dynamodb Client
    ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
    defer cancel()

    cfg, err := config.LoadDefaultConfig(ctx,
        config.WithRegion(os.Getenv("AWS_REGION")),
        config.WithEndpointResolverWithOptions(
            aws.EndpointResolverWithOptionsFunc(
                func(service, region string, options ...interface{}) (aws.Endpoint, error) {
                    return aws.Endpoint{}, &aws.EndpointNotFoundError{}
                },
            ),
        ),
        config.WithCredentialsProvider(
            credentials.StaticCredentialsProvider{
                Value: aws.Credentials{
                    AccessKeyID:     os.Getenv("AWS_ACCESS_ID"),
                    SecretAccessKey: os.Getenv("AWS_SECRET_KEY"),
                },
            },
        ),
    )
    require.Nil(t, err)

    //Create Event Store
    client := dynamodb.NewFromConfig(cfg, EventStoreTable)
    es := store.New(client)
    require.NotNil(t, es)
}

Once we have a failing test we can write the code needed to make this pass, in this case, the Event Store type along with a function constructor:

type EventStore struct {
    DB    *dynamodb.Client
    Table string
}

func New(db *dynamodb.Client, table string) *EventStore {
    return &EventStore{DB: db, Table: table}
}

Different environments will likely have different DynamonDB clients and table names. To handle this, we configure the event store to take the table name and DynamoDB client as arguments.

Event

We described the shape of the event we wanted in Part 1. We'll have that as a struct that holds an id, version, the name of the character, the change to the character's hit points, and a note about what caused this change.

type Event struct {
    Id                      string
    Version                 int
    CharacterName           string
    CharacterHitPoints      int
    Note                    string
}

The two fundamental methods of our event store are Append and Query. We append new events to the event store and query those event from the event store.

Append

We first write our test for the event store's Append method. Based on DynamoDB's PutItem function definition, our Append will need to pass in a context.Context along with the event we want to append. This test will also give us three events to query for later on.

//...
id := uuid.NewString()
events := []store.Event{
    {
        Id:                      id,
        Version:                 0,
        CharacterName:           "cpustejovsky",
        CharacterHitPoints: 8,
        Note:                    "Init",
    },
    {
        Id:                      id,
        Version:                 1,
        CharacterName:           "cpustejovsky",
        CharacterHitPoints: -2,
        Note:                    "Slashing damage from goblin",
    },
    {
        Id:                      id,
        Version:                 2,
        CharacterName:           "cpustejovsky",
        CharacterHitPoints: -3,
        Note:                    "bludgeoning damage from bugbear",
    },
}
t.Run("Append Items to Event Store", func(t *testing.T) {
    for _, event := range events {
        err := es.Append(context.Background(), &event)
        if err != nil {
            t.Fatal(err)    
        }
    }
})

Now we write an append method to satisfy our test:

func (es *EventStore) Append(ctx context.Context, event Event) error {
    input := &dynamodb.PutItemInput{
        TableName: &es.Table,
        Item: map[string]types.AttributeValue{
            "Id": &types.AttributeValueMemberS{Value: event.Id},
            //AttributeValueMemberN takes a string value, see https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_AttributeValue.html
            "Version":                 &types.AttributeValueMemberN{Value: strconv.Itoa(event.Version)},
            "CharacterName":           &types.AttributeValueMemberS{Value: event.CharacterName},
            "CharacterHitPoints": &types.AttributeValueMemberN{Value: strconv.Itoa(event.CharacterHitPoints)},
            "Note":                    &types.AttributeValueMemberS{Value: event.Note},
        },
    }
    _, err := es.DB.PutItem(ctx, input)
    if err != nil {
        return err
    }
    return nil
}

Our test is passing but this Append method will not work for our event store. We need to have optimistic concurrency to make sure events can't be overwritten. As it stands now, a new Version 0 event will overwrite the existing event.

Let's first write the test:

t.Run("Attempt to append existing version to event store and fail", func(t *testing.T) {
    e := store.Event{
        Id:      id,
        Version: 0,
    }
    err := es.Append(context.Background(), e)
    assert.NotNil(t, err)
})

This test will fail with our current Append method, so we'll need to add a conditional to our DynamoDB input:

cond := "attribute_not_exists(Version)"
input := &dynamodb.PutItemInput{
    TableName: &es.Table,
    Item: map[string]types.AttributeValue{
        "Id": &types.AttributeValueMemberS{Value: event.Id},
        "Version":            &types.AttributeValueMemberN{Value: strconv.Itoa(event.Version)},
        "CharacterName":      &types.AttributeValueMemberS{Value: event.Character.Name},
        "CharacterHitPoints": &types.AttributeValueMemberN{Value: strconv.Itoa(event.Character.HitPoints)},
        "Note":               &types.AttributeValueMemberS{Value: event.Note},
},
    ConditionExpression: &cond,
}

Now our test for Append's failure is passing but our Append tests are failing. That's because those three events are already saved to the database. To clean that up, we can write a t.Cleanup function at the bottom of our tests to delete those records and reset the state for us between tests.

//...
    t.Cleanup(func() {
        for i := 0; i < len(events); i++ {
            params := dynamodb.DeleteItemInput{
                TableName: &EventStoreTable,
                Key: map[string]types.AttributeValue{
                    "Id":      &types.AttributeValueMemberS{Value: id},
                    "Version": &types.AttributeValueMemberN{Value: strconv.Itoa(i)},
                },
            }
            _, err := client.DeleteItem(context.Background(), &params)
            if err != nil {
                t.Log("Error deleting items for cleanup:\t", err)
            }
        }
    })
}

Now all our tests are passing, but we still have a problem. How will our users distinguish between failures? Our Append method could fail because the Event Version already exists. It could also fail if there is an internal problem with DynamoDB.

To provide clarity, we can create a sentinel error if the condition failed and have our test check for that. First we can create the error:

type EventAlreadyExistsError struct {
    ID      string ``
    Version int
}

func (e *EventAlreadyExistsError) Error() string {
    return fmt.Sprintf("event already exists for ID %s and Version %d", e.ID, e.Version)
}

Then we add it to our test:

t.Run("Attempt to append existing version to event store and fail", func(t *testing.T) {
    e := store.Event{
        Id:      id,
        Version: 0,
    }
    err := es.Append(context.Background(), e)
    assert.NotNil(t, err)
    checkErr := &store.EventAlreadyExistsError{}
    assert.True(t, errors.As(err, &checkErr))
})

And our tests are now failing. To get them passing, we'll need to add the following error handling into our Append method:

if err != nil {
    //Using the errors package, the code checks if this is an error specific to the condition being failed and, if so, returns a sentinel error that can be checked
    var errCheck *types.ConditionalCheckFailedException
    if errors.As(err, &errCheck) {
        return &EventAlreadyExistsError{
            ID:      event.Id,
            Version: event.Version,
        }
    }
    return err
}

Now our tests are passing again. We have created half of our essential functionality and can move on.

Query

Since we set up a state of three events in our event store, we can write a test for our Event Store to query for them. We know it will need to take an Event ID to query and, like Append, it will need a context.Context to satisfy the DynamoDB API:

t.Run("Query Items from Event Store", func(t *testing.T) {  
   queriedEvents, err := es.Query(ctx, id)  
   assert.Nil(t, err)  
   assert.Equal(t, len(events), len(queriedEvents))  
   for _, event := range events {
       assert.Contains(t, queriedEvents, event)
   }
})

Our tests are failing, and we can begin work on making them pass:

// Query takes a context and DynamoDB query parameters and returns a slice of Events and an errorfunc (es *EventStore) Query(ctx context.Context, id string) ([]Event, error) {  
   var events []Event  
   kce := "Id = :uuid"  
   params := &dynamodb.QueryInput{  
      TableName:              &es.Table,  
      KeyConditionExpression: &kce,  
      ExpressionAttributeValues: map[string]types.AttributeValue{  
         ":uuid": &types.AttributeValueMemberS{Value: id},  
      },  
   }  
   // Query paginator provides pagination for queries until there are no more pages for DynamoDB to go through  
   // See: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Query.Pagination.htm   p := dynamodb.NewQueryPaginator(es.DB, params)  
   for p.HasMorePages() {  
      out, err := p.NextPage(ctx)  
      if err != nil {  
         return nil, err  
      }  
      // The output is unmarshalled into an Event slice which is appended to the events slice  
      err = attributevalue.UnmarshalListOfMaps(out.Items, &events)  
      if err != nil {  
         return nil, err  
      }  
   }  
   // If the slice is empty, then error is returned  
   if len(events) < 1 {  
      return nil, errors.New("no events found")  
   }  
   return events, nil  
}

Most of this code is specific to DynamoDB. In particular, pagination is essential to any queries or scans. They ensure DynamoDB will return more than 1 MB of data back if there is more data than that to return.

At a high level, our Event Store's Query method is:

querying the underlying database
unmarshalling its items to the Event type
ensuring that at least one Event was returned.

Similar to Append, we could add a sentinel error here to help a user differentiate between no event being found and an internal DynamoDB error.

First, write the test:

t.Run("Query returns specific error if no Event is found", func(t *testing.T) {
    _, err := es.Query(ctx, uuid.NewString())
    assert.NotNil(t, err)
    checkErr := &store.NoEventFoundError{}
    assert.True(t, errors.As(err, &checkErr))
})

To get this test passing, we'll create the error and replace the current errrors.New() value with it:

type NoEventFoundError struct{}

func (e *NoEventFoundError) Error() string {
    return "no event found"
}
//...
if len(events) < 1 {
    return nil, &NoEventFoundError{}
}

With that we have created the basic functionality of our Event Store. To see what comes next, lets log the output of our query:

store.Event{Id:"58f02691-78ed-4ca5-8e59-8f4deb44e063", Version:0, CharacterName:"cpustejovsky", CharacterHitPoints:8, Note:"Init"}
store.Event{Id:"58f02691-78ed-4ca5-8e59-8f4deb44e063", Version:1, CharacterName:"cpustejovsky", CharacterHitPoints:-2, Note:"Slashing damage from goblin"}
store.Event{Id:"58f02691-78ed-4ca5-8e59-8f4deb44e063", Version:2, CharacterName:"cpustejovsky", CharacterHitPoints:-3, Note:"bludgeoning damage from bugbear"}

That's not particular helpful and even if it were helpful. We can manually add the 8, -2, and -3 together to conclude that the Character "cpustejovsky" is at 3 hit points. And even if we had the code do that for us, it would take longer and longer to query all the events the more sessions of D&D played.

We will be solving those two issues in the next part when we tackle Projection and Snapshotting.

Have any questions or comments? Is there anything I missed or got wrong about event stores or DynamoDB? Let me know in the comments or reach out to me on Twitter or the Gopher's Slack.

Introduction

Charles Clinton Pustejovsky III — Sat, 14 Jan 2023 00:14:24 +0000

What is an Event Store for?
DynamoDB
- Partition Key
- Sort Key
Test-Driven Development

I first encountered DynamoDB at a company that used it like SQL. As a result, it ran SO SLOWLY. I didn't understand why anyone would use a non-relational database like this if it could be so easily misused and create such costly reads. Thankfully, I later had the opportunity to use DynamoDB to build an event store and see its power on display.

What is an Event Store for?

I recently turned thirty. I could represent that in a database by updating an age value from 29 to 30. But I could also represent it with an initialization event (when I was born) and thirty birthday events each year from 1993 to 2022.

That's a very simple case and wouldn't warrant an event store. However, you may want to consider an event store if you:

need to manage state asynchronously
need to be able to audit and observe changes in states
want to use event-driven architecture

For a simple example, we'll say the Dungeon Master (DM) is a big stickler for the rules of Dungeons and Dragons and wants to make sure all changes to my player character's (PC) hit points (HP) are calculated correctly.

So we'll create an event that has my PC's name, changes to their HP, and a note about what caused the change. But we'll also need:

a unique ID for these events to distinguish these events from any other events related to other PC's HP.
a version to keep track of when each event happened.

Now let's move on to DynamoDB.

DynamoDB

AWS describes DynamoDB as:

a fully managed, serverless, key-value NoSQL database designed to run high-performance applications at any scale.

DynamoDB will preform well for you if you design your schema upfront following a single table design. But that's its own topic which Alex Debrie covers much better than I ever could.

Instead we'll focus on the fundamentals of DynamoDB by building a table that stores a specific kind of event.

The key to understanding DynamoDB is understanding its key. The primary key is the essential, required part of each item in the table. it contains a partition key and an optional sort key. Selecting good partition and sort keys is the most decision you'll make when setting up a DynamoDB table.

Partition Key

The partition key is a string, numeric, or binary value. If you don't take advantage of a sort key, you'll need this to be unique. Regardless, you'll want a partition key with as high a cardinality as possible.

An example of cardinatlity are dice in Dungeons and Dragons.

A D6 die only has six sides and six possible outcomes while a D20 has 20 sides and 20 possible outcomes so a D20 would have a higher cardinality.

With roughly 5.3×10³⁶ possibilities, A uuid is a great candidate both for a DynamoDB primary key and for our event store's id.

Sort Key

The optional sort key allows you to search and sort within items that match a given primary key.

Anything that can satisfy the Interface of Go's sort package can be a sort key. This includes strings, floats, and integers.

For our event store, we'll use an incrementing integer to stand as the event's version.

Test-Driven Development

In the following parts of this series, we'll be building our DynamoDB event store in Go using test-driven development or TDD. The idea behind TDD is to use tests to guide our development. We use them as a specification for the behavior we want for a unit of code or the interaction between different pieces of code. We use the difficulty of tests as a helpful indicator of how rigid our tests may be. If something is hard to test, it's often also not a configurable or modular piece of code to begin with.

For more information on TDD in Go, I highly recommend Learn Go with Tests by Chris James.

Ready to start building this event store? Check out Part 2: Append and Query.

Have any questions or comments? Is there anything I missed or got wrong about event store or DynamoDB? let me know in the comments.

LeetCode Assessments Don't Make Sense

Charles Clinton Pustejovsky III — Fri, 21 Oct 2022 16:53:11 +0000

I've been a professional developer for a few years now. I've worked on monoliths and micro-services, debugged legacy code and built greenfield projects. For all these different projects, my day-to-day has looked similar:

I break down the problem with team members and stakeholders
I work on the problem using my IDE of choice that I've configured to my liking
When I run into issues, I turn to:
- My team members
- Google
- StackOverlow
- Reddit
- Gophers Slack
I often use other people's code as a template or a starting point (even if it's just my past code)
When appropriate, I import 3rd party libraries so I don't re-invent the wheel

As a back-end developer, these problems almost always relate to the following technologies:

Servers (REST, gRPC)
Databases (SQL, MongoDB, DynamoDB)
Data Streams (Kafka, AWS Kinesis, RabbitMQ)

So when I interview for a back-end developer position, why are technical assessments often:

closed book tests (no Google, etc.)
based on data structures and algorithms
using contrived problems
using something other than my own text editor or IDE

Why would companies assess my technical skills with data structure and algorithim questions when the job is about databases and severs? It seems like misplaced priorities. To see what I mean, here's a chart showing how long various events would take if we said a CPU cycle takes 1 second and scaled from there.

Internet calls take much, much longer than operations using the CPU and main memory. Given that, why would companies focus on how well I optimize for "second" and "minute" operations and not for the operations that take "years"?

The answers to these questions vary. Some companies want to follow FAANG's example. Others aren't giving themselves the time to create and maintain a quality technical interview.

I've decided that I won't entertain interviews that use these kinds of assessments.

The questions in this post show a mismatch between the skills LeetCode assesses me for and the skills I'd be doing at the job.

I prefer doing deeper dives into Golang, SQL, DynamoDB, Kafka, and design patterns for distributed systems. I hone those skills because those will provide the most value for companies I work with.

This is risky. I reject a lot interviews as a result of this decision. I decrease my chances of getting a new job. But I've rarely met developers who like these kinds of assessments or think studying for them is a good use of their time. Most people study LeetCode because they have to, because need a job and most jobs have this as an obstacle.

It's past time that we as an industry ask ourselves why we are using these kinds of technical assessments.

If you disagree with me, why do you think LeetCode style assessments are valuable, useful, etc.?

If you agree, what would be a better alternative? My suggestion would be to follow Laurie Barth's guide for Designing a Technical Interview

The Non-Technical Guide to Technical Debt

Charles Clinton Pustejovsky III — Sun, 13 Jun 2021 15:48:17 +0000

Technical debt captured my interest the moment I learned about it. I was learning about programming while working in marketing at a startup with technical debt. I saw the cost of technical debt not only for developers but also for marketing, sales, product, and customer success teams.

As a result, I developed a passion for writing clean code and fixing problems where I saw them. I also started thinking of a way to explain this issue to non-developers. Developers feel the pain of technical debt up close and personal. We understand it like the burned hand understands the hot stove. Since technical debt affects the whole software company, everyone should better understand it.

What is Technical Debt?

Gene Kim's The Unicorn Project defined explains technical debt like this:

Every half-measure and cut corner during the life of a piece of software adds to technical debt. When you solve the immediate technical problem without worrying about the future, you add technical debt.

It's important to realize the "debt" is a debt of time and experience which is very hard to pay back. After talking with enough non-developers, I've come to agree with David Thomas and Andrew Hunt's term for it: "software rot".

Like a neglected tooth can decay or a shoddy house can fall apart, software with too much technical debt can be like a rotten piece of wood.

Reasons Companies Go Into Technical Debt

If software rot is bad, why would any company go into it? This is why the term "technical debt" is useful. A company goes into technical debt for the same reasons they may take on financial debt.

A software company wants to get a feature or product out before the competition. They want to stay on track for a go-to market plan. A small team can often only make this deadline by cutting some corners and duct taping solutions.

This leads to software rot building up. Because the team is still small it's hard to fix the software rot and add new features. Since the code is currently good enough, there's hope that the code will remain that way.

The Costs of Technical Debt

Intense tooth pain can mean a rotten tooth and a root canal. In the same way, the pain developers feel around adding features or updating software points to a real problem with real costs for a company.

I'll illustrate that problem with some Lego blocks and a knitted hat.

Knitted Hat

Here is a hat my wife knitted a while ago:

When I asked her how long it would take to replace the purple with royal blue, she said, "I would need... well... I'd need to start over."

For her, it would be easier to use the current hat as a reference and knit a new one. To untangle the knitted hat and replace the thread color would take even longer.

This is why developers often call bad code "spaghetti code". It's code so tangled that it's easier to look at what it's trying to do and rewrite it from scratch.

Lego Blocks

Here is a structure I made from Lego blocks yesterday:

It would be easy to change out a color, make it taller, make it wider, or even replace all the foundational blocks.

That is because each Lego brick is it's own thing or component. Talk of components, modularity, orthogonality, etc. are all about how to make code like Lego bricks.

Back to Code

This metaphor breaks down because I can build something with Lego blocks faster than my wife can knit a hat. Lego block code, however, takes far more time and effort to build than spaghetti code.

When you need a new feature or a change in functionality, you want Lego blocks. You want things you can plug in, remove, combine, and change without hassle.

But spaghetti code can cause the following symptoms:

Features taking longer than expected
Problems no one can diagnose
Changes breaking unrelated parts of the software.

How to Get Out of Technical Debt

Since technical debt is a debt of time, it takes intentional plans to pay it down.

Technical debt will never feel as urgent as adding a requested feature or fixing a severe bug. I recommend software companies dedicate X% of a sprint to cleaning up software rot. Don't let that X% decrease unless the whole system is on fire.

But feature requests don't come from the ether, they come from clients. What are sales and product teams supposed to tell customers who are impatient for feature X? The keywords I would use are speed and reliability. Untangling spaghetti code and cleaning software rot will almost always make code faster. It will always make bugs easier to spot. In my experience, unexpected errors and lag grind the gears of clients and their users more than a missing feature.

But ultimately, the best way to get out of technical debt is to work together. Together, technology and product teams can balance present needs with long-term stability.

Developers, do you have any analogies or metaphors to explain technical debt?

Product/project managers, what would help you understand developer's concerns about technical debt?

How to Write Asynchronous JavaScript Code

Charles Clinton Pustejovsky III — Sat, 29 May 2021 22:37:18 +0000

How to Write Asynchronous Code in NodeJS

JavaScript is a non-blocking, single-threaded programming language. It will not go from top to bottom, running your functions one line at a time like you would expect.

For example, here is some simple code to read a file:

const fs = require("fs");

console.log("starting");
fs.readFile("/path/to/helloworld.txt", "utf8", (err, data) => {
  if (err) console.log(err);
  console.log(data);
});
console.log("finishing");

You may expect the result to be

starting
<file contents>
finishing

But instead you get:

starting
finishing
<file contents>

This is because JavaScript does not stop. It will keep going down your code while waiting for a process to finish. There are three ways to handle this and I'll go over them from worst to best.

The Humble Callback

To use callbacks for this code, you would do the following:

console.log("starting");
fs.readFile("/path/to/helloworld.txt", "utf8", (err, data) => {
  if (err) {
    console.log(err);
    return; //or throw(err) or something else to strop the function
  } else {
    console.log(data);
    console.log("finishing");
  }
});

**Note: be sure to add a return after an error message and to use if/else to make sure the function does not continue if there is a problem.

Promises

You have to keep nesting callback functions within callback functions which can lead to deeply-nested code that is hard to read, better known as callback hell.

Promises are wonderful additions to JavaScript to rescue JavaScript developers from callback hell.

You can read more about Promises in depth and how to create them on MDN, but here is an example of how to consume them. Most APIs will have some way to use their code as a promise whether it is NodeJS's util.promisify or AWS's .promise() method for most of their API. For this example, we'll use promisify:

const fs = require("fs");
const { promisify } = require("util");
const ReadFilePromise = promisify(fs.readFile);

console.log("starting");
ReadFilePromise("/path/to/helloworld.txt", "utf8")
  .then((data) => console.log(data))
  .catch((err) => console.log(err))
  .finally(() => console.log("finishing"));

You add a .then() for the data, a .catch() for the error, and a .finally() for anything you want to do after the data or error is returned.

Async/Await

Finally we'll come to my favorite way to write JavaScript code, async/await. The async keyword is syntactic sugar that allows a function to return a Promise. So for this example we can use the same ReadFilePromise from the last example. We'll need to wrap this logic inside an async function and call it:

const ReadFileAsync = async(path) => {
  console.log("starting");
  try {
    let data = await ReadFilePromise(path)
    console.log(data)
  } catch (error) {
    console.log(error)
  }
  console.log("finishing")
}

ReadFileAsync("/path/to/helloworld.txt", "utf8")

NOTE: adding async to a function using callbacks does not make it work asynchronously. The function will still be using callback, but JavaScript now thinks it will return a Promise.

You want to wrap your await inside a try/catch to allow for error handling. Speaking of error handling...

How To Do Error Handling

To make sure your function bubbles the error up to the code using your function, throw it!

Let's my ReadFileAsync a function that another function can use.

const ReadFileAsync = async (path) => {
  console.log("starting");
  try {
    return await ReadFilePromise(path);
  } catch (error) {
    throw error;
  }
};

async function main() {
  try {
    let data = await ReadFileAsync("/path/to/helloworld.txt", "utf8");
    console.log(data);
  } catch (error) {
    console.log(error);
  } finally {
    console.log("finishing");
  }
}

main()

Fill Gaps and Learn Faster

Charles Clinton Pustejovsky III — Wed, 14 Oct 2020 11:06:05 +0000

"I know that I know nothing."

- Socrates

We all have gaps. That's not a problem. The problem is we can't see most of these gaps.

I'll give you an example from my liberal arts background. Many scholars think St. Augustine's pneumatology was influenced by Neo-Platonism, but he was in fact more influences by Hilary of Poitiers and Ambrose of Milan.

I'll shamelessly plug my undegraduate thesis on St. Augustine here.

I can almost guarantee there is at least one name or term you'd need to look up to even understand that statement. That's ignorance, that's a gap, and that's okay! You are not stupid for not knowing terms you'd never heard about. You're not stupid for not knowing whether to agree or disagree with that statement (I spent my senior year of college figuring it out).

But I've exposed a gap. It's not a gap you likely have any desire or reason to fill, but I, an outside influence, exposed it. That's what you need.

You will expose few gaps on your own. That's why you need to talk to others. My recommendation is conferences and recorded talks from conferences.

You may find that you don't understand everything the speaker is saying. Good! Write them down. Google them later.

Forge Utah has these talks from the GoWest Conference up on their YouTube channel now.

As an example, I attended a talk on cloud computing in Go two weeks ago. I had to write down so many terms and acronyms and tools I'd never heard of before. After the talk and after Googling what everything meant, I have a much clearer idea of what I need to learn to excel with cloud computing in Go.

If you know of any groups, conferences, videos, and the like that have helped you fill in gaps, share them in the comments below.

How and Where to Start Your Developer Journey

Charles Clinton Pustejovsky III — Sun, 04 Oct 2020 22:02:13 +0000

I started my developer journey around 2016. It's taken me longer than I wanted to get on the path I'm currently on. It took me so long because of the mistakes I made along the way.

I hope the lessons I've learned from those mistakes will help developers just starting their journey.

But let's begin at the beginning. If you want to be a developer, where do you start?

Where to Start

My first mistake was spending too much time looking at all the different languages, trying to figure out which one was "best" to learn. Python? JavaScript? C++? C? C#? Haskell? I couldn't decide.

I'm regularly like Suzie Q in JoJo trying to pick out a dress. I can't make up my mind! Do not succumb to analysis paralysis

I ended up sticking with JavaScript because that's what the company I was working for as a content marketer was using for front-end and back-end. I did this despite knowing I wanted to do back-end. This was mistake #2. JavaScript is a really great and well-used language, but it's used for much, much more for Front-End work than Back-End work.

2019's State of JS really hammered this for me.

How to Choose

So I'll help you avoid these two mistakes with two questions:

1) Do you want to build visual things, stuff that people are going to look at and use? As an example, think about all the things you interact with on Facebook or Twitter or DEV.to.

And

2) Do you want to work with servers and terminals and databases? This is harder to think about since you don't really experience that stuff as an end-user. The short-cut I'd use is if you aren't that interested in what I described in the first question.

If you answered yes to #1 and no to #2, pick JavaScript. If you answered no to #1 and yes to #2, pick Python. If you answered yes to both and are equally excited about both, flip a coin: heads for JavaScript, tails for Python. If you answer is no to both questions or you're really unsure, I'll recommend Python.

JavaScript and Python are both extremely popular, have large communities, and are (relatively) easy to learn.

For Question #1 stuff (Front-End development), JavaScript reigns supreme and even though the work you'd be doing for a company will likely be with React, Vue, or another framework, JavaScript skills are fundamental for your success.

For Question #2 stuff (Back-End development), Python does a great job with Django and Flask. It's also used all over the place for things like machine learning and data analysis.

For JavaScript, I'd recommend learning the basics first. How? Well, I'd Google "Learn JavaScript" and pick one of those to start with (Google your problems is a core skill you want to start working on now). Then, I'd jump into JavaScript30.

For Python, I'd recommend going through Automate the Boring Stuff with Python, a book that teaches you Python basics and then has you build some simple automation scripts.

For either, it is key to get started and to get into the habit of Learning by Doing which is what I'll talk about in my next post.

Let's Go - Book Review

Charles Clinton Pustejovsky III — Mon, 21 Sep 2020 10:33:03 +0000

After getting my feet weth with Test Driven Development in Go, I decided to go through Let's Go by Alex Edwards. My next step is to re-write my web app's backend with Go, so this was the next logical step for me.

Alex shows you how to build a web application from start to finish. He walks the reader step-by-step through:

Setting up a server
RESTful Routing
Middleware
Database Connection
Templating
Testing (Unit, E2E, Integration)

He shows you from the very beginning how to do all this in a way that will keep your web app scalable, composable, etc. I really appreciated this since too many tutorials show you the simple way first and only later add in the clean up and refactoring. I find this way is better for setting up good habits.

While sprinkling advice and best practices throughout the book, He shows the reader exactly which files to create where, and where to put each bit of code for the web app. This is great for making sure you never get lost while following along.

Because all his code samples have in-depth comments explaining what each thing does, even if you mindlessly copied and pasted everything, you'd still end up with a great web app to go back and reference.

This is what I'm talking about. Each new piece of code or update includes comments like this.

If you're on the fence, check out his blogs like this one on interfaces to see if you'll like his teaching style.

But I highly recommend buying and reading this book if you're interested in web development with Golang. It is absolutely worth the price. Check it out!

Beginner Devs Should Use Linux... Maybe

Charles Clinton Pustejovsky III — Sat, 19 Sep 2020 00:01:50 +0000

CAVEAT #1: If you rely on software that is only on MacOS or Windows 10, do not switch to Linux.

CAVEAT #2: This is just my opinion on how beginners could best develop problem solving mindsets by using Linux day-to-day. No developer is any less of a developer based on their choice of OS. You are not a bad developer for not being a terminal junkie.

I switched to Linux when I was still a complete newb in 2013. I didn’t even know how to make a live USB. Somehow I was able to track down an actual CD-Rom of Ubuntu 12 from a Linux magazine at a Micro Center and use it to install my first Linux distro.

the magazine looked something like this

Why did I switch entirely to Linux when I was this new to tech? Well, I originally installed Ubuntu alongside Windows 10 to better learn Linux and the terminal. However, when I ran into trouble with Ubuntu, I’d reboot my computer and do it in Windows because it was comfortable.

No More Half Measures, Walter... I mean Charles

I decided to completely remove that option and uninstall Windows and I’m a better programmer because of it.

Why? Because programming is about problem solving, googling the error message, etc. And when you have to problem solve just to watch Netflix (which was the case in 2013), you pick these skills up quickly.

It’s a sink or swim approach which might not be good for everyone, but if you’re like me, I recommend it. You’ll have every opportunity to lose your fear and anxiety about the BASH terminal which will help you in your journey. Without fail, you’ll have to at some point pop the hood and use the CLI (command-line interface), and it's nice to not be intimidated by it.

The second reason I'd recommend this to newbies is financial. Most of the dev tutorials I've come across in the last six years are assuming you're using a Mac which can cost quite a lot. Windows machines are cheaper but the dev env is 1:1 to MacOS. You can install Linux for free on your Windows machine and since Linux and MacOS are both Unix, the dev env is very nearly 1:1.

This Apple slogan has aged like milk

But what do you think? If you have any suggested operating systems for a new developer or any tips and tricks to tackle tutorials written for MacOS users with a Windows dev environment, share them in the comments.

Learn Go with tests - Book Review

Charles Clinton Pustejovsky III — Mon, 07 Sep 2020 14:55:12 +0000

After going through The Tour of Go and building a small Twitter bot in Go, I wanted to get more serious with this language I love.

So I decided to move on to Learn Go with tests by Chris James (quii). I began learning Test Driven Development (TDD) with NodeJS a few months ago and found that it forced me to break down problems and to think about what I wanted ____ to do which resulted in writing better code.

Here is what the beginning of Learn Go with tests promises:

I think it achieves all of these goals very well.

Quii takes you through the fundamentals of Golang in the first half of the book and then takes you through the process of building a fully functioning application for both the web and a command-line interface (CLI).

You'll acquire the muscle memory for TDD. Each chapter ingrains the process of writing tests that fail, making them pass, and then refactoring the code and tests.

This book also includes many asides about the whys of TDD along with external blog posts and resources that relate to both Go and TDD. This provides additional motivation to keep learning this methodology and good starting points for continuing education.

The only issue I had with the book was in the second half of the book as the application became larger with many files. It was sometimes unclear where a code snippet should go or what should be changed.

If this is an issue you encounter, I'll recommend cloning the Learning Go with tests repo and opening it in another window of your text editor or IDE. With that, you can search through to see where a specific code snippet goes if you're lost.

So if you're new to Go or new to TDD, I highly recommend reading Learning Go with tests.

The Power of Ctrl+Click

Charles Clinton Pustejovsky III — Fri, 14 Aug 2020 21:26:15 +0000

(For JetBrain IDEs, this all works with Ctrl+B or CMD+B and it already opens in a new tab)

Ctrl+Click (CMD+Click if you're using MacOS) in VS Code is amazing. If you're working with someone else's code either because it's imported from a library or because you're working on a code base you didn't create, you'll often be lost as to what x does or how to use y.

So you'll turn to documentation or a README.md, but sometimes those are incomplete or don't provide the answer you're looking for.

Take the Go code I'm working with right now. I'm using code from github.com/dghubble/go-twitter/twitter. dghubble's README is helpful but I couldn't find all the potential parameters I can pass into the UserTimeline function. So what should I do?

Simple. I ctrl click it and I'm taken to where the code lives. I can see all the types of parameters I can pass in.

This will work with any code you're working with as long as it's defined somewhere else in the workplace.

Finally, to make sure you don't lose your place in your own code when ctrl clicking, follow the advice from this Stack Overflow post to make sure files always open in a new tab in VSCode.

Now you're all set. I only realized I could do this a month or so ago and my productivity has skyrocketed. Good luck!