DEV Community: Dave Smith

That Conference in Wisconsin

Dave Smith — Fri, 12 Apr 2024 12:40:19 +0000

I went to That Conference in 2023. No, this is not a recap of my conference experience from 8 months ago. I’m looking forward to the next one. Registration for 2024 is open, and the speaker lineup has been announced. I will be there and I think you should too.

I’ve attended That a few times in the past and was trying to remember which years, but of course I’m drawing a blank. According to my email, I was a camper there in 2013, 2014, 2015, 2018 and of course 2023.

Summer Camp for Geeks

That Conference is a tech/dev focused conference at an indoor water park in the Wisconsin Dells. They say it is summer camp for geeks. Not summer camp like “Salute Your Shorts” (does anyone else remember this Nickelodeon show from the 90s?) summer camp, but I definitely get a Summer camp vibe when I’m there, with uh, much nicer accommodations. Each morning is a large group breakfast followed by a keynote speaker to start the day (I think this is where I get the summer camp vibe).

Tech-wise, the content is more .NET, JavaScript, Cloud, dev tooling and tech adjacent topics like, communication, leadership, creativity, state of job market, etc. The content is good at following trends as they are current (thing about these - SPA, IOT, Crypto, API, AI), and keeping foundational topics (think about these git, C#, JavaScript, testing), plus a few topics that might surprise you.

What Make That Conference Awesome

Besides the great speakers and content, there is so much more than just learning and geeking out. There is a game night (bring all your cards and board games), a pig roast, a night at the water park, and a morning 5K for all the runners. There is a family track for children, so bring your entire family. There is a water park, they won’t be bored. This is not your typical conference where you sit in your hotel room after the last session ends, you have an opportunity for fun everyday.

Memorable Topics

You know how I said content was more focused on .NET, JavaScript and dev tooling. Those are definitely foundational themes you’ll find quite often, however, some of my most memorable experiences include topics outside of those. There are a few topics that I either first learned about at That, or topics where the speaker did an excellent job of doing a deep dive. Here’s a small sample of topics that stick out in mind, some of them almost 10 years later.

Docker in 2014. I remember hearing about it at That Conference for the first time. I had heard the word, but never spent any time trying to learn until then. I went with some co-workers and we questioned how it was different than VMs. I had no idea how big it’s impact would be at the time.
I first learned about Elixir in 2015 at a talk by Ryan Cromwell. I was immediately intrigued by the language and the platform it runs on. I have since spent some time to learn Elixir and really find it one of the more fascinating functional languages
The same year (or maybe 2014) I listened to a deep dive on the git internals from Ed Thomson, one of the git for Windows developers. At the time, I knew relatively little about git. This really opened my eyes to how a source control system manages files across versions and branches. This is what really moved me away from TFS and towards git.
A coworker (Mike Harris) was giving a talk on the next version of C#. I forget the feature he was talking about, but the feature results in the compiler re-writing a goto in the generated. One camper in the room questioned Mike on this, didn’t like the answer and just walked out of the room. I guess the dude was principally against goto statements. I though it was funny and Mike handled it well.
Game Development on the NES, by Kevin Zurawel. This might be my favorite talk ever. It has absolutely nothing to do with my job. I knew this was on the schedule and I spent a few weeks prior to the conference reading up on writing 6502 assembly. The talk was so engaging. You could tell the speaker did this out of pure joy and just enjoyed it. I could have listened to this topic for hours.
Imposter Syndrome. This was an “Open Space” topic put up by a co-worker. I went to the talk out of curiosity - I don’t feel imposter syndrome very often and wanted to hear how others felt about it. Almost all of the 20+ people at the open space had a chance to speak. It was an amazing 45 minutes of hearing how much of a struggle this is for other. I spent some time reflecting on how I might unintentionally cause others to fall into this feeling.
2023 was the year of Rust for me. I attended an all day workshop on developing at API with Rust and Axum. I still struggle a bit with Rust, but it was really awesome to plug in all the pieces of an API in a difficult language where I was definitely a beginner.

Don’t Do The Typical Conference Thing

Speakers and keynote talks are great, but sometimes the best content at That Conference is at the Open Spaces sessions, and talking one-on-one with other campers. If you aren’t familiar with the Open Spaces concept, its a large conference hall with tables and chairs arranged for small groups to discuss a topic. Topics could be anything - JavaScript support club, NeoVim users, career help, horses, etc. These topics and sessions are not planned prior to the conference. Campers post a topic that is important to them, and other attendees show up. This is your chance to turn the typical conference experience around and become an active participant instead of passively consuming a session (no offense speakers). I highly recommend this for anyone, you’ll probably be out of your comfort zone and that’s probably a good thing.

Find people, make a connection. This is a bit harder for the introverted folks, myself included. Apart from Open Spaces, this is an excellent way to make a top notch conference experience. That Conference refers to this as the “Hallway Track”. Find someone and strike up a conversation. Do this on the first day. You now have someone to compare notes with on sessions for the day past and sessions coming up. Connect on LinkedIn or social media. You might find someone who inspires you or someone who is inspired by you. You might find someone with similar interests. We may all be tech geeks, but even tech geeks like to socialize.

Final Thoughts

Going to a conference is a lot of work. The days extend longer than the normal working day, and you may still have some obligations with your family or back at your day job. That said, tech conferences can be very rewarding. I have been to enough conferences to know That Conference is something special. Yes you will learn, but you can do so much more at That Conference. As a professional software developer, I could not recommend this more. I discover enough new ideas During the conference to fill the time until the next year. I find That Conference rewarding and I find myself leaving each time feeling fulfilled. I will be there this year and hope to see you too.

One Billion Row Challenge: way more than I thought

Dave Smith — Fri, 01 Mar 2024 13:32:43 +0000

I found it like many other on January 1st, 2024. The One Billion Row Challenge. I was immediately hooked on the idea of doing this. I was coming off a disappointing non-finish of the Advent of Code and looking to redeem my incomplete effort. This was also the perfect opportunity to write some Go code and do my best to make it fast, like let's break some rules and make bad decisions. This is not intended to be about Go, it is just the language I used for the challenge. The Go terms and concepts used here apply to many languages.

The Challenge

If you haven't heard of the challenge, visit the site linked above 👆 and check it out. The quick and dirty version of it is, given a file with one billion rows of weather station data, find the min, max and average from each weather station. Each row looks something like this City Name;23.9 or City Name;-15.6. Use the standard library in your language of choice, no third-party packages. Simple enough in concept, right?

How It Started

My approach seemed to naturally go to creating a map[string[]float64]], read the data from the file into the map, then do the calculations. I'd use this as my starting point and iterate from there.

I went for it and started coding. Read a line from the file with a bufio.Scanner, parse the line, put it in a map, after reading the file, use math on all the results, and output it into a formatted string. Easy. I didn't want to wait for all 1 billion lines for my early tests, since I would be iterating over this. I used a 1 million line file to make the testing fast. Less than two seconds for a million rows. Not bad.

I created the billion row file and ran against the big file. I waited for minutes ⏳. This was obviously not very good. I was hoping for under 30 seconds, and I have no idea where I came up with that math.

I had a place to start and compare.

What Did I Learn from One Billion

Stating the obvious here, one billion is a lot. Things changed on me when I went to 100 million rows. One billion was yet another significant leap. What was fast and easy soon was slow and inefficient at one billion. I learned that even the smallest change can have a profound impact when the scale is large enough. Which operations did I touch that had a large impact at a scale of one billion?

Parsing numbers from a string. It turns out there are many ways to do this. Not all are similar. My final solution scanned bytes from input to return an integer instead of a float.
Splitting strings with a delimiter. My solution here? Again, scanning bytes to split a byte slice.
Reading data from a file. Line by line, multiple lines, chunked bytes? I landed on reading chunks of a byte slice using the bufio.Reader.Read function. I played with the buffer size and had similar results in the 4 - 32 MB buffer size. Anything outside that range caused slower reads.
Map lookups. Not as fast as I thought. Can I do better with hashing. I achieved faster map lookup times when I hashed the weather station name, at the cost of additional complexity. I backed out the change and hope to revisit it.
Division. Yeah, the math one, do it a billion times or just a couple hundred? I didn't catch this until I read other solutions and blog posts. I was calculating the average for a weather station each time I added a new measurement. This involved float division and I found this to be slow enough to be noticeable.
Concurrency. Of course. One goroutine to produce chunks from the file, and a set of worker goroutines to process the chunks.
Allocations. Is this operation causing a heap allocation or is it on the stack? I know you are supposed to let the Go compiler determine where data is stored, however I made every attempt to limit heap allocations.

How It Ended

I wanted to do a deep dive of the code and the steps I took. I'm not doing that. You can see my code on GitHub if you are interested. There are some really good posts detailing the iterations to get faster and faster, this one by Shraddha Agrawal is fantastic and worth your time to read. I chose to end this after getting it to about 80 seconds. I stopped working on it because I had learned a significant amount of Go and how to make it fast. I came for the challenge and walked away with more knowledge than I planned.

Deep Thoughts

How the heck do databases do stuff like this? Seriously. I was only doing a min, max and average on a simple dataset, large but simple. How do database engines do this for an arbitrary table. I have a new appreciation for databases now. I'd like to learn how database queries do this.

Honestly, I can get away with writing some real crappy code and still process a million rows fast. Maybe not that crappy but starting simple is often fast enough. Especially with Go.

I can do a lot of stuff with only the Go standard library. I never even thought of reaching for a third-party library. Concurrency is built-in along with everything else needed for this problem.

This is the perfect challenge for anyone learning a new language, or looking to do a deep dive on performance or dig one level deeper. Some of the thoughts and questions in the list above forced me to look at the implementation of the function in the standard library. I found many instances where the function in the standard library was not the fastest. String splitting, parsing numbers from strings and a few others. They are probably the best for general purpose use if you aren't processing one billion rows. Regardless, I now know how strings are split with a delimiter in the Go standard library.

Closing

What did I learn? Start simple and measure everything. Get good at profiling and tracing tools. Question everything. I made too many assumptions just using prior experience as my guide. I never thought to question floating point number math. I should have. Learn the implementation of your language and its standard library. Learn one level deeper than where you work, there are so many amazing things just beneath the surface. Finally, sometimes the simplest solution is good enough most days.

The Magical Elixir

Dave Smith — Wed, 28 Feb 2024 03:56:04 +0000

I first saw Elixir at That Conference in Wisconsin sometime between 2015 and 2018, I really can’t remember. Ryan Cromwell was giving a talk on it and I was fascinated by it in less than 1 hour. I set out a goal to learn the language. You’ll notice that it is 2024, so you see how that went.

Why Elixir?

There is about a 0% chance I will use Elixir in my current role at work, so why go through the effort. Elixir is on a different spectrum from my primary languages of C#, JavaScript, and Go. It’s functional, it’s dynamically typed, there are no classes, interfaces, or for loops. I find it beneficial to learn how other languages solve problems in a different way and this one definitely checks the box for ‘different’. My hope is to see how I could have solved prior problems with a different mindset. I am not interested in determining if Elixir is better or worse than languages I already know. I want to see how it is different and where the differences can be beneficial to my thinking and reasoning in the future. Last reason for learning Elixir, I find this fun - I have been writing code in some form for over 20 years and never tire of learning new things. I know languages are not the most important part of software engineering, but I find the diversity in languages fascinating.

The book

Reading book. The book is dense, in a good way. This assumes you have experience in at least one other language as there are allusions to concepts in other languages. The first 3 chapters cover the Erlang platform, language constructs, basic types, and higher order functions. I am not going to regurgitate the book, but take note of things are foreign to me based on my experience.

After reading about a third of the book. I felt this was an appropriate time to review my notes from what I read, write some code and reflect on it.

Initial Thoughts

I have really never done a deep dive into a functional language. FP definitely has its detractors, but I'm determined to understand why the haters don't like it and why the fanboys think its the best. So far the functional style is, different. I understand the constructs and see it as almost declarative instead of the traditional imperative.

Dynamically typed language. I have always used statically type languages (and JavaScript, where ever that falls in the spectrum). I'm supposed to defend and say statically typed is better, right? I think I need to embrace the dynamically typed life for a bit before making an opinion.

Elixir has some features that are new to me and also fascinating. Atoms, macro everything, pipe operator, tail call optimization and pattern matching. These feature really open a new world of thinking when it comes to writing and reasoning about code. Event the simplest of them, atoms open up so many ideas of how I could have used something similar in the past. I'm really excited about these language features.

There are a few features that are 'missing' from the language. for loops and no arrays. I know there are recursive lists that will the role, but arrays and for loops seem so primitive and basic to every language I have used. I'm sure I will learn the idiomatic ways to process data with lists instead of array, but it will take some time.

What's Next?

I'm spending some time on the Elixir Playground just to get comfortable with the syntax. I plan to continue with the book and start writing code in an IDE.

If you have advice regarding Elixir or just want to share your experience, I'd love to read your comments to this.

Practice Your Skills with Code Katas

Dave Smith — Fri, 09 Feb 2024 14:48:07 +0000

A few years ago, I started a weekly code kata at work with a small group of co-workers. My motivation at the time was to help an intern who was struggling with the "real" work. I figured code katas were a good practice. You can try new things, refine what you know, and can't break anything so there is only upside. The intern moved on and this small group of developers still meets every Friday at 10:30. It is the best part of the week, or so I have been told by some.

CodeWars

The platform I chose for the weekly katas is CodeWars. CodeWars is a collection of coding challenges spanning 50+ programming languages. There are thousands of katas, each created by CodeWars members. Each kata gives you a problem description, 2 sets of unit tests, and a difficulty ranking. Your code must pass all the tests. You are provided with an in-browser editor (supports vim motions!) and the build/test process is handled behind the scenes for you. There is literally nothing you have to set up. If you haven't used CodeWars before, give it a try.

What We Practice

We try to stick to katas with a difficulty rank of 6 but will sometimes attempt a 5 (smaller numbers are harder). We can usually work through those in an hour of less. Examples of katas we have done include "Linked Lists - Front Back Split", "Linked Lists - Shuffle Merge", "Tic Tac Toe Checker", and "Maze Runner" (links below). Some of these are truly straight forward solutions, and some will sound simple when reading the description only to have a surprise in the second set of hidden unit tests. Often times, there are complicated edge cases, or extremely large inputs that will cause the naive solution to timeout.

How We Practice

I have been asked by some if we are enforcing best practices or a certain coding style, or if we are choosing katas that are directly applicable to the work we do for our job. In general, the answer to these questions is "no." We chose a kata that looks like an interesting problem to solve. We solve it as a group, or we'll break into small groups when more the 5 show up. The working solution usually contains code following some accepted standard or style, but that is never the goal. We occasionally explore parts of the language that are most definitely not a good practice (looking at you JavaScript eval()). Sometimes you have to write bad code to know why it is bad. After completing the kata, we can see the solutions other CodeWars members have submitted. This is often an enlightening experience. You can see how others solved the problem more efficiently or with clearer code, or used some really awful code (I swear, some people use RegEx for every problem, wtf).

Keep Practicing

This started with the intention to help an intern learn JavaScript and C#. That was three years ago. It has evolved into a fun way to refine our skills without worrying about long term support and maintenance. For some, learning the language is the goal, for others explaining in simple terms why "algorithm A is a better choice than algorithm B in this instance" is the goal. For me, the goal of this is deliberate practice, and getting better by repeated efforts.

These are some katas we have done recently. The last one in the list gave us fits.

Maze Runner
Bouncing Balls
Tic-Tac-Toe Checker
Linked Lists - Shuffle Merge
Linked Lists - Front Back Split

If not 'git add .', then what?

Dave Smith — Tue, 06 Feb 2024 02:00:43 +0000

Countless times you have been told to never use git add . when staging files. You've been to conferences where git experts say you shouldn't use it. Tech people of Twitter also say it. Again and again, you hear never use git add . to stage your files.

Habits

I get it. I don't want to accidentally stage a file, then commit it. Muscle memory is often times the culprit here. I don't even realize I'm typing git add . until I hit enter. If you are like me, you have a git undo alias mapped to reset --soft HEAD^. I can't count the times I have accidentally staged a file, then rapid fire commit with some message. I have done it enough times to justify a git alias.

I try to do as much work as possible in the terminal. On occasion when I need to do a highly selective commit, I will often jump to an IDE and use that to do selective adds. It's not part of my natural workflow, so it feels clunky.

Then What?

If we should never use git add ., then what should we use?

On Twitter a few weeks ago, I saw @systemdesign42 post his top 4 git commands. In there was git add -i. Stage files interactively.

git add -i

I have been using git for 7 years and have never seen the interactive form of git add. I tried it immediately and found it incredibly intuitive. I can add edited files, untracked files, and undo the mistakes.

I have been deliberately using the interactive mode in an effort to break old habits. Now that I use it daily, I find the selective staging forces me to look at each edited file before staging a commit, in fact I don't think I've had to use my git undo alias since.

Maybe I'm late to the game on this, but if you find yourself in the same position, try putting git add -i into your daily workflow.

ref:
https://git-scm.com/book/en/v2/Git-Tools-Interactive-Staging

Exercises in Concurrency - All Goroutines are Asleep!

Dave Smith — Fri, 02 Feb 2024 03:07:35 +0000

How It Started

I saw the One Billion Row Challenge at the beginning of January and was immediately intrigued. This is not about the challenge (more on that later), but instead my side quest in troubleshooting concurrency. After I found a fasting way to read a large file I decided that concurrency will make everything better, ya know, make all the cores work 😉. I fill a buffer, send it to a pool of workers to do slow string functions, then combine results. Easy, right? I write the code yada yada yada, fatal error: all Goroutines are asleep - deadlock!

The error:

fatal error: all goroutines are asleep - deadlock!

goroutine 1 [semacquire]:
sync.runtime_Semacquire(0xc00001e0c0?)

The code. Can you see my mistake before reading on?

const workers int = 32
const chunkSize int = 8 * 1024 * 1024

func chunkedReadWithWorkerPool(filename string) {
    chunks := make(chan []byte, 100)
    res := make(chan int)
    wg := &sync.WaitGroup{}

    for i := 0; i < workers; i++ {
        wg.Add(1)
        go worker(chunks, res, wg)
    }

    go readChunks(filename, chunks)

    length := 0
    length += <-res
    wg.Wait()
    close(res)
}

func worker(chunks <-chan []byte, res chan<- int, wg *sync.WaitGroup) {
    wg.Add(1)
    defer wg.Done()
    length := 0
    reads := 0
    for in := range chunks {
        length += len(in)
        reads++
    }
    log.Printf("read %d bytes, in %d reads\n", length, reads)
    res <- length
}

func readChunks(filename string, chunks chan []byte) {
    file, err := os.Open(filename)
    if err != nil {
        panic(err)
    }
    defer file.Close()

    newLine := byte('\n')
    r := bufio.NewReaderSize(file, chunkSize+64)
    for {
        buf := make([]byte, chunkSize)
        n, err := r.Read(buf)
        if n == 0 {
            if err == io.EOF {
                break
            }
            if err != nil {
                fmt.Printf(err.Error())
            }
        }
        b, err := r.ReadBytes(newLine)
        if err == nil || err == io.EOF {
            buf = append(buf, b...)
        } else {
            fmt.Printf(err.Error())
        }
        chunks <- buf
    }
    log.Println("finished reading file. Closing channel")
    close(chunks)
}

Okaaaaaay

The google results for the error message all tell me that a channel is not getting closed. I attach the debugger and the debugger never goes past the wg.Wait() function. Thinking it is a problem with the res channel, I removed the results channel from the solution. Same error, but only one goroutine is throwing an error now.

The internet recommended that the worker should be in a closure to make sure the it tells the WaitGroup it is done.

Same error.

go func() {
    defer wg.Done()
    worker(chunks, wg)
}

I decide o take a break from this and write about it in hopes of a mental breakthrough.

Duh

I feel dumb. I set this aside for a day and looked at it with fresh eyes. I added to the wait group inside the for loop and also in the worker function 😬. The wait group is incremented by 2 for each worker, but done is called once per worker. The wait group will wait forever. What is amazing is Go can detect this at runtime and panic. I must admit, I don't know how that works.

    for i := 0; i < workers; i++ {
        wg.Add(1)
        go worker(chunks, res, wg)
    }

Some Reflection

Go makes concurrency easier than other languages, but concurrency will always be more challenging that synchronous code.

Why I am writing this when this is clearly not my best effort? I could just never mention this and the results is the same. Working through a challenging problem over an extended timeframe, finding a resolution (usually an aha moment) and reflecting on it could possibly be the best learning method. If you are experienced in Go, you likely would not have even made the mistake, I hope I don't toil on this same error in the future.

Hello, World

Dave Smith — Wed, 17 Jan 2024 19:01:38 +0000

Hey there. This is my first post on here so I have to do the obligatory “Hello, World”. I promise you it is not my last post.

My name is Dave Smith. I am a software developer. I am also a runner. You’ll see content for both activities here, heavy on the developer topics, with other topics that interest me. I did my first programming in 1996 on a TI-86 calculator, modifying games and programming trig equations to help me do homework faster. You could say I program more interesting problems now, but not everyday. Why include running and fitness here? As tech workers it’s easy to live a sedentary life. Running is my attempt to combat that. I have been running consistently since 2011. I particularly enjoy running long ultramarathons. People ask if it is boring. Yes, it can be boring, and boring is what I need some days.

I can’t believe I’m writing something that isn’t for work. My high school English teachers would be more surprised. I did the bare minimum when it came to writing in school. I hated it. That was part of the reason for going into software development. I only have to write code, right? Why am I doing this? I have a lot of thoughts banging around the old wetware. I have tried to get them into a coherent medium for quite some time. My personal goal for 2023 was to write blog posts. We’re in 2024, how did that work out for me? I was approaching my goal the wrong way. My goal now is to just write down thoughts as they come to me with a date. This post is the result of writing down a few thoughts and turning them into sentences.

Help keep me accountable with this. If you see something you like or don’t like, leave a comment, ask a question. I‘m always trying to get better at anything I spend time doing, including writing. There will be more, hopefully soon and it will be better over time.