DEV Community: Jason Steinhauser

Field Guide to Merging Codebases

Jason Steinhauser — Sat, 15 Jan 2022 20:31:09 +0000

Congratulations, your company has just acquired another smaller company to expand your product line! Pop the champagne, dress up the puppies, eat so much cake that you wake up full of regret! Now that the dust has settled and you've met your new teammates, you all come to the sudden realization that...

Oh. No.

Your out-of-date documentation? It'll be on full display to a new group of developers. That important business logic you wrote 2 years ago with the TODO: Write some tests above it? Better scramble to get some in there! The spreadsheet that product shares around for features, well... you get the idea. And that doesn't even start to cover what the other folks have stewing in their code cauldron.

Never fear! Many have been there before you. Many are currently in the middle of their own jungle of curly braces. When I joined Terminus about a year ago, they had already acquired 3 different companies and have since purchased another 2. Your mileage may vary, but here are some ways we've worked our way through the proverbial jungle.

Let's Talk

After a merger the product team, investors, and executives will want to see progress made towards merging the companies' product offerings. This is a good thing for the business! However, you're still going to have figure out how to talk - in just about every sense of the word.

Final form of fully merged products

Documentation

Documentation is a four-letter word, I know. But when you start interfacing with OtherCorp's whizzbang analytics dashboard, you're going to wish they'd documented it.

If you haven't documented your services yet... you're not alone! But to make it easier for you to work on your new buddies' services (or vice versa), you'll need to have things documented. Whether it's via a wiki (we have one conveniently named man) or through testing, you'll want to have an understood method of documentation.

Lingua Franca

When you're smashing together multiple products' codebases, you're likely going to have several different languages floating around. At Terminus, we've had/still have services written in Ruby, Groovy, Elixir, Go, Node (both JS and TS), Angular, and React. They all have their purpose and their reasons for being used, but moving forward you've gotta be able to have folks work across different repositories. People have their pet languages for sure. But people who have only ever written JS/TS are not going to want to "pick up Haskell" for that one service that Kelly wrote before leaving for AnotherCorp. (This is just an example, not trying to pick on JS/TS devs!)

If only they spoke the same way..

As much as it pains me to say it, coming up with a common set of languages to use is going to pay dividends in the end. I've worked on projects in different areas of our platform, and because they're in languages that are common across our product suite (and sometimes documented and tested well!) I spent minimal time ramping up to contribute to those services. It doesn't feel like it while you're adjusting, but it really does make things better in the long run.

Infrastructure

So we've identified the end goal: merging our codebases into a more cohesive platform! But where do we start? Just like with constructing a house, we need to have a good foundation to build everything on. We need... infrastructure!

Okay, so hopefully not like this.

You've got an AWS account... for most of your stuff. You've got that one Heroku experiment that too that we try not to mention often. And now your new coworkers are bringing all of their cloud accounts as well! I swear, if they even mention on prem hardware...

The end goal is more than likely going to be a single cloud provider account. So how do you get there? It's not going to be an easy or short-term fix. Environment variables to different services are scattered around everywhere. Identity and Access Management (IAM) roles and policies exist in more places than you realize. Let's make sense of all this.

Security

You don't want to inadvertently put your users' data at risk especially now that you're growing as a company! If I haven't hammered it home enough yet, security will have to be something people agree on. How should we encrypt data at rest? How do we protect services that are meant for internal use from outside snooping? Whether that's configured with IAM roles/policies, cordoned off in different VPCs/VPNs, use of some sort of protocol or token-based security (JWTs for example), work together to figure out the best (possibly not the easiest) way to keep data secure with minimal impact to performance.

Infrastructure as Code

In the beginning, you and/or your founders were probably tinkering around with the cloud, learning new concepts, and breaking all the things. This is perfectly normal in the startup world, but eventually it gets to be unwieldy.

My first cloud development toolkit

Now that you're joining forces with others, you'll want to keep track of things in a more programmatic way. Using an Infrastructure as Code tool such as Terraform is a great way to express platform configuration in an understandable fashion. You can import existing infrastructure into Terraform for configuration management, and when you want to modify your architecture you can see what will change before you pull the metaphorical trigger. And with IaC, you can version control all of your infrastructure changes in Git just like application code!

Tooling

Everyone monitors differently. Papertrail, New Relic, Datadog, Kibana, a shell into k8s pods... I've seen LOTS of ways that people monitor things, but y'all are going to have to figure out a unified way to keep an eye on services. If there's one thing I've learned, it's that people do not want to check more tools than they have to. Settling on one monitoring platform (if possible) will take effort, but in the end it'll make everyone's life easier.

CI/CD pipelines are another thing that people can be strongly opinionated about. I'll just echo here what I'd said about monitoring - there's a lot of ways to do the things. Bring everyone together to work out a common solution that everyone hates the least. Keeping stuff the same across the now singular organization will help folks move around as needed to different projects more easily.

Conclusion

So what's the big takeaway here? You're going to need common practices, common definitions, and common languages. It will take time to do it, but do as much of it up front as you can. Product will still want to ship new features and that's fine; you can do that while integrating everything together.

Our CEO has said this multiple times in regards to business strategy, but it applies to engineering as well: "Sometimes you have to slow down to speed up." When you take the time to blend together up front (or at least start executing a plan to unify processes 'n' such), it's going to help you to execute more quickly in the future.

I hope you enjoyed this! If you have experience with uniting codebases post-M&A, I'm happy to hear your comments below!

Artist's rendition of your new, unified platform

"Oh I Can Do That!" - How Everyone Is An "Expert"

Jason Steinhauser — Wed, 17 Feb 2021 14:51:50 +0000

One of the upper level executives at my company (Terminus Software, and shameless plug: we're hiring software engineers!) recently posted this on Twitter:

// Detect dark theme var iframe = document.getElementById('tweet-1361866448562438144-102'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1361866448562438144&theme=dark" }

And I mean, he's not wrong in that everyone seems to think they could be an "actual marketer". However, I've had many people tell me that they can do something better than I've done things all throughout my career. Sometimes they had good advice and prefaced it this way ("have you considered x?" when I actually hadn't), but most akin to "I saw this Python tutorial on how to do a weather service and you need to do that... but for your project" 😒.

And I get it why Sangram has his viewpoint - he's been in marketing for nearly 2 decades, and he's probably heard "marketing is easy" constantly. And people seem to think marketing is magic, because marketers' worth is hard to quantify without solid data. When I ran a local escape room with my wife, I tried my hand at marketing. It turns out... I had no idea what I was doing. WTH are impressions? I ran TV and radio spots. How effective were they? I have no idea because I didn't know how to get data to see which marketing channels were working for me. And as you can undoubtedly tell, I decided to continue a software career.

Hell, I've even done it too. While I was running that escape room, I was also writing the software to interface with the electronically-controlled puzzles in our room, as well as the monitors in the rooms, and the game masters' consoles. I definitely thought, "I've written software; I can make good UIs!" As it turns out, I had to learn a lot before I could "make good UIs". I'm still not great at it, but I'm passable!

When I dabbled in standup comedy, I took classes on how to craft jokes, and then expand those into bigger bits, and then into solid timed sets. That never stopped people from saying "Hey, I've got a joke idea for you!" 🧐 Someone outside of your field will inevitably tell you they can do your job.

So... what stories have you got? I want to hear them below!

Modern C++ Isn't Scary

Jason Steinhauser — Fri, 16 Aug 2019 16:58:06 +0000

C++ has a reputation for being an obtuse, antiquated language that is only for neckbeards and angry professors. Sure, it's fast, but why do you want to learn all of that nasty overhead just for a little more speed? And I love my package manager and tooling! Does C++ have anything modern?

This is exactly how I felt after every time I looked at C++ or worked on a C++ project since I graduated college. However, I've had to write a considerable amount of C++ recently and I have to say...

C++17 ain't your dad's C++.

Less horrendous syntax

C++ is notorious for its horrendous syntax. However, starting with C++11 the syntax has gradually been made much more appealing. A lot of it involves the use of the keyword auto. You can now use range-based for loops like in Python and C# (technically foreach):

double arr[3] = { 1.0, 2.0, 3.0 };

for (auto& x : arr) {
    std::cout << x << std::endl;
}

The & after auto signifies that we don't want to make a copy of the item in the array (or std::vector or any other iterable collection).
Auto can be used to destructure std::tuples and std::pairs, as well as structs and arrays! If we use our array above, we can destructure the whole thing:

auto [a, b, c] = arr;
// auto [a, b] won't compile because our array is length 3

You can use it in for loops on std::maps as well:

for(const auto& [k,v] : dictionary) {
...
}

Destructuring tuples works the same way:

std::tuple<double, std::string> t{ 4.0, "GPA" };
auto [value, label] = t;

That tuple looks to be defined in standard, verbose C++. Well, they've got a fix for that too...

Template Parameter Inference

If the type can be inferred in a template type, then there is no need to specify the types! I can rewrite the std::tuple definition above as:

std::tuple t{ 4.0, R"(GPA)" };
// or better yet
auto t = std::tuple{ 4.0, R"(GPA") };

The R"(...)" creates a string literal instead of a character array. This is considerably less bulky than previous iterations of the C++ language standard, and it makes type signatures a lot less irritating. You'll still have to fully qualify the type on function return types, but you can auto the pain away everywhere else.

The language standard committee has also added std::tie to integrate with pre-existing code as well, where you want to tie values as output of a function to variables that already exist. The language has really made it MUCH nicer than it used to be regarding functions that have multiple output values, and I'm quite satisfied with it.

Literals

C++ has long had some literals for integer and floating point types (i.e., defining zero as an unsigned long as 0UL or as a float as 0.0f). In C++11/14, have expanded the scope of literals to include booleans, strings (both UTF-8 and UTF-16), units of time (42ns is the same as std::chrono::nanoseconds(42)), and a lot more! There is also the option for user-defined literals, and the documentation is pretty solid. This has been one of the more exciting features for me personally!

More platform independent libraries

Several things were classically OS-specific in C++; there were no overarching abstractions. Thankfully, in C++11 and beyond that has been remedied.

For example, if you wanted to sleep the current thread for 5 seconds, in grandpa's C++ you'd write:

#include <unistd.h>

...
void sleep_func(unsigned int milliseconds) {
    usleep(milliseconds * 1000);
}
...

unsigned int sleep_seconds = 5;
sleep_func(sleep_seconds * 1000);

and that would work for *nix environments only. If you wanted to do sleep for some unit other than microseconds (the Unix implementation's unit of measure), you would have to do the conversion yourself. While not difficult, it's still prone to human error. If you wanted to do it on a specific thread... well, that's a whole different story. In C++11, you can use the std::chrono library, with a variety of clocks, as well as the std::thread library for thread-specific tasks.

#include <chrono>
#include <thread>

...
void sleep_func(std::chrono::duration duration) {
    std::this_thread::sleep_for(duration);
}
...

auto sleep_seconds = 5;
sleep_func(std::chrono::seconds(sleep_seconds));

There are several pre-defined durations in the std::chrono namespace also, so that it is incredibly clear to see how the units of your time span, and all the conversions are handled by the compiler. Less work for us!

They've also finally implemented a filesystem abstraction in C++17! It was experimental in C++14 but officially became a part of the language standard with the last release.

Creating objects in template classes

In the "good ol' days" of C++, using template collections was super annoying. If you wanted to push something to the back of a queue, for instance, you would have to create the object and then pass a copy of that object into the queue.

std::vector<thing> things;

for (int i = 0; i < 50; i++) {
   thing t("foo", i);
   things.emplace_back(t);
}

A lot of template classes now have functions that would copy previously that have an Args&&-style implementation now so that a new object of that type can be created in place in the template class! That looks something like:

std::vector<thing> things;

for (int i = 0; i < 50; i++) {
    things.emplace_back("foo", i);
}

This saves some copying overhead and speeds things up as well, and discourages the use of pointers in collections (when appropriate).

Better Pointers

Let's face it: dealing with raw pointers SUCKS. Allocating memory. Freeing memory. Creating a pointer in one scope and assuming ownership transfers to a new scope later. All of these cases require a lot of ceremony, can cause memory leaks, and require a lot of mental overhead to make sure you don't cause memory leaks. Thankfully, C++11 brought different pointer types developed in the Boost library into the language specification.

Unique Pointers

std::unique_ptr<T> can only ever be referenced by one object. If ownership needs to be transferred to a different scope (but still only maintain one copy), std::move can be used to transfer ownership. This can be useful in factories, or any other time that you might want to create something and pass its ownership to another object. For example, you may want to create a stream of bytes coming off a Socket and pass the ownership of that data to a requesting object.

Shared Pointers

std::shared_ptr<T> are officially one of my favorite things in C++. If some point needs to be referenced by multiple objects (like, say, a Singleton for a Websocket), in old school C++ you would've created a raw pointer to that object and destroy it on cleanup or after its last use... hopefully. Raw pointers are one of the single biggest contributors to memory leaks in C++. They probably rank up there with NULL pointers as a billion dollar mistake.

Thankfully, shared pointers are now a thing and widely accepted in the C++ community. When copied (and they should always be copied, not passed by reference or pointer), they increment an internal reference count. When these copies are destroyed, the reference count is decremented. When the reference count reaches zero, then the object is destroyed and the memory is freed up. No more manual memory management hassle! Yay! You can still shoot your foot off with shared pointers if you don't strictly copy them, but there's a better safety mechanism available now over shared, raw pointers, IMHO.

Conclusion

Though it still feels like it has more syntax than necessary, C++ is not as bad as I remembered it being. I've enjoyed the past 4 months developing in it on a daily basis much more than I have in past jobs. If you check it out again, hopefully it won't be as scary to you either!

Happy coding!

Congressional District-based Encoding

Jason Steinhauser — Mon, 05 Aug 2019 01:23:08 +0000

Note: this project has no political basis. I did this just because I thought it would be funny.

My Twitter-verse was all aflutter this late last week due to the release of a typeface based on gerrymandering. Neutral of its possible perceived political implications, I thought this was hysterical, and that it warranted having its own encoding.

Introducing Gerry Cipher!

Gerry Cipher is basically translating Base64 encoding into congressional districts. This was interesting in a couple of regards:

Some letters were formed by combining two districts
M and W were the same district, just mirrored
There were no districts provided for numerals

To address converting the district-based encodings to upper-/lowercase letters, it was fairly trivial: if a letter was comprised of one district, the uppercase encoding had uppercase letters, and likewise for lowercase. If the letter was a composite, then the left/upper district was the uppercase encoding, and the right/bottom district was the lowercase encoding. In the case of M/W being mirrored (NY's 8th district), the encodings are mirrored (NY08/80YN and ny08/80yn for M/W and m/w).

Addressing numerals was something not covered by the Ugly Gerry typeface. To this effect, I used the ten largest congressional districts by area. Since I was using Base64 encoding as a basis, there was the matter of handling + and /. I determined that the two longest-running non-voting delegations would be used - Puerto Rico (PR01) and Washington D.C. (DC01).

What's Next?

It's just a silly little project, and I may or may not translate it to other languages just to propagate the silliness. I will code up... er... decoding... soon as well. I welcome you to check it out!

Happy coding!

Wormhole - Data Collection Cheat Sheet and Library in 4 Languages

Jason Steinhauser — Mon, 25 Feb 2019 21:09:02 +0000

During the Advent of Code this past year, I was trying to enhance my knowledge of Elixir, as well as just functional programming in general. There were times where I found a function that most other languages I used didn't have an analogous function (a glaringly obvious one being Enum.reduce_while), and other times where I was writing functions I'd used often in other languages (Clojure's frequencies would've been mighty handy!). I finally decided to bite the bullet and try to create a list of collection manipulation functions I used often in other languages, implement them in others I've learned/currently trying to learn, and discover new functions that I wouldn't want to live without!

What Is Wormhole?

It's this, really:

jdsteinhauser / wormhole

Some of my most used functions implemented in different languages

Wormhole

You ever think, "Hey I wish this langugage had the capability of some other language I like?" Enter Wormhole.

Motivation

During the Advent of Code 2018, I found myself writing the same functions in Elixir that I knew I had used in Clojure, F#, or some other language. In order to prevent myself from doing this in the future, I decided that building a library to do house all of these helpful functions across languages that I either knew or wanted to learn.

Desired functions

These are the functions that I've used and that I'd like to have in multiple languages. Some of them are already implemented in the language already, so I won't reimplement them. Each implementation will list the functions as implemented in the language, as well as links to their documentation.

map
filter
reduce
reduce_while
chunk
chunk_by
juxt
min_by
max_by
frequencies
group_by
scan
inc
dec
zip…

View on GitHub

I've got an addiction. I love learning new languages. With learning new languages, you end up finding functions, classes, and concepts that you wish that you had in other languages. Sometimes, those functions are named different things and it gets confusing when you switch between languages. I end up doing a lot of data collection manipulation, and so I decided to start with what I knew best and branch out from there!

What Functions Am I Looking For?

For a non-exhaustive list, I wanted to have at least the following:

Collection basics: map, filter, reduce, and scan
Chunking data: chunk, chunk_by
Common stats: min_by, max_by, group_by, frequencies
Other hella useful things: reduce_while, juxt, identity

What Languages Am I Targeting?

For now, I have filled in my perceived gaps in functions in C#, Clojure, and Elixir. I have an F# solution that I'll be comfortable with early this week, and I've started looking at a comprehensive list of Ruby functions as well. After that... well, I'm not entirely sure! I think I'm going to go through Rust, JavaScript, Java, and possibly Kotlin and Python 3 to see what other handy things I can implement across all those languages.

Will These Be Deployed to Package Managers?

Yes... but not right now. I need to get the documentation to a suitable state. I've pulled down several packages before but I've never pushed mine up to any! I'm sure that will end up being a blog post in and of itself.

Current Cheat Sheet

Here's a summary of the languages I've targeted so far, with documentation links to each function that either already exists, or that I've implemented in Wormhole.

Function	C#	F#	Clojure	Elixir
`map`	`Enumerable.Select`	`Seq.map`	`clojure.core/map`	`Enum.map/2`, `Stream.map/2`
`filter`	`Enumerable.Where`	`Seq.filter`	`clojure.core/filter`	`Enum.filter/2`, `Stream.filter/2`
`reduce`	`Enumerable.Aggregate`	`Seq.reduce`	`clojure.core/reduce`	`Enum.reduce`
`reduce_while`	`ReduceWhile`	`reduceWhile`	`reduce-while`	`Enum.reduce_while/3`
`scan`	`Scan`	`Seq.scan`	`clojure.core/reductions`	`Enum.scan`, `Stream.scan`
`chunk`	`Chunk`	`chunk`*	`clojure.core/partition`	`Enum.chunk_every/4`, `Stream.chunk_every/4`
`chunk_by`	`ChunkBy`	`chunkBy`	`clojure.core/partition-by`	`Enum.chunk_by/2`, `Stream.chunk_by/2`
`juxt`	`Juxt`	`juxt`, `juxt2`, `juxt3`	`clojure.core/juxt`	`Wormhole.juxt/1`
`min_by`	`MinBy`	`Seq.minBy`	`min-by`	`Enum.min_by/3`
`max_by`	`MaxBy`	`Seq.maxBy`	`max-by`	`Enum.max_by/3`
`frequencies`	`Frequencies`	`freqs`	`clojure.core/frequencies`	`Wormhole.freqs/1`
`group_by`	`Enumerable.GroupBy`	`Seq.groupBy`	`clojure.core/group-by`	`Enum.group_by/3`
`identity`	`Identity`	`Operators.id`	`clojure.core/identity`	`Wormhole.identity/1`

F# contains a Seq.windowed function, but it only moves the chunk one element at a time.

Why Is This Stuff Useful?

Well, some of the functions are either self-explanatory or already written about in several other articles. I'll cover some of the lesser known ones and why I personally found them useful.

Chunking

I've written about chunk and chunk_by before, but in case you missed it, check out my previous article!

Alright, Break It Up! Using Partition/ Chunk

Jason Steinhauser ・ Oct 18 '18

#functional #programming #coding #algorithms

Reduce While

I'll admit that this is possibly a not-so-often used case. Sometimes you don't want to reduce an entire sequence - just up to a certain point. Unfortunately, reduce is typically all or nothing. That doesn't really work when you have a potentially infinite series of data. However, Elixir's reduce_while helped me keep my solution for AoC 2018 Day 1 Part 2 compact. I'm hoping to find more real-world use cases for it... but it's still one of my favorite data processing functions I've found.

Juxt

While I admit that, at first glance, juxt is nothing special. Take an array of functions that operate on the same parameters, and then return a single function that takes that parameter and returns an array of each function run on those parameters? Why use that?

I've ported this function from Clojure into other work projects before. For instance, I had a very large collection of data (1MM+ entries!) and I couldn't afford to iterate over them multiple times. I used juxt to compose my analysis functions together so that I only had to iterate over the collection one time.

Similarly, since a keyword in Clojure can be treated as a function for retrieving a value out of a map with that key ((:foo {:foo 5 :bar 3}) returns 5), you can compose several keywords for accessing data out of a collection of maps and returning the results in kind of like a table format. I wrote about that as part of a previous post on dense Clojure code:

A verbose explanation of compact code

Jason Steinhauser ・ May 21 '18

#showdev #clojure #productivity

Frequencies

Because sometimes, you just need a histogram. frequencies provides that in one single function!

Conclusion

Hopefully someone out there will find this useful, either as a cheat sheet or as a library. In the near-term, I will be investigating Ruby and Rust (in that order) to see what other handy functions I could foresee using across multiple languages. I'll also put Wormhole up as a package in your favorite package managers soon, and probably write about the things I do/don't like about each.

Happy coding, and I'd love to hear about other general purpose data manipulation functions you've found useful!

I Don't Know How To Take Compliments

Jason Steinhauser — Tue, 23 Oct 2018 21:14:02 +0000

Today was a good day. My wife complimented me in my new shirt and pants (totally rocking some floral print), I helped a coworker with design on a different project, and I wrapped up a proposal and thanked a coworker for his assistance. I can handle my wife's compliments... but I wasn't sure how to deal with the other two situations.

What Happened?

My coworker asked me about handling role-based view components. We white-boarded it for a few minutes, and I finally mentioned that we could just use the role names as map keys and the values as components that role had access to as a list. We finally settled on (as his idea) the map values being another map which contained boolean properties that we could use in an ngIf so that there was constant lookup time and minimal logic in his view. He said to me "I've had to do role-based management on several different projects, and this is by far the cleanest way I've ever seen it done. Thanks for your help!"

So what's wrong with that?

I don't really feel like the credit is all mine. His idea was half of the implementation, and the last part to make the lookup constant time instead of linear time. It's not a big deal, considering we have few components and roles right now, but still! Half of the idea was his. It doesn't feel like anything incredible, honestly. I was just doing my job. I mean, I reminded him that he was part of the solution, but I still felt a little... not really embarrassed? More like sheepish, I guess?

Anything Else?

Another coworker had been helping me work on my proposal and I pulled him in at the last minute to help, and I apologized to him up front for not engaging him earlier. After we wrapped everything up and I went to his desk to thank him for his help and apologized again for roping him in so late. He said "Well, I definitely appreciate you thanking me. A lot of people expect me to just drop what I'm doing even with late notice, but I appreciate your courtesy."

AND?!?!

I blushed when he thanked me for being courteous and told him, "Well, I didn't know what else to do but apologize and thank you." And I just really didn't think that what I did was extraordinary by any means. I thought it was... well... just what should happen.

So what are you getting at?

The first situation, where I helped my friend/coworker with a design issue... I can handle relatively well. I did help him with something, and he finished out the idea. I'm totally fine for taking the partial credit for a good idea, but I really just felt like I was doing my job... nothing special.

The second situation, however, I can't really get past that one. I was raised to be polite to everyone who is providing you service - a waiter, a mechanic, a coworker - and so it made me uneasy to know that I was the exception rather than the rule when it came to showing appreciation.

Is it wrong to feel like this? To feel like what you think is common courtesy or just doing your job is praised? Have any of y'all experienced this feeling and worked through it? I'd love to hear some feedback!

Alright, Break It Up! Using Partition/ Chunk

Jason Steinhauser — Thu, 18 Oct 2018 22:08:08 +0000

Programming Should Be About Transforming Data [...] I don't want to hide data I want to transform it.

Dave Thomas, Programming Elixir

Whether it's calculating students' final grades, turning mouse clicks into t-shirt orders, or reading and publishing temperature data from a RPi, programming is basically distilling data from one form into another. Oftentimes, it doesn't feel like pure data transformation because we try to handle how the data is stored as well as its transformation in one step. This can make code awfully confusing, especially to anyone that has to maintain the code.

Any fool can write code that a computer can understand. Good programmers write code that humans can understand.

Martin Fowler

I've got a couple of examples from past projects that I'd like to share, going from how I'd written algorithms when I was more concerned about the place of my datapoints, and then after I discovered partitioning/chunking functions.

What is Partitioning/Chunking?

Partitioning and chunking, in the context of this article, is subdividing existing collections, streams, and enumerables/iterables, into pieces more useful to your algorithm. This chunking can be done either by:

Size (from ClojureDocs)

;; partition a list of 22 items into 5 (20/4) lists of 4 items 
;; the last two items do not make a complete partition and are dropped.
(partition 4 (range 22))
;;=> ((0 1 2 3) (4 5 6 7) (8 9 10 11) (12 13 14 15) (16 17 18 19))

Max Size (from LoDash)

_.chunk(['a', 'b', 'c', 'd'], 2);
// => [['a', 'b'], ['c', 'd']]

_.chunk(['a', 'b', 'c', 'd'], 3);
// => [['a', 'b', 'c'], ['d']]

A mapping function to group contiguous items in the stream that map to the same value (from Elixir Docs)

Enum.chunk_by([1, 2, 2, 3, 4, 4, 6, 7, 7], &(rem(&1, 2) == 1))
# => [[1], [2, 2], [3], [4, 4, 6], [7, 7]]

Basically, yes... divide and conquer.

Parsing Arrays from Binary Streams

If you've ever had to get raw data off of a wire or read in a proprietary/lesser-used binary format, then you've probably had to write your own parser for it. And more often than not, you've had to read an array whose length is defined by an earlier declared variable. Endianness aside, that in and of itself is not the most fun. The format is usually akin to:

Value	Type	Size
Sync Word (0xC011FEFE)	int32	4
Timestamp (µsec)	int64	8
nElements	int32	4
Data elements	int32[]	4*nElements

Now, normally how I would have normally built something to read in the packet (without validation) like:

public class DataFrame {
    public readonly DateTime Timestamp;
    public readonly int[] Data;

    public DataFrame(DateTime timestamp, int[] data) {
        Timestamp = timestamp;
        Data = data;
    }

    public static FromBytes(byte[] rawData) {
        var timeUsec = BitConverter.ToInt64(rawData, 4);    // Skip the sync word
        var timestamp = DateTime.FromFileTimeUtc(timeUsec);
        var count = BitConverter.ToInt32(rawData, 12);
        var data = new int[count];

        // Skip the first 16 bytes before we start reading the data
        for (int i = 0; i < count; i++) {
            data[i] = BitConverter.ToInt32(rawData, 16 + i * 4);
        }

        return new DataFrame(timestamp, data);
    }
}

Granted, I have to care a lot about where the header information is, but there has to be a way to make my intention clearer on reading the array of data into data. The same thing, written in F#, feels a little cleaner to me:

type DateFrame = { Timestamp : DateTime; Data : int array }

let fromBytes data = 
  let time = BitConverter.ToInt64(data, 4)
  let numItems = BitConverter.ToInt32(data, 12)
  {
    Timestamp = DateTime.FromFileTimeUtc time
    Data = data
           |> Array.skip 16
           |> Array.chunkBySize 4
           |> Array.take numItems
           |> Array.map (fun x -> BitConverter.ToInt32(x, 0))
  }

Chunking data into 4 byte increments allows me to very cleanly pass info to the BitConverter to make the conversion to 32-bit integers. I don't have to keep track of offsets, and have effectively removed the entire category of offset or "off by x" errors from my code!

Only parts of the data matter

Let's face it: sometimes you just don't care about a lot of your data.

Image credit to this blog post

Let's say you're a delivery company and you want to make sure that your drivers are obeying speed limits. You really don't care about times that they're actually doing that, but when they go over the speed limit they might want to figure out why. Maybe they were going downhill and started to brake late. Perhaps they were texting and not paying attention. Who knows? But you're supposed to take the speed limit from your map API and velocity from the GPS and figure out when, and for how long, a driver was going over the speed limit. Let's see what it might look like to find drivers that were speeding for more than one minute:

public class DriverData {
    public readonly DateTime Timestamp;
    public readonly double Speed;
    public readonly double SpeedLimit;

    public DriverData(DateTime time, double speed, double speedLimit) {
        Timestamp = time;
        Speed = speed;
        SpeedLimit = speedLimit;
    }
}

public class Driver {
    public readonly Name;
    public IEnumerable<DriverData> DrivingHistory;

    public IEnumerable<(DateTime, TimeSpan)> SpeedingMoreThanAMinute() {

        // If the collection has no values or only one value, result is empty
        if (DrivingHistory == null || DrivingHistory.Count() <= 1) {
            yield break;
        }

        var isSpeeding = false;
        var speedingStart = new DateTime(0);
        var lastTime = new DateTime(0);
        var i = 0;

        foreach (var point in DrivingHistory) {
            var isSpeedingNow = point.Speed > point.SpeedLimit;

            if (isSpeedingNow) {
                if (!isSpeeding) {
                    speedingStart = point.Timestamp;
                }
                isSpeeding = true;
                lastTime = point.Timestamp;
            }
            else {
                if (isSpeeding) {     // We were speeding, and now we're not
                    var duration = (lastTime - speedingStart).TotalSeconds();
                    if (duration >= 60) {
                        yield return (speedingStart, duration);
                    }
                }
                isSpeeding = false;
            }
        }
        yield break;
    }
}

Yuck. That's a lot of ugly logic! There are six possible conditions there, and we have to keep all of those internal state variables and their functions in our head as we write this out. We need all of this overhead to find out when our drivers are speeding? Surely you can't be serious!

This where chunking by a function comes in handy. Trying this in Elixir (EDIT: slight fix thanks to Aleksei Matiushkin!) :

defmodule DriverData do
    defstruct [:timestamp, :speed, :speed_limit]

    def is_speeding_one_minute_plus(data)
        data
        |> Enum.drop_while(& &1[speed] <= &1[speed_limit])
        |> Enum.chunk_by(& &1[speed] > &1[speed_limit])
        |> Enum.take_every(2)
        |> Enum.map(fun x -> 
            %{start => hd(x)[timestamp],
              duration => DateTime.diff(List.last(x)[timestamp] - hd(x)[timestamp])
            end)
        |> Enum.filter(& &1[duration] >= 60)
    end
end

Since this is one of five languages I've used in this post (six if you count English!), I'll walk you through what's going on since it's less simple. The Elixir |> operator takes the previously evaluated expression and uses this as the first argument in the next function call. This functions the same way as Clojure's -> macro, but differently than F#'s |> operator (which uses the previous statement as the last argument in the next function). I use Enum.drop_while/2 to drop off all the points at the beginning from where the driver isn't speeding, and then I use Enum.chunk_by/2 then break the list into chunks based on whether or not the driver's speed exceeds the speed limit, and then use Enum.take_every/2 to take every 2nd element (as specified by the 2 in the arguments). That leaves me with only Lists of times where the driver was speeding. I then use Enum.map/2 to turn those lists into maps with keys of :start and :duration, and then filter those based on a duration >= 60 seconds. We did the whole thing solely by operating on the chunks of data we cared about, filtered out what we didn't need, and turned the output into something we could use, all without using any conditional branches or any internal state.

Conclusion

Admittedly, it took me quite a while to wrap my head around chunking/ partitioning data into useful bits. It's hard to make that transition from how I'm used to developing algorithms in over a decade of procedural and OO programming, to thinking about everything as just data transformations. If the concept seems difficult for you, just understand that lots of other people, including me, have been there too! If you find yourself building considerable amount of internal state just to parse some data out of an iterable/ enumerable collection, use that as a code smell to see if chunking is right for your problem.

Happy coding!

We Don't Need No Stinking map() or filter()

Jason Steinhauser — Sat, 22 Sep 2018 22:29:33 +0000

Pretty clickbait-y title, I know, but hear me out! I posed a shower thinking question to some of our junior developers. If you had to choose one of map, filter, or reduce to use for the rest of your life, which one would you pick? While I use map and filter quite regularly in programs - pretty much independent of language - and I prefer to use them... they're basically derivatives or optimizations of reduce.

How?!

So, let's look at some examples of Array.prototype.reduce() in JavaScript. It takes a callback function and an optional initial value. The callback function takes an accumulator and the currentValue in the array that we're reducing. Reduce is often used to sum the terms of a collection:

const sum = (accumulator, currentValue) => currentValue + accumulator

let val = [1,2,3,4,5].reduce(sum,0)

console.log(val)  // => prints 15 to the console

Let's walk through the individual steps of what happens. In the first step, we have an accumulated value of 0 (we pass that as the initial value), and the current value is 1. We repeat this for each element in the array, and we return the final accumulated value.

sum(0,1) = 0 + 1 = 1     // remaining values to accumulate: [2,3,4,5]
sum(1,2) = 1 + 2 = 3     // remaining values to accumulate: [3,4,5]
sum(3,3) = 3 + 3 = 6     // remaining values to accumulate: [4,5]
sum(6,4) = 6 + 4 = 10    // remaining value to accumulate: [5]
sum(10, 5) = 10 + 5 = 15 // remaining values to accumulate: []

// return accumulator of 15

This is a pretty straightforward use case. Reduce seems to be most often used for sums and products.

You sure we don't need a map?

Photo from Julian Peters Photography

Array.prototype.map() takes a function to apply to each element of the array, and returns the resulting array. The classic example of mapping is doubling integers:

const doubler = x => x * 2

let val = [1,2,3,4,5].map(doubler)

console.log(val)    // => prints [2,4,6,8,10] to the console

Pretty straightforward, right? So how in the world can we build map using reduce? Instead of our accumulator being a single value now, our accumulator will be an array. We'll apply the function to each currentValue and append that to the end of the accumulator array:

const pushAndReturn = (arr, item) => { arr.push(item); return arr }

const map = (f, arr) => arr.reduce( (acc, x) => pushAndReturn(acc, f(x)), [])

// Let's try it!
console.log( map(doubler, [1,2,3,4,5]) )    // => prints [2,4,6,8,10]

Excellent! Works exactly as we expected. But what were all the steps taken to get that answer?

[].pushAndReturn(doubler(1)) = [2]
// remaining values to accumulate: [2,3,4,5]
[2].pushAndReturn(doubler(2)) = [2,4]
// remaining values to accumulate: [3,4,5]
[2,4].pushAndReturn(doubler(3)) = [2,4,6]
// remaining values to accumulate: [4,5]
[2,4,6].pushAndReturn(doubler(4)) = [2,4,6,8]
// remaining value to accumulate: [5]
[2,4,6,8].pushAndReturn(doubler(5)) = [2,4,6,8,10]
// remaining values to accumulate: []

// return accumulator of [2,4,6,8,10]

It's important to keep in mind that the accumulator can be any type we want. It doesn't just have to be number for summing a list of numbers. It can be literally anything.

Let's filter our $#!+

Image from Outside Magazine

Array.prototype.filter() takes a predicate function and returns an array of values that return a truthy value when the predicate is applied. For example, if we want to get only odd numbers out of an array, we can use:

const isOdd = x => x % 2 === 1

let val = [1,2,3,4,5].filter(isOdd)

console.log(val)        // => prints [1,3,5]

Let's implement filter with just reduce! It's going to seem awfully similar to our map implementation above.

const filter = (pred, arr) => arr.reduce( (acc, x) => pred(x) ? pushAndReturn(acc, x) : acc, [])

// Let's try this one now!
console.log( filter(isOdd, [1,2,3,4,5]) )    // => prints [1,3,5]

Again, let's walk through the steps to see what happens:

isOdd(1) ? [].pushAndReturn(1) : [] = [1]
// remaining values to accumulate: [2,3,4,5]
isOdd(2) ? [1].pushAndReturn(2) : [1] = [1]
// remaining values to accumulate: [3,4,5]
isOdd(3) ? [1].pushAndReturn(3) : [1] = [1,3]
// remaining values to accumulate: [4,5]
isOdd(4) ? [1,3].pushAndReturn(4) : [1,3] = [1,3]
// remaining value to accumulate: [5]
isOdd(5) ? [1,3].pushAndReturn(5) : [1,3] = [1,3,5]
// remaining values to accumulate: []

// return accumulator of [1,3,5]

So... why?

While this was mostly a thought exercise (I wouldn't necessarily use reduce() to replace map() in production code), it's important to remember that reduce() can return any sort of value. Sure you can do sums, products, and concatenation with reduce... but you can also do mapping, filtering, taking elements while they match a predicate, build tree structures, create dictionaries/maps from lists/arrays... all by using reduce.

Thanks for reading!

When Math Fails You

Jason Steinhauser — Wed, 01 Aug 2018 17:46:44 +0000

It was after midnight, and I couldn't sleep. I opened up Chrome on my phone and I'd pulled up an editorial about COBOL earlier in the evening, and it gets me thinking about how our computers actually do math. The author linked an article all about issues with floating point math and some of the issues it's caused. You'd be surprised how many issues come from just not realizing that 0.1 can't be exactly represented in binary. So I decided to check out an example in the article with their test values, x=77617, y=33096:

Thanks to Quick LaTeX for the formatting!

Trying It Out

I wanted to see if I could replicate their findings:

Calculation Precision	Result
Single	1.172603...
Double	1.1726039400531...
Extended	1.172603940053178...
Actual Value	-0.827396059946

So I implemented the function in Clojure:

(defn trial-double [x y] 
    (+ 
        (* 333.75 (Math/pow y 6))
        (* x x (- 
                (* 11 x x y y)
                (Math/pow y 6)
                (* 121 (Math/pow y 4))
                2.0))
        (* 5.5 (Math/pow y 8))
        (/ x 2 y)))

Calling (trial-double 77617 33096) yields the result -1.1805916207174113E21. Oddly, this is not the same incorrect answer that they achieved in the article. Mine was 21 orders of magnitude different, and the wrong sign! I tested it out in Excel as well...

=333.75 * y^6 + x^2 * (11*x^2*y^2 - y^6 - 121 * y^4 - 2) + 5.5 * y^8 + x/2/y)

... and I got the same wrong number that I did with Clojure!

However, as the article mentions, this is not the correct answer. So I set out to calculate the right answer. I ended up trying it out with a combination of Clojure's Ratio type and Java's BigInteger type (since Clojure's BigInt had an overflow issue):

(defn trial-rational [^java.math.BigInteger x ^java.math.BigInteger y] 
    (+ 
        (* 1335/4 (.pow y 6))
        (* x x (- 
                (* 11 x x y y)
                (.pow y 6)
                (* 121 (.pow y 4))
                2))
        (* 11/2 (.pow y 8))
        (/ x 2 y)))

Evaluating this with the test values, (trial-rational (biginteger 77617) (biginteger 33096) yielded the fraction -54767/66192. Coercing that to a double evaluated to -0.8273960599468214. I'd gotten the right answer!

What Does It Mean?

While this example was clearly meant to show that floating point operations can break in dramatic fashion, let's not chalk it up as just an academic example. Instead, let's think about things that you've written in the past. How many times have you tried to increment something by 0.1 or 0.01, only to have errors accrue? If I add up 0.1 ten times (like you would expect to do in a simulation with 10 Hz data), depending on the precision I get either 1.0000000149011612 or 0.9999999999999999, not 1.0. Accretion of floating point errors just like this caused Patriot missiles to miss their targets in the 90s during the Gulf War, and soldiers died as a result. Regardless of your stance on the military, I think we can all agree that people ought not die because people and machines represent numbers differently.

How Did You Get Here from COBOL?

The editorial that got me thinking about all this was diving into why a lot of the financial sector - especially the IRS - was still using COBOL. It turns out that floating point calculations were easier to understand (though more costly to perform at the time) than fixed point calculations, and so they progressively won out. However, when more exact representation of numbers are required, it turns out that fixed point gets the job done much better. I have no clue if the formula at the center of this thought exercise would be --more accurate-- even in the ballpark of the correct answer, so I can't say that it is always going to be better than floating point calcs.

What Should We Do?

As it turns out, floating points are generally accurate enough for most applications, and that accuracy tends to degrade only as you add/multiply more and more numbers that don't have an exact floating point representations. However, if you absolutely have to have accuracy, there are a couple of options.

Store as Integers

If you're consistently dealing with a consistent number of decimal places, you could consider storing values as integers with a known scale factor. For example, You Need a Budget stores currency values as "milliunits". So (USD) $123.45 is stored as 123450 millidollars. Why not 100? Because some currencies (they list the Jordanian dinar as an example) go to 3 decimal places. Storing things as scaled integers and casting to floating point as needed will increase the accuracy of your calculations in this case.

Use Rational Numbers

If you find yourself dealing with fractional values often, it might be worth considering a library for rational numbers. Some languages have them built-in - Haskell and Clojure come immediately to mind, and I'm sure that others exist. Using rational numbers under the hood and only coercing to a floating point when necessary will reduce the amount of error that can creep into your calculations.

Symbolic Computing

What if you find yourself dealing with lots of numerical methods? Anyone who has dealt with ordinary differential equation solvers has had to worry about floating point errors in fixed step solvers (Runge-Kutta 4 from Numeric Recipes, anyone?). While you could always migrate to an adaptive step size solver (assuming you have a continuous function), we now have the computing power available to us to perform symbolic computations now. Think of it like Wolfram Alpha but built in to your program. Remember the power rule for derivatives? So do symbolic computation engines. Colin Smith has ported a lot of Gerald Sussman's ideas from SICP and Structure and Interpretation of Classical Mechanics (SICM) from Scheme into Clojure. If you don't need to go quite that far, Automatic Differentiation may work out well.

Conclusion

Floating point calculations are generally accurate enough. If you need better accuracy, thankfully there are several options available that you can explore. Just remember to take the time to determine what level of accuracy you need (or more importantly, what your customers need), and pick the type of calculations that suit those needs best. Good luck, and happy coding!

What is today's "goto"?

Jason Steinhauser — Wed, 20 Jun 2018 15:37:15 +0000

50 years ago, Edgar Dijkstra published a controversial (at the time) editorial entitled "Go To Statement Considered Harmful". In it, he stated what others had already hinted at - notably, Tony Hoare (creator of ALGOL and the billion dollar mistake), and John McCarthy (creator of LISP) - that goto statements in high-level languages impeded understanding of the code. Since then, we've seen movement away from goto... but we're still not fully clear of it.

One of my first projects as a software engineering intern was to take legacy FORTRAN and convert it into true OO C++... and make it a drop-in replacement to be called from other legacy FORTRAN as a full conversion of the codebase was performed. This was around 30 years after Djikstra's editorial and yet... in 1999, my partner and I found excessive used of gotos in functions that were hundreds of lines long. Having the code printed out, with lines drawn between labels and their respective gotos, trying to adapt the logic into if..else and return, continue, and break equivalents took the better part of the summer.

Unfortunately, that wasn't my last encounter with that dreaded four-letter word: goto. On one of my former projects, I tried to use my "free time" in long release cycles to try to remove one goto that we had in a relatively new C# project. It was buried deep inside of a numerical method. I spent the better part of a day trying to figure out how in the world I could rid of it, and... there wasn't a good, clean way. I called up the developer who'd written the code (he had moved to another company, but we're still good friends) and asked him "What in the world were you thinking?" After a long laugh at my misfortune, he said he'd ported that method from C++ at the recommendation of one of our data scientists and that was the one goto he was unable to remove cleanly. And so it stayed, presumably to this day.

It's actually pretty amazing to me that C#, a language developed around the turn of the millennium, still allowed developers to use goto! Java actually has goto as a reserved word, but has it marked as unused. JavaScript, with all its faults, doesn't have goto defined as a keyword. Of course, that has stopped someone from adding them in for some reason.

This kind of led me to an interesting question: what do we do in everyday development that we can do, but shouldn't? People have since posited that several things are "considered dangerous," from NULL references, to IPv4 wrapped addresses to even entire chipset families. I've found one that I agree with (to an extent) about the the overuse of Electron due to its severe bloat.

What practices and processes do you "consider harmful"? JavaScript everywhere? Measuring the wrong metrics for software quality? Crypto-everything? Let's discuss in the comments!

Intro to Property-Based Testing

Jason Steinhauser — Sat, 09 Jun 2018 01:17:00 +0000

Property-based tests make statements about the output of your code based on the input, and these statements are verified for many different possible inputs. - Jessica Kerr (@jessitron)

Property-based testing is generative testing. You do not supply specific example inputs with expected outputs as with unit tests. Instead, you define properties about the code and use a generative-testing engine (e.g., QuickCheck) to create randomized inputs to ensure the defined properties are correct.

What is the purpose of generative testing?

Generally speaking, property-based tests require only a few lines of code (like unit tests), but unlike unit tests they test a different set of inputs each time. Because of this, you end up covering more domain space with roughly the same amount of test code.

Property-based testing also promotes a more in-depth understanding of the function under test. Sure, we know that for addition 2 + 2 = 4, but how can you say that you truly have implemented an addition function correctly with that one example?

From a business standpoint, a better understanding of the functionality of your code will lead to more accurate determination of requirements being met in your code.

Does it replace unit tests?

While it is tempting to say that unit tests could be completely replaced by high quality property-based tests, unit tests still have their place in the software development cycle. Example-based testing is fantastic for early stages of TDD. These serve as anchor points to ensure that your development efforts are proceeding as desired. For example:

You can ensure that your sine function is generating proper values (sin(0) = 0, sin(π/2) = 1, etc.)
HTTP POSTs to your /login route without proper authentication headers/cookies results in a 401 response code

However, eventually these example-based tests end up being passive tests; the tests become part of a regression suite and provide no new information about the functionality. Property-based tests, however, are always active tests as they generate new data each time the test suite is run. This can help root out issues that developers overlook. For example, you may expect that the output of your sine function for any floating-point representation to be in the set [-1,-1], but the developer may not think to test for ±∞ or NaN. To this end, when a failing case is identified, this unexpected error can be used as a future unit test to validate fixes were made. This reason alone makes example-based testing an effective practice for enhancing property-based test suites.

"Don't write tests. Generate them!" - John Hughes, co-author of Haskell's original QuickCheck

What are some common properties?

If you correlate software functions with mathematical functions, then you can apply some standard mathematical properties to some expected functionality. However, some of those properties aren't as apparent in non-mathematical applications. Here are a few example mathematical properties translated to function properties (with updates from Eric Normand), ignoring performance differences:

Associative – a + (b + c) = (a + b) + c
- hashmap1.merge(hashmap2.merge(hashmap3)) = (hashmap.merge(hashmap2)).merge(hashmap3)
- list1.append(list2.append(list3)) = (list1.append(list2).append(list3)
- Math.max(Math.max(a, b), c) = Math.max(a, Math.max(b, c))
- (bool1 && bool2) && bool3 = bool1 && (bool2 && bool3)
Commutative – a + b = b + a
- users.Sort().Filter(x => !x.IsAdmin) = (users.Filter(x => !x.IsAdmin)).Sort() (relocated from Associative)
- image.flipX().flipY() = image.flipY().flipX()
- Math.max(a,b) = Math.max(b,a)
- bool1 && bool2 = bool2 && bool1
- average(a,b) = average(b, a)
Distributive – a(b + c) = ab + ac
- title.ToUpper() + author.ToUpper() = (title + author).ToUpper()
Idempotent – f(a) = f(f(a))
- Math.abs(x) = Math.abs(Math.abs(x))
- hashmap.merge({a:1}) = hashmap.merge({a:1}).merge({a:1})
- title.Trim() = title.Trim().Trim()
- list.Sort() = list.Sort().Sort()
Identity – f(a, i) where i is identity value of f = a
- a + 0 = a
- a * 1 = a
- userNames.append([]) = userNames
- hashmap.merge({}) = hashmap
- bool1 && true = bool1
Zero – f(a, z) where z is zero value of f = z
- a * 0 = 0
- intersect(valueSet, emptySet) = emptySet
- bool1 && false = false

There are additional, commonly-tested properties that are not necessarily rooted in math, but are equally as useful:

Bilbo Testing (aka, There And Back Again)
- list = list.Reverse().Reverse()
- obj = JSON.parse(JSON.stringify(obj))
No Unexpected Changes
- list.Length = list.Sort().Length
Hard to Prove, Easy to Verify
- Sorting: each element should be greater than or equal to the previous element
- Tokenizing: concatenating tokens with separator interposed should equal the original string, number of tokens should equal number of separators in original string - 1

Is PBT actually used in the "real world"?

In short, yes. Property-based testing is absolutely used to solve real-world problems.

Volvo

Volvo has employed QuviQ's QuickCheck to validate third-party components conforming to communication bus standards. John Hughes and his team analyzed 3k pages of requirements, wrote 20k lines of QuickCheck tests, and applied those properties to 1MM+ lines of vendor code. By checking their properties generated around the specifications, they found 200 issues that had slipped through the test fixtures that the vendors had generated... and 100 of those issues were in the actual specifications themselves. By refining the code analyzed (as well as the defining specifications), property-based testing has saved lives.

Clojure

Clojure (a LISP variant on the JVM) uses immutable data structures by default. However, mutable datatypes are available for performance increases. Using property-based testing, an issue was found when converting to/from transient (mutable) and persistent (immutable) structures that none of the example-based tests (re: unit tests). However, property-based testing was able to readily reproduce the issue. A diff patch was provided to Cognitect (the developers/maintainers of Clojure), and it was incorporated in the Clojure 1.6 release.

Personal experience

I've also used property-based testing on a project where we were calculating network performance by various metrics (QoS, packet type, packet size, etc.). I had to prove that my packet classification and matching algorithm wouldn't provide false matches, so I set my CI server up to generate 100 million different packets, distributed between the different packet types we were interested in. As of now, we have randomly generated over 10 billion packets and have not had a single false matches.

What are some awesome parts to look for in PBT suites?

Shrinking

While large inputs may produce the errors, several property-based test suites (most QuickCheck variants) will attempt to shrink the input sequence to the smallest possible that will reproduce the error. The smaller the input, the easier it is to reproduce and fix.

Race Conditions

Race conditions are notoriously hard to set up via example-based testing. Some PBT suites (notably, QuviQ's QuickCheck in Erlang) can perform many actions in parallel in order to test if the actions, performed in any serial combination, would produce the same output. If the parallelized version can not be matched to a serialized version, then the race condition is identified and shrunk to the smallest possible set of operations to reproduce the error.

Custom generators

You may want to limit the input domain you are testing, or have data structures constructed in a particular fashion. Most PBT suites will allow you to create custom generators, and many common custom ones are generally included as well – limiting strings to printable characters, constraining floating point values, etc.
If you're migrating from one algorithm to another, a PBT test suite that passes the same input to both the old and new implementations can be used to ensure that the new implementation produces the same output.

Where can I learn more?

Thankfully, there are several good learning resources for property-based testing lurking around on the Internet. Here are some of the ones I've found to be extremely helpful.

Don't Put Personal Information in Code!

Jason Steinhauser — Mon, 04 Jun 2018 22:14:52 +0000

Normally, I wouldn't cringe during a code review when I see

var testValue = 716.1976;

BUT when I include the rest of the line...

var testValue = 716.1976;      // Carl's birthday

Well, then I start to question what in the world you were thinking! (Note: some names and dates have been changed to protect the innocent)

Most online tutorials involving DevOps or API keys will warn you to not include keys and passwords in your repo; those are the sort of things that are best relegated to environment variables (or some other secure method). But usually you don't come across tutorials warning to exclude Personally Identifiable Information (PII).

What is PII?

In short, PII is anything that you can use to figure out who somebody is. You usually hear it in identity theft discussion. Several things are included as PII, such as:

Birth date
Home city
Social Security Number (in the U.S.)
High school
Location of birth

You know all those security questions for password recovery? Yeah, all of that is PII.

What can you do with that?

Well, there are several websites that, given a few tidbits of your personal info, can be used to find out all sorts of things. Current and past addresses. Names of relatives. Former/maiden names. All with a few simple keystrokes on Pipl or Intelius and you can find a lot of information.

Going back to the comment that started all of this, I didn't have a surname for "Carl," but I did have the developer's last name thanks to a comment block at the top of the file (not to mention, a commit history). A quick search got me his address, Facebook profile, occupation (a nurse!), and an age that corroborated his birthday in the code.

Did you send "Carl" a birthday card?

I totally did... in my mind. But then, I played through how freaky that would be for Carl, and how mad he would be at "Lucy" despite no ill intentions. I didn't want to fracture any relationships solely to make a point... But "Lucy" is now well aware of the ramifications.

Is this an isolated incident?

I know people that have hidden spouse's birthdays in code as "magic" or "test" values... but they've never explicitly stated that in comments like "Lucy" did. Has anyone else come across any issue like this before? How did you resolve it? And how would you have liked to resolve it?

Please don't use real names/info if at all possible!