DEV Community: Cohere

Episode 2 of #DebuggingDowntime is live-streaming tomorrow!

Zee — Tue, 29 Jan 2019 01:46:37 +0000

Hey everybody!

Two weeks ago, we live-streamed a mob-programming session where we attempted to debug and resolve a recurring downtime problem for Notabli. Unfortunately, it didn't completely work, so we're heading back into the trenches tomorrow at 10 AM PT / 1 PM ET. You can join us for the live-session tomorrow, or you can wait until we publish the uncut version to our RealWorldCode YouTube channel.

While all live-streamed, un-cut and rough-cut sessions will be available free to stream, we want to be able to afford to hire a professional video editor to tighten everything up and help us highlight the diamond in the rough.

To finance that, we are selling a Debugging Downtime season pass. This gives you the rights to download the uncut, rough cut, and gold cut content, as well as funds our hiring of a rough cut of each of the sessions and if we can make enough to justify it, a tight cut that highlights the diamonds in the rough.

Lessons Learned from our First #RealWorldCode Live Stream

Zee — Mon, 14 Jan 2019 18:31:21 +0000

Earlier this month, Betsy and I tried to live stream a pairing session and failed miserably. But! We had fun, so we spent about 30m performing a retrospective on our Debugging a Race Condition in Cypress event. To expose our ignorance (and therefore, learn more!) we've made our retro notes available to you.

The joy of retrospectives is we now have a plan to make the stream flow much more smoothly! The angst of retrospectives is that it can be touch to talk about failure-modes in a positive, proactive way. Thankfully though our years of working together, we've developed a level of physchological safety that allows us to have healthy, reflective conversations.

How We'll Do Better Next Session

The next live-streaming sessions are going to be longer! Yes, Longer. The main reason for this is we realized that due to A) us having solved the problem before and B) us having logistical difficulties, we didn't get much of an opportunity to enter a state of productive flow.

We know this may be… boringer for you, but we're hoping it will allow us to dive deeper into insights in a more casual manner; so we can pull out the diamonds in the rough out and provide them in more easily accessible chunks.

We're also planning to have someone who is not streaming monitoring the #RealWorldCode hashtag and mentions to Cohere's Twitter account to make sure that if logistical issues arise we can resolve them in a timely and inclusive manner.

How You Can Attend our Next Livestream

To attend our next livestream, Preventing Cascade Failures Caused by Excessive Concurrent Writes on January 15th, from 10 AM PT / 1 PM ET to 12 PM PT / 3PM ET, you may join the stream directly. You may also RSVP via EventBrite by tomorrow, January 15th at 8 AM PT/ 11AM ET. This should work better than the horrible form I put on the website last time.

Once again, you should be able to bookmark this link which will give you a window into the stream when we go live.

We're looking forward to seeing you there! If it doesn't work or you have any questions, reply to this post or tweet at Cohere with the hashtag #RealWorldCode and we'll do our best to get you in quickly <3.

Zee, Betsy and Jackson

Incident Report! How We Used the OODA Loop to Triage a Amazon RDS Cascade Failure for a Heroku-hosted Rails App

Zee — Thu, 10 Jan 2019 23:48:04 +0000

This article was originally posted on Cohere's Website with the title Real World Code Incident Report! How We Used the OODA Loop to Triage a Amazon RDS Cascade Failure for a Heroku-hosted Rails App

Notabli reached out to us the day after Christmas because their API was falling over. Their Friendly Engineer, Ty Rauber was working hard with their Head of Product, Jackson Latka, to bring it back up. They wanted a second pair of programmer-eyes, and I was bored. Isn't that what everyone uses the Holidays for? Triaging production issues for fun?

Triaging a Cascade Failure - OODA Loop 1 - Maybe it's the N+1 Query?

We hopped into a Zoom room together with their Product Lead, Jackson Latka and began the debugging process. When debugging, I try to consciously follow the OODA loop: Observe, Orient, Decide and Act.

First, Observe! Jackson granted me rights to manage their Amazon Relational Database Service (RDS), Heroku for hosting, Papertrail for logging and Scout for application monitoring. Huge shout-out to Jackson for the trust he granted, it's the main reason this worked so well. Teams move at the speed of trust, after all!

Second, Orient! I began exploring the architecture with Ty. Notabli's API is a Ruby on Rails application hosted on Heroku. It uses Sidekiq to perform background jobs, many of which are computationally intensive as they process video and images. Further, the main endpoint is a stream of Moments, which is resolved by a number of data fetching operations, one of which is an N + 1 Query, the other of which is a massive cross-table inner join.

Third, Decide! Being (primarily) a programmer, I immediately decided to attempt to resolve the problem via code.

Fourth, Act! Because they use the jsonapi-resources gem, a library I'm personally unfamiliar with, it was difficult to find a seam to begin resolving the query problems. Ty and I blundered through, poking and prodding; but after 15 minutes, we decided this was a dead-end for now.

On reflection, I realized I had violated one of my principles of effective downtime recovery: Attempt to solve downtime first at the infrastructure level before making changes that would require a redeploy of the application. I don't follow this as hard and fast, but it's served me well over the decades.

OODA Loop 2 - Maybe it's the disk?

We went back to observing the AWS dashboard and what did I find? The first major outage had happened when the system ran of storage space!

After briefly kicking myself for not noticing that sooner; I dug in a bit further and discovered that right before failure, the server was running at the maximum IOPS it could provide! IOPS, for those whose eyes glaze over when they see a Wikipedia article, is the throughput of read and write disk operations.

Diving in deeper we realized it was the write operations that were using up all the IOPS. I resumed kicking myself for wasting ~30 minutes on mitigating a read related issue; and proposed a two-step bandaid:

We upgrade the disk, as AWS grants IOPS based on disk-size; this should bring the application back up and allow it to handle the increased traffic.
Once the system is stable, we look for why there are so many writes and determine if any of them can be removed from the system.

This, thankfully, worked. The application came back up! It maintained a reasonable degree of performance, and there was a reproducible way to bring the application back up in the event the database fell over again.

Unfortunately, because all their work is done on a volunteer basis; we ran out of time to perform step 2!

To help fund the time we're going to spend fixing it, we're offering our friends and followers the opportunity to join us free or with donation on January 15th, 2019 at 10AM PT/1PM ET for a two hour Real World Code Livestream: Preventing Excessive Concurrent Write Induced Cascade Failures in a Heroku Hosted, Amazon RDS Backed Ruby on Rails App. We're not sure if we'll fully diagnose and debug why the write cascade happens in a single session, so we may have a second session on January 29th at the same time!

Haskell, Vectors, and Implicit Knowledge

Betsy Haibel — Thu, 03 Jan 2019 16:01:36 +0000

Hey folks!

This piece is a few years old now, but I'm republishing it on dev.to as I move more of my stuff here. It's kind of personal. Unlike a lot of my other work, it's not about teaching folks how to tech better -- it's about why it's so important to me to do so in an accessible way.

This winter I was teaching myself Haskell.

I'd tried before, with everyone's darling Learn You a Haskell for Great Good. My old boss had compared its "offbeat" approach to _why's Poignant Guide to Ruby, which I love. Both have an informal tone, but the similarities stop there: where the Poignant Guide has adorable cartoon foxes, Learn You a Haskell has upsetting bro humor. I think I snapped at one of the (many) fat jokes.

This time around I tried Real World Haskell, and at first things went much better. The material was more usefully organized, and the examples and exercises better suited to my learning style.
Then I came to The Problem.

The Problem

The problem set at the end of Real World Haskell’s third chapter contains the following sequence:

Consider three two-dimensional points a, b, and c. If we look at the angle formed by the line segment from a to b and the line segment from b to c, it either turns left, turns right, or forms a straight line. Define a Direction data type that lets you represent these possibilities.

Write a function that calculates the turn made by three 2D points and returns a Direction.

Define a function that takes a list of 2D points and computes the direction of each successive triple. Given a list of points [a,b,c,d,e], it should begin by computing the turn made by [a,b,c], then the turn made by [b,c,d], then [c,d,e]. Your function should return a list of Direction.

Using the code from the preceding three exercises, implement Graham's scan algorithm for the convex hull of a set of 2D points. You can find good description of what a convex hull is, and how the Graham scan algorithm should work, on Wikipedia.

Problem 9 was pretty simple:

data Direction = Left | Right | Straight

Problem 10 took me a month.

I figured it had to be simple; if it were a difficult geometry problem, it wouldn’t be in a beginners' Haskell book, it would be in a "Haskell as applied to tricky math" book. But approach after approach after approach failed me. I compared slopes. I tried complicated conditionals based on what quadrant each line was in when its base was set at origin.

Did you know that Haskell has two Data.Vector modules? There's one in the standard library that deals with Vectors like the array-like CS concept. There's one on Hackage that deals with Vectors like the geometry and second-year algebra concept. They are called the same thing. I know this now, because I found the docs for the latter, imported Data.Vector like a good girl, and spent aeons of subjective time banging my head against arity mismatch errors before giving up on a dot-product-based solution.

Eventually I finally hammered out all of the special cases.

I’d been pushing myself to finish, rather than cutting my losses and moving on in the book, because I figured it would feel good when I finally did. It didn’t. I just felt stupid, because it had taken me a month to figure out what the authors of the book had clearly expected to be simple.

I moved on to Problem 12. Looked up the Graham Scan algorithm on Wikipedia, as ordered.

Again, determining whether three points constitute a "left turn" or a "right turn" does not require computing the actual angle between the two line segments, and can actually be achieved with simple arithmetic only. For three points , and , simply compute the z-coordinate of the cross product of the two vectors and , which is given by the expression . If the result is 0, the points are collinear; if it is positive, the three points constitute a "left turn" or counter-clockwise orientation, otherwise a "right turn" or clockwise orientation (for counter-clockwise numbered points).

Reader, I saw fire.

At first I was angry at myself, because there had been a straightforward answer and '’d "just been too stupid" to get it. Then I was angry at the authors for not putting that "simple arithmetic" answer in the text for problem 10. Then I was furious at the authors — because I realized why they hadn't.

They thought it was too simple to explain. They thought that anyone learning Haskell would have retained all the random topics that are contained in high school precalculus. They thought that anyone learning Haskell would be the Kind Of Person who just "naturally" remembers that sort of stuff.

And now, because it's a woman writing this, you're probably assuming that this is a rant about how you don't need to know math to computer.

Think again. I am That Kind Of Person. I started programming at either twelve or eight, depending on whether you count HyperCard. I learned Scheme in ninth grade. I am That Kind of Person, and the problem still made me feel like a fucking idiot. Like an impostor. Like I ought to just give up.

Over whether I could remember offhand that vector cross products were not commutative, and what implications this had for turn direction problems.

This could be a rant about the arrogance of the functional programming ivory tower. But a lot of people have made that rant already, and I'm more interested in self-reflection than in being yet another Ruby programmer who’s snotty about non-Rubyists.

Closer to Home

I teach people to program in my free time. I co-organize a drop-in group called Learn Ruby in DC, so every other week I walk people of all different skill levels through how to code. This includes honest-to-god raw beginners, as well as apprentices and juniors.

There is one understanding gap that every single one of them runs into at some point. They mix up local variables and instance variables, or variables and strings, or ivars and symbols — the exact mixup doesn't matter. They respond by randomly adding and subtracting @s and colons and quotation marks until it works, and then they sigh relief. This happens because they don’t have mental scaffolding for the differences between variables and symbols, between locals and ivars — to them, they’re just words that may or may not have magic before them, and so you fiddle with the magic until it runs.

It always takes me a few minutes of observation to figure out that that's what's happening in their heads. I’ve been writing Ruby since 2008; the scope differences between raw ivars and accessors and local variables are like breathing now. My intuitive brain assumes that everyone knows this, and I need to back up and remind it that that’s not true.

I need to do this even though in 2008, as a woman who’d worked with various conventional languages for 8 years, I'd struggled like hell to internalize that difference myself.

Am I making my students feel stupid? I hope not, but… sometimes I probably am.

Where do we go?

The educational materials we create encode our values -- often accidentally. _why's Poignant Guide reflects an ethos where programming is a joyful, creative activity. Learn You a Haskell conveys a worldview that fat jokes are hilarious fun. Real World Haskell assumes that all truly educated people remember vector-math intricacies off the top of their heads.

The values of our educational materials create our community values. People whose assumptions mesh with theirs proceed onward to community participation; people whose assumptions and values don't leak out of the pipeline. I think Haskell is a really pretty language, but I can't tell if the Haskell community wants me.

It’s human to forget that knowledge one has internalized is not common knowledge. It's human to be a poor teacher. It's human to confuse students; to make them feel terrible. It's still cruel. And when we encode that cruelty into our educational materials -- however accidentally -- we turn surviving that cruelty into something we value above all.

What implicit knowledge is needed in your language of choice? How do you deal with that around juniors?

If you found this post helpful, I'd love it if you subscribed to my company's monthly newsletter! We share some of the things we've found valuable each month, as well as exclusive insights into being a more effective programmer and engineering leader.

Cohere 2018 Year in Review - Reflections on Financial, Mental, Physical and Relational Health

Zee — Thu, 03 Jan 2019 00:44:09 +0000

In 2017, Jennifer, Betsy, and I were burnt out. Over the past two years, we had grown an engineering team from 4 to 40 and extended a product that served hundreds of smaller lending institutions to be the first production, cloud-hosted software as a service at one of the largest banks in America.

We'd done the near impossible and made lifelong friends while doing it. More importantly, we had launched the careers of a dozen first-time developers. But, as the dust settled, we had to ask ourselves: was it worth it?

It took sacrifice. I gained 30 pounds. My significant other and I broke up. My alcohol use skyrocketed. I lost touch with friends and lacked the spoons to keep those friendships active.

So we took a step back and concluded that while our financial health was important, our physical, mental, and relational health was just as important.

Origins - Q4 2017 - Formation as a Worker Cooperative

In Q4 2017, we formed a three-person worker cooperative. Our goals were to build an environment where we each could:

Be financially comfortable without compromising our relationships, health, and happiness.
Make space for the things we value outside of work.
Do work we enjoy doing.
Develop long-term financial security independent of client-work.

Initially, we were a bog-standard technology agency. My experience running Zinc and Cognizance, and working for Leandog and Pillar Technology, made tech consulting an easy way to generate revenue. The structure was similar to many small agencies: three partners; a few subcontractors.

In Q4 2017, we earned almost what it would take for us to be financially comfortable. All of our revenue came from shipping features for clients.

My physical health improved. I dropped 10 pounds. My non-work relationships improved. My relationships with my co-founders were mostly solid, with some cracks that were easily attributable to our stage in the norming, forming, storming, performing cycle and us being crispy from our previous job.

We liked getting paid to do good work on hard problems for our clients. But it was impossible to pursue our goal of long-term financial freedom independent of client-work. We were three people billing as near-to-full-time as possible, after all! Further, the lack of slack in our work schedules meant we didn't really get much opportunity to work together.

So we began to experiment.

Experiment 0 - Can We Make a Viable Product?

In December, we went on "break" and attempted to launch an open-source, crowd-funding SAAS: Nourish. It didn't work. We built a proof of concept; but lacked the marketing, design, and business development chops to make it desirable and viable.

Since we could build a proof of concept with a small budget why not sell that to people who do have the product marketing and business development chops? Try to get a cut of the equity pie on the same terms as the seed and angel investors?

Experiment 1 - Q1 2018 - Angel/Seed Startup Launcher

We began by pitching to a few friends and colleagues who had pre-seed startups. We offered a severely restricted rate in exchange for a piece of the equity. No one gave us equity, but we launched both Walden and BlockButler in Q1 2018 for clients who wanted MVPs.

Meanwhile, Jennifer started pitching training and coaching to startups on how to do engineering, hiring, and interviewing and landed our first training client.

But again we saw a high cost to our personal health, happiness, and relationships with one another.

The stakes were too high.

Our clients were relying on us to launch their businesses; most of which were funded via friends and family rounds or their own investment. It was difficult for us to bill at a reasonable rate when the hours necessary to adapt to their learnings rose.

So we cut our rates and fell prey to a common failure mode of good consultants: prioritizing other people's needs too far above our own.

Our revenues per-hour billed declined precipitously. Yet we learned two things!

1. We find more joy in helping teams solve a particular social or technical challenge than typing code.

2. When we do type code, we'd rather do it for stable nonprofits, and for businesses where we're getting paid out of revenues, not retirement accounts, or capital investments.

In Q1 2018 we earned just over what we consider enough revenue to be sustainable, 28% of income came from training and coaching, and 62% of it from typing code. Unfortunately, due to how unbalanced our time for dollars rate was; we compromised our goal to leave space for the things we enjoy outside of work. As a result, my physical activity levels were way down and I began gaining weight again. Further, our working relationships suffered.

Something had to change.

Experiment 2 - Q2 2018 - Technical Training and eCourses

This time we pivoted our market towards mature startups, bootstrapped revenue-positive businesses and nonprofits that seek to fill a niche instead of dominating a market.

This was challenging. First, most technology companies want full-time employees or contractors. We don't want to be full-time anything. Second, bootstrapped, revenue-positive businesses need a pretty significant pain-point in order to justify spending. So we diversified.

Jennifer spearheaded the work of developing a sales and marketing practice for our myriad of coaching and other training offerings. Betsy recorded and launched our first video course, Mastering the Object Oriented Mindset and took full responsibility for our technical coaching clients.

We made our first public offerings of our Interviewer Skills and Real-World Refactoring workshops at RailsConf and the public launch of our Pairing with Privilege workshop at PearConf.

This shifted both Jennifer and Betsy's time and attention investment towards primarily non-billable activities. In order for us to keep making rent, I focused on typing code for existing clients and following up on leads for new delivery projects.

At the end of Q2 2018, our revenues had slipped to halfway between sustainable and survival levels. 77% of those revenues were from delivery and 23% of them were through training and coaching. Our relationships which had begun sliding in Q1 stabilized. Betsy and Jennifer were working closely together, which was great; but I was off all by myself.

We were excited that our revenues were slowly being decoupled from typing code or otherwise billing dollars for hours, but we still struggled to market and do business development for our own products.

Experiment 3 - Q3 2018 - Will Ship for Revenue Share?

In Q2, one of our long-time technical coaching clients had a problem to solve. One of the products they've had for decades felt like it could be a blue ocean in their market niche, but they lacked the funds to invest in integrating it with their main product line.

Over a few months of discussions, we decided to pull the trigger on a deal where we matched their investment dollar for dollar in exchange for a 50% revenue share from the product line we were extending.

Jennifer scaled back her coaching and training marketing and business development efforts and took point on delivering the product with Betsy. The entire project has taken us six months of quarter-time investment, and our partners are demoing it and beginning to transition clients to it in Q1 2019.

At the end of Q3 2018, our revenues had rebounded to slightly below sustainable levels; with 26% of those revenues from the Mastering Object Oriented Programming course, 15% from training and coaching, and 59% from typing code for money. Our relationships were recovering, and my physical and mental health had begun to stabilize.

Experiment 4 - Q4 2018 - Reaching a Broader Market

We stepped into Q4 feeling pretty good. Our non-delivery revenues were trending in the right direction. Our first product launch was a success. There was demand for the services we were offering beyond typing code for money.

But there was another trap we had to navigate. Our customers are predominantly people who already knew us.

This isn't a bad thing! It's given us an almost sustainable livelihood in 2018. But we're not entirely sure people are buying our services because they value the services or because they value us.

This is not a complaint. It's a beautiful feeling to be so well-liked that someone will carve out a few grand from their training budget to work with you. In fact, I'm bragging a little bit and reveling in the feeling. Mmmmm.

But from a long-term business viability perspective, it kept us up at night. So we invested Q4 in building out a more intentional practice of marketing.

And it failed.

We spent a lot of time on self-promotion instead of meeting people where they are. Yes, we're super excited about helping people have more delightful pair-programming experiences and we successfully partnered to kickstart a zine series on the topic but we were spending a lot of the trust we'd earned in the community to do so.

We did not like that our marketing efforts were spending trust instead of building it. We went back to the drawing board and hired a marketing professional. Further, we shifted our marketing efforts from convincing people to buy our shit, to sharing what we were thinking, learning and doing on dev.to.

Jennifer's work in Q1 and Q2 on marketing and business development for our coaching and training services paid off in Q4, capturing 60% of our revenue. Product and revenue share revenue dipped to 5%, and our typing code for money revenue dropped to 35% of our revenues.

My physical and mental health is mostly recovered to a reasonable point. I've reduced my drinking to a drink (or three) one or two days a week. I'm spending my free time doing fun things (like running a very casual gaming guild!) and re-investing in the friendships I'd neglected since 2015.

TL/DR - What We Learned in 2018

In 2018 we learned what kinds of work we didn't enjoy doing. And perhaps just as importantly what kind of work we enjoy that we may even be able to make a living doing. The TL/DR?

We adore helping people solve their most interesting and painful programming and engineering management challenges.
Businesses are pretty willing to pay part-time people to solve a pressing need; especially if it's something that they think is valuable but can't justify the opportunity cost to take the time to do it internally.
Our most effective sales and marketing efforts are giving away free help when we can.
Coaching and training for teams and individuals on nuanced topics such as refactoring, testing, pair programming, hiring, interviewing, management etc. is tough to sell.
Our favorite way to type code is maintaining software with a demonstrably viable business model. We love gently iterating it towards something its customers find more useful and its stakeholders find more profitable.

From a financial perspective, a picture paints a thousand words. The following graph shows our gross revenue, broken down into net revenues and across revenue sharing, product sales, training and coaching, and typing code for money.

We don't know what 2019 holds, but one of the things we're planning to do is shift even further toward technology education, training, and coaching. We probably won't be sharing this much detail too often, but If you'd like to learn a bit more about engineering leadership from us every month you can sign up for our newsletter, or you can follow Jennifer, Betsy, Zee, or Cohere on Twitter.

Watch Us Debug a Race Condition in a Brittle Cypress Test Live on January 3rd, 2019!

Zee — Thu, 27 Dec 2018 22:19:28 +0000

A few weeks ago, we asked the dev.to and broader programming community what programming problems they would like watch being solved on stream in a real-world context. ~40% of you wanted to see us solve a race condition and another 30% wanted to see us solve a brittle JavaScript feature test.

Seeing as Betsy and I are both "yes, and" kind of people (and there happened to be a really good representation of this in one of our technical coaching clients codebases) we figured we could show a race condition in a brittle JavaScript feature test!

If you have:

Struggled to figure out when, exactly, test data setup is being completed
Been baffled by an undefined or null value while chaining Promises
Deleted an end-to-end, browser feature test because "I can show the feature working when I do it, but the browser test is failing!"

Then RSVP to attend our free live-stream on January 3rd from 10 AM PT/1PM ET to 11 AM PT/2PM ET where we will step through diagnosing, debugging, and resolving a race condition in a brittle cypress feature test!

We're both pretty new to the streaming thing and want to limit our exposure to embarrassment; so we're requiring people to RSVP. Attendees will be sent an email a few minutes before the stream starts with a link to join.

We're hoping to record the screen, audio, and video stream for later, but… again.. We're new to streaming video. So no promises.

We'll be monitoring the #RealWorldCode and #UntanglingJS hashtags on Twitter before, throughout and after the stream and would love to hear your questions, thoughts, and insights!

This article was originally published on wecohere.com.

Understanding Git: Configuration of Repositories and Remotes

Zee — Tue, 18 Dec 2018 22:06:48 +0000

This post was originally published by Cohere, we have a number of other articles on Engineering on our website.

Recently, a dev.to community member asked if someone could explain.git/config like they are five.

When I first started attempting to help, I realized that to explain .git/config, we need to start a few steps back. First, by looking at the history of software configuration; then investigating the differences between a remote and a repository is in Git. and finally showing how .git/config pulls them together in a powerful tool that decreases your cognitive load when shoveling code back and forth on the Internet.

Git is difficult by design. That's why I struggle with Git. If you also struggle with Git, that might be why too. Git's deliberate difficulty isn't necessarily a bad thing. But it does mean that becoming fluent in Git requires us to invest in building a deep and accurate mental model.

But first, let's look at how the current model for Unix program configuration came to be. This will let us diagnose unusual behaviors in any application that's configured with dotfiles, not just Git! If you're already familiar with user and directory level dotfiles, you may want to skip ahead to Configuring Git. For those who love history and nuance, read on!

A (Sharply Abridged) History of Software Configuration

Once upon a time, we operated computers by writing a program's instructions directly in binary on punch cards. The program was executed card-by-card until it was complete, at which point the output was printed onto paper.

This meant changing the behavior of a program required physical changes to the cards. Either by injecting new cards or replacing them. There weren't any programs that changed their behavior depending on where or by whom the program was run.

Over time, computers became more robust and the mechanisms for inputting and distributing programs less precarious. More and more people adopted them. And people have preferences. Programs needed a way to know the default behavior the person running it wanted.

So programmers instructed their programs to look for a file specific to the person using the program, load the data stored in that file, and reference it when making decisions about how to behave.

These configuration files are often readable by both humans and computers. This lets both robots and people perform diagnostics and adjustments.

This worked pretty well! People could decide if they want the program to bleep when done instead of blorp, or otherwise adjust details the programmer took the time to expose as configurable.

As programs became more general purpose, the working directory (the location on the filesystem where the person is running the program) became relevant in determining the program's behavior.

People don't want to have to pass in --group-by spending_category when we run a summarize_csv program in my business-finance/ directory! That's a whole extra 26 keystrokes! We want to run summarize_csv, and have the computer understand the rest automatically.

So programmers instructed their program to also look for a configuration file within the current working directory. As personal or project level needs became more complex, these individual files evolved into folders to store the myriad details necessary to meet these needs.

So… what does this have to do with Git?

Configuring Git

Git's purpose is to keep track of changes to files within a particular directory and to allow one to distribute those changes to other git users. It relies on the .git folder within that directory to store both those changes and how it tracks them. The .git/config file in that directory exists to provide default or contextual behavior to the git program. If you poke around inside the .git folder in a project, you'll see all kinds of files and directories that map back to concepts in Git.

When distributing changes, there are two sides to consider:

Where may I find changes that I want to apply?
Where may I send changes so that others may apply them?

Before Github, common practice was for each person kept a local copy of the folders history on their machine as a local repository. They would then grant access to it via HTTP, SSH or email.

This required the repository owner to maintain a server that others could access as a remote repository. They'd also need to do the work to ensure the repository remained accessible to those who wanted it and manage who is allowed to write to the repository.

Developers with write permissions make changes locally and push them to their publicly readable remote server.

Github removed the requirement to manage your own infrastructure when you want other people to be able to read or contribute to your code. Github absorbed the hosting and maintenance costs of the remote and provided powerful tools for offering changes to the project maintainer. While people have been forking (creating a copy of a project so you can customize it) open source projects for decades, Github made it drop-dead easy to fork the original maintainer's remote repository from the original maintainer to a remote repository you own. You can then clone your copy down and push whatever changes to it you like; then suggest to the maintainer that they apply your changes using a pull request.

If you're the maintainer or person contributing a change there are now 4 repositories in play:

The maintainer's local repository
The contributor's local repository
The maintainers GitHub remote that mirrors their local repository
The contributors GitHub remote that mirrors the maintainers GitHub remote

Each repository is accessible via its own unique URL. You may notice you don't have to type that URL each and every time you push to or pull from Github. That's because the git command line program stores this configuration information in the .git/config file. When you crack that file open, you see something like the following structure:

[core]
    repositoryformatversion = 0
    filemode = true
    bare = false
    logallrefupdates = true
    ignorecase = true
    precomposeunicode = true

[remote "origin"]
    url = git@github.com:your-git-username/your-project.git
    fetch = +refs/heads/*:refs/remotes/origin/*

[branch "master"]
    remote = origin
    merge = refs/heads/origin

The syntax of this file mirrors that of the INI file format, which is a human and machine readable format that uses square brackets to group configuration options and key-value pairs that use a single equals sign to assign values to the configuration.

If you want to learn more about what options are available to configure in the .git/config option, you can run the man program from your terminal like so:

man git-config

This opens the (extensive!) manual for git-config and allows you to use the / key to enter "search" mode or use the up and down arrows and page up/page down keys to browse through and search for options. There's a lot of options to wade through! If you want more detail on the "remote" and "branch" configuration groupings, you can read my original reply on dev.to.

For those of us still working our way towards a strong mental model of Git, keep strong! You can do it! But keep in mind that it's OK to not know everything about the tool. All you need is a level of fluency that meets your own needs and desires. You don't need to know all the things just because the technocratic gatekeepers in our industry think you "should."

If you found this post helpful, we'd love it if you subscribed to our monthly newsletter! We share some of the things we've found valuable each month, as well as exclusive insights into being a more effective programmer and engineering leader.

What Rails or JS programming problem(s) do you wish you could watch someone work out?

Zee — Mon, 17 Dec 2018 18:14:20 +0000

Hello Friends!

In 2019, we're shifting our focus away from client work and towards accessible programming education.

We're in a lucky position where we have some real-world, open-or-near-open source production applications that are giving us a lot of nuanced problems to work through. And we're thinking "Why not take the time to package some of that up into something that's useful for people that aren't us?"

If you could ask two people with ~30-ish combined years of experience writing production web applications to walk you through step-by-step how to solve a particular problem in video form what would it be?

Would it be adding a feature to an existing code base over a few hours? Diagnosing and resolving performance issues? Performing a hairy, multi-step refactor?

Maybe something completely different we'd never thought of? Whatever it is, we'd love to hear about it so we can figure out a way to show how we would approach the problem together. Also, if you want you can vote in an informal twitter poll, in which we regret not providing an option for CodeyMcCodeFace.

Zee & Betsy

Engineering Managers: It's Time to Write Better Job Descriptions

Zee — Mon, 10 Dec 2018 19:59:13 +0000

Software engineers have the highest turnover rate of any industry with an average tenure of 2~3 years. Why? Because people leave managers, not companies, and as Nicole Sanchez and Danilo Campos point out, tech is full of poor managers. I've been one of them!

Part of what makes a poor manager is a lack of investment into the day-to-day activities of management and an over-emphasis on the day-to-day activities of a practitioner. One of these day-to-day management activities is understanding the work that is being done, why it's being done, and how it's being done. Because this is a constantly adapting answer, it's important to spend time communicating these things to potential new-hires and existing team members. How else can we expect others to understand what is expected of them? Or what a growth trajectory could look like within your team or broader organization? One way to communicate this is to write job descriptions.

It's very very tempting to rely on job titles as a proxy for a job description. But when we rely mostly on job titles we are implicitly considering three things: the skills we're looking for, the level those skills are at, and what behaviors we want to encourage or discourage. This reduction collapses context and we wind up posting think-pieces about "what does 'senior' even mean anyway?" or twitter rants declaring "devops is distinct from site reliability engineering!". That communication to the future hire or to the team often does not occur.

To communicate the necessary skills and levels and behaviors you seek, write a job description with clearly written expectations about what skills at which level are necessary for the given job titles activities within your team or departments context.

"Ugh" you grunt. "I read my job description, the most important phrase was 'other duties as assigned!'" And you're right. A lot of job descriptions are miserable. They're written under duress by overloaded engineering managers trying to satisfy HR while simultaneously navigating the difficulties of leading an engineering team towards accomplishing the organization's goals.

As a result, job descriptions are often a list of buzzwords tied to the technology stack with years of experience assigned. It's the simplest path towards a somewhat valid job description. Plus it allows the recruiting team to write job posts and perform pass/fail analysis of candidate resumes or brief phone calls.

We can do better, and our current and future employees and team mates deserve better. They deserve a functional understanding of what is expected from them for the day to day and what they can anticipate as a reward for growing their skills.

So how do we write useful job descriptions? Ones that accurately convey expectations and career trajectory in a meaningful way to team members?

Write Better Job Descriptions

When writing job descriptions, explicit is better than implicit. Describe the behavior, activity, or skill that you use day-to-day. . Don't rely on "computer science fundamentals" to convey "Transforms data from event streams into a reportable format."

Second, use language that is growth oriented, not credential oriented. Sure, you can use "computer science degree or relevant experience" as shorthand, but isn't what you really want "has delivered features and bug fixes to production web applications"?

Third, avoid value judgements about the person. "Good engineers perform code review in a timely manner without being an ass," while laudable, places too much emphasis on the actor and not enough on the activity. Instead, "Performs code reviews by providing specific, actionable and compassionate feedback in a timely manner."

Fourth, remove anything that is extraneous. Attempting to put every last detail of what it means to be a senior developer on your team into your job description can wind up encouraging investment in minor activities or skills that are nice to have but not core. Instead, leave enough white-space that your team members can fill in the margins with the unique and beautiful ways they build up your team. These unique skills may even lead to writing a new job description if they wind up being incredibly worthwhile.

Describing Behaviors and Activities

When describing activities for your job description, focus on the activities that people with the role spend a significant amount of time on. If you expect your engineers to invest in test automation, for instance, you would include a behavior of "writing automated tests that validate the use-cases for features under the team's purview."

Further, It is useful to include why the activity is performed or the behavior is encouraged. While "Delivers features and bug fixes at the direction of the business and product team" is a decent start for developers and may even be acceptable for an engineer in their first or second job it fails to encourage developers to understand and act in line with the motivation for the features. This results in a "I do as I'm told" mentality. Instead, consider "Makes discernable progress towards business objectives by delivering features and bug fixes aligned with the direction of the product and business team."

We've made the explicit additional responsibility that the work that is being done drives tangible progress. Now a team member has a better idea of how they'll be assessed and how to adjust their behavior to support that assessment.

Describing Skills

Skills are even harder to describe objectively than behaviors and activities. This is where most job descriptions and job postings fall over, and how you get job listings asking for 8+ years swift experience, I encourage the use of a fluency model to describe skills.

Fluency models focus on the level of mental effort required to leverage a particular skill. Technology organizations greatest costs are time, thought and effort. Making explicit the relationship between time, thought and effort and the skills you are cultivating brings the focus towards what the organization gets out of higher-skilled workers.

Fluency has less to do with the person and more to do with their abilities. This helps us more effectively decouple value judgements, i.e. "Great React Developer." Instead, we can be more specific and say that an associate front end developer "requires active support to design and implement React components with data or event inter-dependencies." It avoids judging the person and sets an expectation that support will be provided. For more senior team members, a corresponding "supports other team members in designing and implementing components with complex data or event inter-dependencies" may be appropriate; and will also signal to your associate level team members that one of the ways they can advance their career is by supporting those around them.

Most people are familiar with fluency as a concept. When we ask questions about fluency we give people the opportunity to communicate in stories. Stories give a higher-fidelity understanding of the person's actual skills. The question "What was the last challenge you struggled with when implementing a feature in React?" is both much easier to ask and answer and gives a much richer field for follow-on questions to dive in and develop a more nuanced view (or detect possible bullshit).

Now Write A Job Description!

Take 30 minutes right now to write a job description. What are the day-to-day activities you perform in your role? What are the skills you use most frequently? Where are the edges of fluency between your role and more or less senior ones?

Next, post it up as a comment on this post on dev.to! I'll do my best to read each and every one and give specific, actionable, and compassionate feedback.

If you like this, we offer a monthly engineering leadership newsletter.

Our newsletter gives you a never-before-seen, fresh-from-our desks article about engineering leadership or programming, as well as summaries and links to other articles or podcasts we found informative.

Plus, it lets you know what kind of public training we're offering over the next few months, often with a pretty decent discount!

Implementation Strategies for Integrating External Services Into Your Application

Zee — Thu, 29 Nov 2018 22:28:40 +0000

Much of modern software relies on external services to take responsibility for some aspect of the product's feature set. This trend will only continue as mostly-single-purpose, non-database, API-first services such as Auth0, Drip, Twilio, Nylas, and Stripe continue to evolve and come to market.

This is a good thing! It provides developers with leverage so we can focus on solving the customer's pain-points instead of special-purpose, very hard problems over and over again.

However, adopting an external service has costs. There are (broadly speaking) three ways developers tend to integrate with these external services, each with different pros and cons. Here they are ranked from 'simplest' to 'most complex':

Directly embedding calls to the service or its SDK into existing application code. In Rails, this may look like calling the service directly from your models or controllers. In React or Vue, you may call these services directly from the components.
Wrapping the service or its provided SDK in a custom library, and calling that library from within your application code.
Writing an internal service (Micro or otherwise) that wraps the external service or its SDK and an internal library for calling that service.

Each of these options are entirely valid. Different organizational contexts, team dynamics, and technical affordances and constraints fit different solutions better. I ask myself a number of questions to figure out which tactic to apply:

Am I trying to learn how the service works and apply it in a proof of concept or am I shipping a production feature?
How closely does the service's or SDK's interface align with the conventions and shared understandings that are baked into the existing codebase? Cognitive load is increased when the affordances of a library conflict with the norms established within the codebase.
Am I interacting with the service in a variety of locations in the codebase? Are there other internal consumers of the service?
Am I testing using grey or black box strategies that don't have access the the processes memory? Or am I testing in a way affords convenient in-process test-doubles or fakes?
What is the likelihood that the service or its SDK will change its interface in a way that requires us to change our code?
How do I intend to handle the service failing? Is it possible to recover without blocking the end-user from using the feature, or am I going to have to tell the user to try again later?

The Simplest Case: Embedding Calls to an External Service Directly

If this is the first implementation of a feature that relies on the service and I can comfortably encapsulate responsibility for handling the service failure modes within the ruby class or node module I'll go with the simplest option: writing the service calls directly into the existing application code.

Getting Stronger: Encapsulating External Service with a Wrapper Library

Once I'm hitting more complex failure cases, using the service across a variety of features, or experiencing friction due to conflicts between the affordances the library affords and the norms established within the codebase I tend to sprout an internal library that wraps the external SDK. The internal library starts as a very small class with a method that calls the SDK, or if there are several methods and the interface seems good enough I'll apply the delegator pattern using Forwardable or ActiveSupport's delegate core extension.

Advanced Hardening: Encapsulating the External Service with an Internal Service

If the application deployment and configuration ecosystem is already geared towards inter-service communication, and there are other internal services that consume the external service I may take it a step further and sprout an internal service that encapsulates the external service.

Keeping it Together

The end goal in these cases is not to make a canonical, brilliant adapter that perfectly encapsulates the service. Instead, it's to help future-me and my teammates by:

Leaning into our existing conventions and designs.
Providing a seam that allows us to inject behavior or change naming in order to more accurately reflect what we're using the library for.
Provide an initial pattern for us to follow (or adapt) as we implement new features with the service.
Ensures that if the underlying API changes, we can update the usages in one place.

External services are powerful tools and can save many thousands of dollars in engineering time. Investing in integrating them in an easier-to-maintain and adapt manner will pay dividends within months. Next time you're about to add in a new service:

Start with the cheapest and easiest solution: embedding the external service directly into your existing classes and modules.
As complexity increases, extract a new class or module which encapsulates the responsibility in a manner that fits with your existing patterns.
If you're on the services bandwagon, consider extracting an internal service once it becomes useful; Say when you have a few different teams and their services using the same external service.

For further reading, I'd start with Alistair Cockburn and Michael Feathers article on the c2 wiki on Ports and Adapters (aka Hexagonal Architecture).

If you liked this, consider signing up for free engineering leadership newsletter. We offer monthly insights, both technical and social that help you be a more effective engineering leader.

Pair programming problems are not a smell.

Zee — Fri, 16 Nov 2018 20:42:38 +0000

Earlier this year, Betsy and I sat down and wrote down a few dozen types of dysfunctional behaviors we’ve experienced while pair programming, then sorted them into categories:

Taking up too little space (ex: lost but not speaking up)
Taking up too much space (ex: keyboard hog)
Pair has left you behind (ex: they’re on their phone and you don’t know why)
You’ve left your pair behind (ex: you’re tunnel visioned and typing without talking)
Out of sync with pair (ex: over-correcting the other person’s typos)

Descriptions sorted into categories.... these looked a lot like programming smells! this was familiar territory! Programming smells are solved with recipes, and this was a pattern we could follow. We set out to create a recipe index, but quickly found that even a single recipe creation was impossible.

Here’s the thing. Programming smells and recipes work because they are scenarios that can be addressed without understanding the why. It doesn’t really matter why some code ended up with a primitive obsession, you shove that obsession into a new class and everything works out great. Like programming smells, dysfunctional pairing behaviors are an early warning system that indicate a change is warranted. But that’s where the similarity ends. People are much more complex than programs, and why a pairing problem is manifesting changes how to address it. If your pair seems unengaged, treating them as a problem to be solved will make them feel worse.

Instead of looking at unengagement and other pairing dysfunctions as smells, we should look at them as symptoms. Smells lead us to clean up an area. When you have a smelly house, you can clean it up room by room. Symptoms show a system not operating as expected. When you have a system that’s not working as expected, you have to understand the problem before you can start to address it.

Let’s take a deeper look at one of the pairing symptoms we identified: your pair seems unengaged. Here are just a few reasons why your pair might seem unengaged:

Doesn’t understand what you’re doing in this moment
Doesn’t understand the problem
Doesn’t know how to contribute
Doesn’t want to take up too much space

These are all really different reasons that could be behind the same symptom! Addressing pairing symptoms is kind of similar to addressing health symptoms: you’re more likely to be effective if you know what you’re treating. A few weeks ago I had a bad headache, and painkillers weren’t effective. It turned out my allergies were being really bad, and I needed allergy meds, not headache meds, to address my symptom and make the headache go away. In order to help your pair become engaged, you need to pick the right approach based on why they might not seem engaged.

If you have a good working relationship with your pair, you can probably stop and ask. Hey, you doing ok? you’re more distracted than usual. But if you don’t already have a lot of pre-existing trust in your relationship, it’s not that simple. Your pair might not trust you enough to talk with you about it.

How about picking a neutral-seeming action? Maybe you could suggest they drive? Even that might not go over so well. If you don’t know why your pair seems unengaged, asking them to drive might not address the symptom. What if the cause was that they weren’t sure what you were doing and were scared to ask? And if your pair’s scared to ask, that’s an additional symptom as well..

It might seem silly for your pair to be scared. But your pair has had different life and education and career experiences leading up to your current pairing, and that may give them good reason to be scared:

Poor past experience with speaking up
Poor past experience with displaying ignorance
Inexperience with pair programming
Discomfort with the expected learning style (maybe your team does a code tour, but they feel more comfortable with sitting and reading on their own)

Betsy and I set out to find solutions to common pairing dysfunctions that we could share in a new workshop. What we learned is that there is no one-size-fits-all answer for any of it. How you and your pair have been encouraged and discouraged throughout your life and career have an enormous impact on how you will behave and respond while pairing. You don’t need to know anyone’s past to be a better pair for them in the present. You do need to know not everyone has had the same positive or negative past experiences you’ve had. When you carry that awareness with you into your interactions is when you can be a better pair.

If you liked this article, Cohere publishes a monthly Engineering Leadership newsletter and we would be delighted if you signed up!

A Brief Introduction to GraphQL and Rails

Zee — Fri, 02 Nov 2018 02:25:48 +0000

Over the past 9 months, we've applied GraphQL and Rails as the API platform on client projects. Most of these projects rely on rapid iteration across both the user interface and the data model as we follow the signals for product market fit. Rails and ActiveRecord continue to provide incredibly flexible data management and persistence techniques that is well complemented by GraphQL's fitness as a seam for stubbing out fake data, rolling out new fields, and deprecating existing fields in a safe way that is decoupled from the persistence layer.

That said, GraphQL in general and GraphQL in Rails in particular have a few caveats worth considering before jumping on the train:

While the marketing will tell you the concepts behind GraphQL are simple, GraphQL is quite complex. GraphQL is a distributed database composed of four major parts: a strongly typed data model, a persistence and retrieval layer, a transportation layer, and a client (many of which have caches!). Understanding each of those things independently is difficult enough, but the interactions and subtle tensions between them brings the complexity to a higher level.
GraphQL is significantly less opinionated about design than REST or HATEOS. This forces us to make more decisions about how to design our API and how to make changes happen than relying on the json:api spec.
The GraphQL ecosystem still feels slightly to the left of the break even point on technology maturity life cycle. The community is vibrant and growing. Plus, they're investing in tooling that provide quality of life improvements at a breakneck pace (a blessing and a curse).

All that being said, if you choose to join the ranks of Github, Shopify, and others who are building GraphQL APIs on top of Rails apps there are a few suggestions to help lighten your load.

Join the GraphQL Slack. Yes, we are all a bit overwhelmed by Slack Overflow, but the community there is both friendly and helpful. Plus, it's where the developers of the GraphQL Ruby library hang out. Having direct access to people working in Ruby and GraphQL has leveled me up quite a bit faster than relying purely on my own experience.
Take a gander at the GraphQL Rails Example that pulls some of the lessons learned about implementing and testing a Rails GraphQL server I’ve gleaned over the last 6 months. Yes, the feature tests are using cucumber-js instead of Ruby. That’s because 99% of the work we’ve been doing is in react-native and vue apps with Apollo as the client, and doing feature testing with the same stack your client app uses is immensely useful at finding and isolating odd corner cases.
Read through this excellent guide to designing GraphQL APIs by the folks at Shopify.
Accept that it takes time to find the grain of GraphQL, both as a design philosophy and the Rails implementations particular idioms. I’ve found that my learning has been very much stepwise, with plateaus that eventually result in conceptual breakthroughs.
Embrace service objects. Mutations can quickly become complex, firing off events of their own and making multiple data changes at once. While my mutations and even some queries start as methods, they are quickly get extracted to objects that can be tested at the rspec level once they reach a dozen or so lines of code.

Overall, I’m happy with how well Rails and GraphQL apps serve as the core for an API-driven
platform.