DEV Community: Schneems

A Fast Car Needs Good Brakes: How We Added Client Rate Throttling to the Heroku Platform API Gem

Schneems — Tue, 07 Jul 2020 20:30:14 +0000

When API requests are made one-after-the-other they'll quickly hit rate limits and when that happens:

// Detect dark theme var iframe = document.getElementById('tweet-1138899094137651200-432'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1138899094137651200&theme=dark" }

That tweet spawned a discussion that generated a quest to add rate throttling logic to the platform-api gem that Heroku maintains for talking to its API in Ruby.

If the term "rate throttling" is new to you, read Rate limiting, rate throttling, and how they work together

The Heroku API uses Genetic Cell Rate Algorithm (GCRA) as described by Brandur in this post on the server-side. Heroku's API docs state:

The API limits the number of requests each user can make per hour to protect against abuse and buggy code. Each account has a pool of request tokens that can hold at most 4500 tokens. Each API call removes one token from the pool. Tokens are added to the account pool at a rate of roughly 75 per minute (or 4500 per hour), up to a maximum of 4500. If no tokens remain, further calls will return 429 Too Many Requests until more tokens become available.

I needed to write an algorithm that never errored as a result of a 429 response. A "simple" solution would be to add a retry to all requests when they see a 429, but that would effectively DDoS the API. I made it a goal for the rate throttling client also to minimize its retry rate. That is, if the client makes 100 requests, and 10 of them are a 429 response that its retry rate is 10%. Since the code needed to be contained entirely in the client library, it needed to be able to function without distributed coordination between multiple clients on multiple machines except for whatever information the Heroku API returned.

Making client throttling maintainable

Before we can get into what logic goes into a quality rate throttling algorithm, I want to talk about the process that I used as I think the journey is just as fascinating as the destination.

I initially started wanting to write tests for my rate throttling strategy. I quickly realized that while testing the behavior "retries a request after a 429 response," it is easy to check. I also found that checking for quality "this rate throttle strategy is better than others" could not be checked quite as easily. The solution that I came up with was to write a simulator in addition to tests. I would simulate the server's behavior, and then boot up several processes and threads and hit the simulated server with requests to observe the system's behavior.

I initially just output values to the CLI as the simulation ran, but found it challenging to make sense of them all, so I added charting. I found my simulation took too long to run and so I added a mechanism to speed up the simulated time. I used those two outputs to write what I thought was a pretty good rate throttling algorithm. The next task was wiring it up to the platform-api gem.

To help out I paired with a Heroku Engineer, Lola, we ended up making several PRs to a bunch of related projects, and that's its own story to tell. Finally, the day came where we were ready to get rate throttling into the platform-api gem; all we needed was a review.

Unfortunately, the algorithm I developed from "watching some charts for a few hours" didn't make a whole lot of sense, and it was painfully apparent that it wasn't maintainable. While I had developed a good gut feel for what a "good" algorithm did and how it behaved, I had no way of solidifying that knowledge into something that others could run with. Imagine someone in the future wants to make a change to the algorithm, and I'm no longer here. The tests I had could prevent them from breaking some expectations, but there was nothing to help them make a better algorithm.

The making of an algorithm

At this point, I could explain the approach I had taken to build an algorithm, but I had no way to quantify the "goodness" of my algorithm. That's when I decided to throw it all away and start from first principles. Instead of asking "what would make my algorithm better," I asked, "how would I know a change to my algorithm is better" and then worked to develop some ways to quantify what "better" meant. Here are the goals I ended up coming up with:

Minimize average retry rate: The fewer failed API requests, the better
Minimize maximum sleep time: Rate throttling involves waiting, and no one wants to wait for too long
Minimize variance of request count between clients: No one likes working with a greedy co-worker, API clients are no different. No client in the distributed system should be an extended outlier
Minimize time to clear a large request capacity: As the system changes, clients should respond quickly to changes.

I figured that if I could generate metrics on my rate-throttle algorithm and compare it to simpler algorithms, then I could show why individual decisions were made.

I moved my hacky scripts for my simulation into a separate repo and, rather than relying on watching charts and logs, moved to have my simulation produce numbers that could be used to quantify and compare algorithms.

With that work under my belt, I threw away everything I knew about rate-throttling and decided to use science and measurement to guide my way.

Writing a better rate-throttling algorithm with science: exponential backoff

Earlier I mentioned that a "simple" algorithm would be to retry requests. A step up in complexity and functionality would be to retry requests after an exponential backoff. I coded it up and got some numbers for a simulated 30-minute run (which takes 3 minutes of real-time):

Avg retry rate: 60.08 %
Max sleep time: 854.89 seconds
Stdev Request Count: 387.82

Time to clear workload (4500 requests, starting_sleep: 1s):
74.23 seconds

Now that we've got baseline numbers, how could we work to minimize any of these values? In my initial exponential backoff model, I multiplied sleep by a factor of 2.0, what would happen if I increased it to 3.0 or decreased it to 1.2?

To find out, I plugged in those values and re-ran my simulations. I found that there was a correlation between retry rate and max sleep value with the backoff factor, but they were inverse. I could lower the retry rate by increasing the factor (to 3.0), but this increased my maximum sleep time. I could reduce the maximum sleep time by decreasing the factor (to 1.2), but it increased my retry rate.

That experiment told me that if I wanted to optimize both retry rate and sleep time, I could not do it via only changing the exponential factor since an improvement in one meant a degradation in the other value.

At this point, we could theoretically do anything, but our metrics judge our success. We could put a cap on the maximum sleep time, for example, we could write code that says "don't sleep longer than 300 seconds", but it too would hurt the retry rate. The biggest concern for me in this example is the maximum sleep time, 854 seconds is over 14 minutes which is WAAAYY too long for a single client to be sleeping.

I ended up picking the 1.2 factor to decrease that value at the cost of a worse retry-rate:

Avg retry rate: 80.41 %
Max sleep time: 46.72 seconds
Stdev Request Count: 147.84

Time to clear workload (4500 requests, starting_sleep: 1s):
74.33 seconds

Forty-six seconds is better than 14 minutes of sleep by a long shot. How could we get the retry rate down?

Incremental improvement: exponential sleep with a gradual decrease

In the exponential backoff model, it backs-off once it sees a 429, but as soon as it hits a success response, it doesn't sleep at all. One way to reduce the retry-rate would be to assume that once a request had been rate-throttled, that future requests would need to wait as well. Essentially we would make the sleep value "sticky" and sleep before all requests. If we only remembered the sleep value, our rate throttle strategy wouldn't be responsive to any changes in the system, and it would have a poor "time to clear workload." Instead of only remembering the sleep value, we can gradually reduce it after every successful request. This logic is very similar to TCP slow start.

How does it play out in the numbers?

Avg retry rate: 40.56 %
Max sleep time: 139.91 seconds
Stdev Request Count: 867.73

Time to clear workload (4500 requests, starting_sleep: 1s):
115.54 seconds

Retry rate did go down by about half. Sleep time went up, but it's still well under the 14-minute mark we saw earlier. But there's a problem with a metric I've not talked about before, the "stdev request count." It's easier to understand if you look at a chart to see what's going on:

Here you can see one client is sleeping a lot (the red client) while other clients are not sleeping at all and chewing through all the available requests at the bottom. Not all the clients are behaving equitably. This behavior makes it harder to tune the system.

One reason for this inequity is that all clients are decreasing by the same constant value for every successful request. For example, let's say we have a client A that is sleeping for 44 seconds, and client B that is sleeping for 11 seconds and both decrease their sleep value by 1 second after every request. If both clients ran for 45 seconds, it would look like this:

Client A) Sleep 44 (Decrease value: 1)
Client B) Sleep 11 (Decrease value: 1)
Client B) Sleep 10 (Decrease value: 1)
Client B) Sleep 9 (Decrease value: 1)
Client B) Sleep 8 (Decrease value: 1)
Client B) Sleep 7 (Decrease value: 1)
Client A) Sleep 43 (Decrease value: 1)

So while client A has decreased by 1 second total, client B has reduced by 4 seconds total, since it is firing 4x as fast (i.e., it's sleep time is 4x lower). So while the decrease rate is equal, it is not equitable. Ideally, we would want all clients to decrease at the same rate.

All clients created equal: exponential increase proportional decrease

Since clients cannot communicate with each other in our distributed system, one way to guaranteed proportional decreases is to use the sleep value in the decrease amount:

decrease_value = (sleep_time) / some_value

Where some_value is a magic number. In this scenario the same clients A and B running for 45 seconds would look like this with a value of 100:

Client A) Sleep 44
Client B) Sleep 11
Client B) Sleep 10.89 (Decrease value: 11.00/100 = 0.1100)
Client B) Sleep 10.78 (Decrease value: 10.89/100 = 0.1089)
Client B) Sleep 10.67 (Decrease value: 10.78/100 = 0.1078)
Client B) Sleep 10.56 (Decrease value: 10.67/100 = 0.1067)
Client A) Sleep 43.56 (Decrease value: 44.00/100 = 0.4400)

Now client A has had a decrease of 0.44, and client B has had a reduction of 0.4334 (11 seconds - 10.56 seconds), which is a lot more equitable than before. Since some_value is tunable, I wanted to use a larger number so that the retry rate would be lower than 40%. I chose 4500 since that's the maximum number of requests in the GCRA bucket for Heroku's API.

Here's what the results looked like:

Avg retry rate: 3.66 %
Max sleep time: 17.31 seconds
Stdev Request Count: 101.94

Time to clear workload (4500 requests, starting_sleep: 1s):
551.10 seconds

The retry rate went WAAAY down, which makes sense since we're decreasing slower than before (the constant decrease value previously was 0.8). Stdev went way down as well. It's about 8x lower. Surprisingly the max sleep time went down as well. I believe this to be a factor of a decrease in the number of required exponential backoff events. Here's what this algorithm looks like:

The only problem here is that the "time to clear workload" is 5x higher than before. What exactly is being measured here? In this scenario, we're simulating a cyclical workflow where clients are running under high load, then go through a light load, and then back to a high load. The simulation starts all clients with a sleep value, but the server's rate-limit is reset to 4500. The time is how long it takes the client to clear all 4500 requests.

What this metric of 551 seconds is telling me is that this strategy is not very responsive to a change in the system. To illustrate this problem, I ran the same algorithm starting each client at 8 seconds of sleep instead of 1 second to see how long it would take to trigger a rate limit:

The graph shows that it takes about 7 hours to clear all these requests, which is not good. What we need is a way to clear requests faster when there are more requests.

The only remaining option: exponential increase proportional remaining decrease

When you make a request to the Heroku API, it tells you how many requests you have left remaining in your bucket in a header. Our problem with the "proportional decrease" is mostly that when there are lots of requests remaining in the bucket, it takes a long time to clear them (if the prior sleep rate was high, such as in a varying workload). To account for this, we can decrease the sleep value quicker when the remaining bucket is full and slower when the remaining bucket is almost empty. To express that in an expression, it might look like this:

decrease_value = (sleep_time * request_count_remaining) / some_value

In my case, I chose some_value to be the maximum number of requests possible in a bucket, which is 4500. You can imagine a scenario where workers were very busy for a period and being rate limited. Then no jobs came in for over an hour - perhaps the workday was over, and the number of requests remaining in the bucket re-filled to 4500. On the next request, this algorithm would reduce the sleep value by itself since 4500/4500 is one:

decrease_value = sleep_time * 4500 / 4500

That means it doesn't matter how immense the sleep value is, it will adjust fairly quickly to a change in workload. Good in theory, how does it perform in the simulation?

Avg retry rate: 3.07 %
Max sleep time: 17.32 seconds
Stdev Request Count: 78.44

Time to clear workload (4500 requests, starting_sleep: 1s):
84.23 seconds

This rate throttle strategy performs very well on all metrics. It is the best (or very close) to several metrics. Here's a chart:

This strategy is the "winner" of my experiments and the algorithm that I chose to go into the platform-api gem.

My original solution

While I originally built this whole elaborate scheme to prove how my solution was optimal, I did something by accident. By following a scientific and measurement-based approach, I accidentally found a simpler solution that performed better than my original answer. Which I'm happier about, it shows that the extra effort was worth it. To "prove" what I found by observation and tinkering could be not only quantified by numbers but improved upon is fantastic.

While my original solution had some scripts and charts, this new solution has tests covering the behavior of the simulation and charting code. My initial solution was very brittle. I didn't feel very comfortable coming back and making changes to it; this new solution and the accompanying support code is a joy to work with. My favorite part though is that now if anyone asks me, "what about trying " or "have you considered " is that I can point them at my rate client throttling library, they have all the tools to implement their idea, test it, and report back with a swift feedback loop.

`gem 'platform-api', '~> 3.0'`

While I mostly wanted to talk about the process of writing rate-throttling code, this whole thing started from a desire to get client rate-throttling into the platform-api gem. Once I did the work to prove my solution was reasonable, we worked on a rollout strategy. We released a version of the gem in a minor bump with rate-throttling available, but with a "null" strategy that would preserve existing behavior. This release strategy allowed us to issue a warning to anyone depending on the original behavior. Then we released a major version with the rate-throttling strategy enabled by default. We did this first with "pre" release versions and then actual versions to be extra safe.

So far, the feedback has been overwhelming that no one has noticed. We didn't cause any significant breaks or introduce any severe disfunction to any applications. If you've not already, I invite you to upgrade to 3.0.0+ of the platform-api gem and give it a spin. I would love to hear your feedback.

Get ahold of Richard and stay up-to-date with Ruby, Rails, and other programming related content through a subscription to his mailing list.

Puma 4: Hammering Out H13s—A Debugging Story

Schneems — Fri, 12 Jul 2019 12:53:55 +0000

For quite some time we've received reports from our larger customers about a mysterious H13 - Connection closed error showing up for Ruby applications. Curiously it only ever happened around the time they were deploying or scaling their dynos. Even more peculiar, it only happened to relatively high scale applications. We couldn't reproduce the behavior on an example app. This is a story about distributed coordination, the TCP API, and how we debugged and fixed a bug in Puma that only shows up at scale.

Connection closed

First of all, what even is an H13 error? From our error page documentation:

This error is thrown when a process in your web dyno accepts a connection, but then closes the socket without writing anything to it.One example where this might happen is when a Unicorn web server is configured with a timeout shorter than 30s and a request has not been processed by a worker before the timeout happens. In this case, Unicorn closes the connection before any data is written, resulting in an H13.

Fun fact: Our error codes start with the letter of the component where they came from. Our Routing code is all written in Erlang and is named "Hermes" so any error codes from Heroku that start with an "H" indicate an error from the router.

The documentation gives an example of an H13 error code with the unicorn webserver, but it can happen any time a connection is closed via a server, but there has been no response written. Here’s an example showing how to reproduce a H13 explicitly with a node app.

What does it mean for an application to get an H13? Essentially every one of these errors correlates to a customer who got an error page. Serving a handful of errors every time the app restarts or deploys or auto-scales is an awful user experience, so it's worth it to find and fix.

Debugging

I have maintained the Ruby buildpack for several years, and part of that job is to handle support escalations that are too tricky for our core supporters. In addition to the normal deployment issues, I've been developing an interest in performance, scalability, and web servers (I recently started helping to maintain the Puma webserver). Because of these interests, when a tricky issue comes in from one of our larger customers, especially if it only happens at scale, I take particular interest.

To understand the problem, you need to know a little about the nature of sending distributed messages. Webservers are inherently distributed systems, and to make things more complicated, we often use distributed systems to manage our distributed systems.

In the case of this error, it didn't seem to come from a customer's application code i.e. they didn't seem to have anything misconfigured. It also only seemed to happen when a dyno was being shut down.

To shut down a dyno two things have to happen, we need to send a SIGTERM to the processes on the dyno which tells the webserver to safely shutdown. We also need to tell our router to stop sending requests to that dyno since it will be shut down soon.

These two operations happen on two different systems. The dyno runs on one server, the router which serves our requests is a separate system. It's itself a distributed system. It turns out that while both systems get the message at about the same time, the router might still let a few requests trickle into the dyno that is being shut down after it receives the SIGTERM.

That explains the problem then, right? The reason this only happens on apps with a large amount of traffic is they get so many requests there is more chance that there will be a race condition between when the router stops sending requests and the dyno receives the SIGTERM.

That sounds like a bug with the router then right? Before we get too deep into the difficulties of distributed coordination, I noticed that other apps with just as much load weren't getting H13 errors. What did that tell me? It told me that the distributed behavior of our system wasn't to blame. If other webservers can handle this just fine, then we need to update our webserver, Puma in this case.

Reproduction

When you're dealing with a distributed system bug that's reliant on a race condition, reproducing the issue can be a tricky affair. While pairing on the issue with another Heroku engineer, Chap Ambrose, we hit an idea. First, we would reproduce the H13 behavior in any app to figure out what curl exit code we would get, and then we could try to reproduce the exact failure conditions with a more complicated example.

A simple reproduction rack app looks like this:

app = Proc.new do |env|
  current_pid = Process.pid
  signal = "SIGKILL"
  Process.kill(signal, current_pid)
  ['200', {'Content-Type' => 'text/html'}, ['A barebones rack app.']]
end

run app

When you run this config.ru with Puma and hit it with a request, you'll get a connection that is closed without a request getting written. That was pretty easy.

The curl code when a connection is closed like this is 52 so now we can detect when it happens.

$ curl localhost:9292
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
  0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
curl: (52) Empty reply from server

A more complicated reproduction happens when SIGTERM is called but requests keep coming in. To facilitate that we ended up with a reproduction that looks like this:

app = Proc.new do |env|
  puma_pid = File.read('puma.pid').to_i
  Process.kill("SIGTERM", puma_pid)
  Process.kill("SIGTERM", Process.pid)

  ['200', {'Content-Type' => 'text/html'}, ['A barebones rack app.']]
end

run app

This config.ru rack app sends a SIGTERM to itself and it's parent process on the first request. So other future requests will be coming in when the server is shutting down.

Then we can write a script that boots this server and hits it with a bunch of requests in parallel:

threads = []

threads << Thread.new do
  puts `puma > puma.log` unless ENV["NO_PUMA_BOOT"]
end

sleep(3)
require 'fileutils'
FileUtils.mkdir_p("tmp/requests")

20.times do |i|
  threads << Thread.new do
    request = `curl localhost:9292/?request_thread=#{i} &> tmp/requests/requests#{i}.log`
    puts $?
  end
end

threads.map {|t| t.join }

When we run this reproduction, we see that it gives us the exact behavior we're looking to reproduce. Even better, when this code is deployed on Heroku we can see an H13 error is triggered:

2019-05-10T18:41:06.859330+00:00 heroku[router]: at=error code=H13 desc="Connection closed without response" method=GET path="/?request_thread=6" host=ruby-h13.herokuapp.com request_id=05696319-a6ff-4fad-b219-6dd043536314 fwd="<ip>" dyno=web.1 connect=0ms service=5ms status=503 bytes=0 protocol=https

You can get all this code and some more details on the reproduciton script repo. And here's the Puma Issue I was using to track the behavior

Closing the Connection Closed Bug

With a reproduction script in hand, it was possible for us to add debugging statements to Puma internals to see how it behaved while experiencing this issue.

With a little investigation, it turned out that Puma never explicitly closed the socket of the connection. Instead, it relied on the process stopping to close it.

What exactly does that mean? Every time you type a URL into a browser, the request gets routed to a server. On Heroku, the request goes to our router. The router then attempts to connect to a dyno (server) and pass it the request. The underlying mechanism that allows this is the webserver (Puma) on the dyno opening up a TCP socket on a $PORT. The request is accepted onto the socket, and it will sit there until the webserver (Puma) is ready to read it in and respond to it.

What behavior then do we want to happen to avoid this H13 error? In the error case, the router tries to connect to the dyno, it's successful, and since Puma accepts the request, it expects the dyno to handle writing the request. If instead, the socket is closed when it tries to pass on the request it will know that Puma cannot respond. It will then retry passing the connection to another dyno. There are times when a webserver might reject a connection, for example, if the socket is full (default is only to allow 1024 connections on the socket backlog), or if the entire server has crashed.

In our case, closing the socket is what we want. It correctly communicates to the router to do the right thing (try passing the connection to another dyno or hold onto it in the case all dynos are restarting).

So then, the solution to the problem was to close the socket before attempting to shut down explicitly. Here's the PR. The main magic is just one line:

@launcher.close_binder_listeners

If you're a worrier (I know I am) you might be afraid that closing the socket prevents any in-flight requests from being completed successfully. Lucky for us closing a socket prevents incoming requests but still allows us to respond to existing requests. If you don't believe me, think about how you could test it with one of my above example repos.

Testing distributed behavior

I don't know if this behavior in Puma broke, or maybe it never worked. To try to make sure that it continues to work in the future, I wanted to write a test for it. I reached out to dannyfallon who has helped out on some other Puma issues, and we remote paired on the tests using Pair With Tuple.

The tests ended up being not terribly different than our example reproduction above, but it was pretty tricky to get it to have consistent behavior.

With an issue that doesn't regularly show up unless it's on an app at scale, it's essential to test, as Charity Majors would say "in production". We had several Heroku customers who were seeing this error try out my patch. They reported some other issues, which we were able to resolve, after fixing those issues, it looked like the errors were fixed.

Rolling out the fix

Puma 4, which came with this fix, was recently released. We reached out to a customer who was using Puma and seeing a large number of H13s, and this release stopped them in their tracks.

Learn more about Puma 4 below.

// Detect dark theme var iframe = document.getElementById('tweet-1143577608791220224-822'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1143577608791220224&theme=dark" }

Debugging in Ruby—Busting a Year-old Bug in Sprockets

Schneems — Tue, 26 Feb 2019 18:30:20 +0000

Debugging is an important skill to develop as you work your way up to more complex projects. Seasoned engineers have a sixth sense for squashing bugs and have built up an impressive collection of tools that help them diagnose and fix bugs.

I'm a member of Heroku’s Ruby team and creator of CodeTriage and today we’ll look at the tools that I used on a journey to fix a gnarly bug in Sprockets. Sprockets is an asset packaging system written in Ruby that lies at the heart of Rails’ asset processing pipeline.

At the end of the post, you will know how Sprockets works and how to debug in Ruby.

Unexpected Behavior in Sprockets

Sprockets gives developers a convenient way to compile, minify, and serve JavaScript and CSS files. Its extensible preprocessor pipeline has support for languages like CoffeeScript, SaaS, and SCSS. It is included in Rails via the sprockets-rails gem but can also be used in a standalone fashion, for example, to package Sinatra assets.

Earlier this month, we recorded a live-debugging session where we experienced particularly curious issue in Sprockets. We noticed that the bug broke the critical asset precompilation rake task, but only if the name of the project folder was changed between successive task executions. While project folder renames might seem relatively uncommon, they happen frequently on Heroku because each build happens in a unique directory name.

While this bug itself is interesting, what’s even more interesting is learning from our debugging process. You can learn about the tools and steps we use to narrow down the root cause, and ultimately fix the bug.

If you’d like to watch the full debugging session, check out the video or just follow along by reading the text below. We’ll walk through a debug workflow and find the root cause of this bug.

A Guide to Debugging in Ruby

Head-scratching, non-obvious bugs are worth investigating because they may lead to other unnoticed or unreported bugs.

Thankfully, Ruby comes with some powerful debugging tools that are easy to use for beginners. For a nice overview, check out this Ruby debugging guide that covers basics like the difference between p and puts and also discusses a few of the interactive debuggers that are available in the Ruby ecosystem. For the rest of this post, however, you won’t need to know anything more advanced than puts.

Reproducing the Bug

The best way to learn debugging is just to dive in and try it. Let’s set up Sprockets in a local environment.

Clone CodeTriage

We need a Rails app to reproduce this bug so we’ll use an open source example. I am the creator of CodeTriage so it’s natural to use that application to demonstrate the problem, although you can reproduce it with any Rails app that uses Sprockets. CodeTriage has helped developers triage issues for thousands of open-source projects.

First, clone the CodeTriage repository, install dependencies, then switch to a branch that contains the code we need to reproduce the bug. A working Ruby environment is assumed.

$ git clone git@github.com:codetriage/codetriage
$ cd codetriage

$ gem install bundler
$ bundle install

$ cp config/database.example.yml config/database.yml
$ git checkout 52d57d13

Compile the Assets with Rake

Next, execute the following steps to make the bug show up in our local environment.

$ rm -rf tmp/cache
$ rm -rf public/assets

Next, run the rake task for precompiling assets, which should succeed:

$ RAILS_ENV=production RAILS_SERVE_STATIC_FILES=1 RAILS_LOG_TO_STDOUT=1 bin/rake assets:precompile

Rename the Project Folder

Now, change the name of the project directory by copying its files into a new directory called codetriage-after and deleting the old codetriage directory.

$ cd ..
$ cp -r codetriage codetriage-after
$ rm -rf codetriage
$ cd codetriage-after

One more time, run the assets:precompile rake task:

$ RAILS_ENV=production RAILS_SERVE_STATIC_FILES=1 RAILS_LOG_TO_STDOUT=1 bin/rake assets:precompile

The task should fail this time and produce the following error message:

Sprockets is complaining that it can’t find the file /private/tmp/repro/codetriage/app/assets/javascripts/application.js.erb.

This actually makes sense because in the last step we changed codetriage to codetriage-after as our project folder name, yet it is looking in codetriage.

(Note that the /private/tmp/repro part of the path may be different for you based on where you cloned the codetriage repository.)

Finding the Root Cause of the Bug

Now that we have reproduced the bug in the video, the next step is to jump into the code of the Sprockets dependency at one of the lines in the stack trace in a method called fetch_asset_from_dependency_cache. Your application depends on reading code in the libraries, which is required when debugging, especially once you have ruled out any issues with the code you’ve written.

Read gem code with bundle open

Ruby’s de-facto gem manager Bundler contains a helpful command called bundle open that opens the source code of a gem in your favorite editor. Run it like this:

$ bundle open sprockets

As long as you have a $EDITOR or $BUNDLER_EDITOR environment variable set, your preferred code editor will open to the project directory of the specified gem.

Now you can browse the gem source code and even modify it, adding print statements to see the value of variables or trying out various fixes to see if they work.

How Sprockets Caches Files

The error message above implied that the wrong value is being stored in the Sprockets cache, so the next step is to look at the cache to confirm. The cache is stored on disk across many files, so first we need to find the specific file that contains the record we want to inspect. The key to that record is a digest of the Sprockets cache ID. That’s the value we’ll try to find in the files.

Once you have the Sprockets code open, navigate to lib/sprockets/loader.rb, where you’ll find the method fetch_asset_from_dependency_cache toward the end. The documentation for this method provides insight into how Sprockets uses the idea of pipelines, histories, and dependencies to aid in caching. To get more of the backstory, I recommend watching the video starting from about the six-minute mark.

We examined the on-disk contents of the Sprockets cache, looking for the ID cache key of a specific object in the Sprockets cache.

$ grep -R 5d0abb0a8654a1f03d6b27 tmp/cache

This is a helpful debugging command to file away for later. grep -R searches through the tmp/cache directory looking for any files that contain the string “5d0abb0a8654a1f03d6b27”, which is a Sprockets cache key. The -R flag is what makes it traverse directories recursively.

In our case, the grep command does produce a cache file and we can use cat to view the contents. Inside of that cache file, we find something unexpected: an absolute path to an asset. Sprockets should only cache relative paths, not absolute paths. Since we changed the absolute path to our project directory to create this bug, it’s quite likely that this is the culprit.

Loading Up IRB

To investigate further and confirm his suspicion, we fire up IRB, the interactive Ruby debugger. If you’re new to Ruby or the IRB, wee recommend How To Use IRB to Explore Ruby as a good way to see how to use it. It’s simple but powerful and is a must-have in your Ruby debugging toolkit.

We then use IRB to inspect the file cache from Sprockets point of view.

$ irb
reirb(main):001:0> require ‘sprockets’
reirb(main):001:0> Sprockets::Environment.new.cache
reirb(main):001:0> Sprockets::Environment.new.cache.get(“5d0abb0a8654a1f03d6b27”)

Unfortunately, this does not work because the cache key is not the same as the cache ID. So, we move on to confirming his hypothesis in another way. We still include this example here to let you know that IRB is something you can use for any Ruby code and specifically with the handy Environment class in Sprockets.

Fixing to_load and to_link

To fix the bug, let’s modify the to_load and to_link methods in loader.rb to force relative paths for objects going into the cache and coming out, using the compress_from_root and expand_from_root utility methods from Sprockets base.rb. This ensures that absolute paths won’t make their way into the cache again, and consequently, that renaming the project directory won’t cause any issues in subsequent asset compilations.

if cached_asset[:metadata][:to_load] && !cached_asset[:metadata][:to_load].empty?
  cached_asset[:metadata][:to_load] = cached_asset[:metadata][:to_load].dup
  cached_asset[:metadata][:to_load].map! { |uri| compress_from_root(uri) }
end

 if cached_asset[:metadata][:to_link] && !cached_asset[:metadata][:to_link].empty?
  cached_asset[:metadata][:to_link] = cached_asset[:metadata][:to_link].dup
  cached_asset[:metadata][:to_link].map! { |uri| compress_from_root(uri) }
end

Our pull request to fix the bug contains a test to prove that the fix works. Writing tests for your bug fixes is a best practice that you should always strive to follow. It’s the best way to prevent old bugs from crawling back into your codebase.

Wrap-up

Inevitably, your code will do something that couldn’t possibly happen. That’s when you need to get out your debugging tools. We hope that you have picked up a few new ones from this post.

Your code will do something that couldn’t possibly happen in production. If your app runs on Heroku, make sure to familiarize yourself with the variety of logging solutions available as add-ons. These add-ons will make running and debugging problems on Heroku easier and they only take seconds to set up.

Rails 5.2 Active Storage: Previews, Poppler, and Solving Licensing Pitfalls

Schneems — Thu, 10 May 2018 15:58:51 +0000

Rails 5.2 was just released last month with a major new feature: Active Storage. Active Storage provides file uploads and attachments for Active Record models with a variety of backing services (like AWS S3). While libraries like Paperclip exist to do similar work, this is the first time that such a feature has been shipped with Rails. At Heroku, we consider cloud storage a best practice, so we've ensured that it works on our platform. In this post, we'll share how we prepared for the release of Rails 5.2, and how you can deploy an app today using the new Active Storage functionality.

Trust but Verify

At Heroku, trust is our number one value. When we learned that Active Storage was shipping with Rails 5.2, we began experimenting with all its features. One of the nicest conveniences of Active Storage is its ability to preview PDFs and videos. Instead of linking to assets via text, a small screenshot of the PDF or Video will be extracted from the file and rendered on the page.

The beta version of Rails 5.2 used the popular open source tools FFmpeg and MuPDF to generate video and PDF previews. We vetted these new binary dependencies through both our security and legal departments, where we found that MuPDF licensed under AGPL and requires a commercial license for some use. Had we simply added MuPDF to Rails 5.2+ applications by default, many of our customers would have been unaware that they needed to purchase MuPDF to use it commercially.

The limiting AGPL license was brought to public attention in September 2017. To prepare for the 5.2 release, our engineer Terence Lee worked to update Active Storage so that this PDF preview feature could also use an open-source backend without a commercial license. We opened a PR to Rails introducing the ability to use poppler PDF as an alternative to MuPDF in February of 2018. The PR was merged roughly a month later, and now any Rails 5.2 user - on or off Heroku - can render PDF previews without having to purchase a commercial license.

Active Storage on Heroku Example App

If you've already got an app that implements Active Storage you can jump over to our DevCenter documentation on Active Storage.

Alternatively, you can use our example app. Here is a Rails 5.2 app that is a digital bulletin board allowing people to post videos, pdfs, and images. You can view the source on GitHub or deploy the app with the Heroku button:

Note: This example app requires a paid S3 add-on.

Here's a video example of what the app does.

When you open the home page, select an appropriate asset, and then submit the form. In the video, the mp4 file is uploaded to S3 and then a preview is generated on the fly by Rails with the help of ffmpeg. Pretty neat.

Active Storage on Heroku

If you deployed the example app using the button, it's already configured to work on Heroku via the app.json, however if you've got your own app that you would like to deploy, how do you set it up so it works on Heroku?

Following the DevCenter documentation for Active Storage, you will need a file storage service that all your dynos can talk to. The example uses a Heroku add-on for S3 called Bucketeer, though you can also use existing S3 credentials.

To get started, add the AWS gem for S3 to the Gemfile, and if you’re modifying images as well add Mini Magick:

gem "aws-sdk-s3", require: false
gem 'mini_magick', '~> 4.8'

Don't forget to $ bundle install after updating your Gemfile.

Next up, add an amazon option in your config/storage.yml file to point to the S3 config, we are using config set by Bucketeer in this example:

amazon:
  service: S3
  access_key_id: <%= ENV['BUCKETEER_AWS_ACCESS_KEY_ID'] %>
  secret_access_key: <%= ENV['BUCKETEER_AWS_SECRET_ACCESS_KEY'] %>
  region: <%= ENV['BUCKETEER_AWS_REGION'] %>
  bucket: <%= ENV['BUCKETEER_BUCKET_NAME'] %>

Then make sure that your app is set to use the :amazon config store in production:

config.active_storage.service = :amazon

If you forget this step, the default store is to use :local which saves files to disk. This is not a scalable way to handle uploaded files in production. If you accidentally deploy this to Heroku, it will appear that the files were uploaded at first, but then they will disappear on random requests if you're running more than one dyno. The files will go away altogether when the dynos are restarted. You can get more information about ephemeral disk of Heroku in the DevCenter.

Finally, the last thing you'll need to get this to work in production is to install a custom buildpack that installs the binary dependencies ffmpeg and poppler which are used to generate the asset previews:

$ heroku buildpacks:add -i 1 https://github.com/heroku/heroku-buildpack-activestorage-preview

Once you’re done you can deploy to Heroku!

Adding Active Storage to an Existing App

If your app doesn't already have Active Storage, you can add it. First, you'll need to enable Active Storage blob storage by running:

$ bin/rails active_storage:install

This will add a migration that lets Rails track the uploaded files.

Next, you'll need a model to "attach" files onto. You can use an existing model, or create a new model. In the example app a mostly empty bulletin model is used:

$ bin/rails generate scaffold bulletin

Next, run the migrations on the application:

$ bin/rails db:migrate

After the database is migrated, update the model to let Rails know that you intend to be able to attach files to it:

class Bulletin < ApplicationRecord
  has_one_attached :attachment
end

Once that's done, we will need three more pieces: a form for uploading attachments, a controller to save attachments, and then a view for rendering the attachments.

If you have an existing form you can add an attachment field via the file_field view helper like this:

<%= form.file_field :attachment %>

You can see an example of a form with an attachment in the example app. Once you have a form, you will need to save the attachment.

In this example app, the home page contains the form and the view. In the bulletin controller the attachment is saved and then the user is redirected back to the main bulletin list:

def create
  @bulletin = Bulletin.new()
  @bulletin.attachment.attach(params[:bulletin][:attachment])
  @bulletin.save!

  redirect_back(fallback_location: root_path)
end

Finally, in the welcome view we iterate through each of the bulletin items and, depending on the type of attachment we have, render it differently.

In Active Storage the previewable? method will return true for PDFs and Videos provided the system has the right binaries installed. The variable? method will return true for images if mini_magick is installed. If neither of these things is true then, the attachment is likely a file that is best viewed after being downloaded. Here's how we can represent that logic:

<ul class="no-bullet">
  <% @bulletin_list.each do |bulletin| %>
    <li>
      <% if bulletin.attachment.previewable? %>
        <%= link_to(image_tag(bulletin.attachment.preview(resize: "200x200>")), rails_blob_path(bulletin.attachment, disposition: "attachment"))
        %>
      <% elsif bulletin.attachment.variable? %>
        <%= link_to(image_tag(bulletin.attachment.variant(resize: "200x200")), rails_blob_path(bulletin.attachment, disposition: "attachment"))%>
      <% else %>
        <%= link_to "Download file", rails_blob_path(bulletin.attachment, disposition: "attachment") %>
      <% end %>
    </li>
  <% end %>
</ul>

Once you've got all these pieces in your app, and configured Active Storage to work in production, your users can enjoy uploading and downloading files with ease.

The Programmer's Guide to Pairing on Pregnancy

Schneems — Wed, 07 Jun 2017 16:07:19 +0000

title: "The Programmer's Guide to Pairing on Pregnancy"
layout: post
published: true
date: 2017-06-07
permalink: /2017/06/07/the-programmers-guide-to-pairing-on-pregnancy/
categories:

- ruby

You don't have to be physically carrying a child to be involved in a pregnancy. If you pair program, you know that you don't have to have your hands physically on the keyboard to contribute to the experience. I'm currently on track for my second little one and wanted to give a shout out to some things I've seen that partners of all genders have done to help with pregnancies. While I cannot physically carry my child to term, that doesn't mean pregnancy is a passive event for me. Let's get started.

Note: I'm using the word partner to refer to anyone related to the person carrying the child.

Go to Appointments

Difficulty: Easy
Required: Extremely

It may seem like a small thing, but go with your partner to pregnancy checkups. Even if you've got that big meeting the next day, it's something both of you should be there for. Besides being supportive and holding their hand while bloodwork is drawn, there are a few excellent reasons to go together.

While everyone plans for the best, one out of five pregnancies ends in a miscarriage. If you find out something is wrong during a visit, your partner shouldn't have to be there alone.

Even when things are going well, there are so many details the doctor may talk through, that it can be helpful to have a second set of ears. Heads up: there will be handouts, and you may consider taking notes (if that's your thing). There were more than a few times my wife and I had slightly different takeaways, and after comparing notes, we were forced to clarify to make sure we had the right information.

Another way to think of this: if your partner cannot skip something, neither should you. You'll never be able to contribute equally to the carrying and delivering of the child, so the least you can do is be there through every step of the way.

Birth Plan

Difficulty: Easy
Required: Major brownie points if you bring this up

Want to delight your partner when they're not pregnant: surprise them with flowers for no reason. During pregnancy, you can be the "best partner ever"™ by proactively considering things you'll need to do and driving them forwards instead of waiting for your baby-momma to take charge of everything.

One of the first things you'll need is a birth plan. At a high level, this consists of where the two of you want to give birth, where you want to have your checkups, and how you want to give birth. There are more details for an actual "birth plan", but at the beginning these are what you'll need. I bring this up early because the answers to the three questions aren't independent. If you find an OB you like and start doing your checkups there, and the find they will not support your method of delivery, you'll have to switch doctors, and that will be a pain.

Tell your partner you want to talk about a birth plan. You'll need to do some research on available options. At a very high-level, there are birthing centers, hospitals or home births. We gave birth at a birthing center that was attached to a hospital, so there are also spectrums. You can take tours of hospitals and birthing centers. I highly recommend doing this. You can look up times and book the tours. Afterward, talk about what you like and what you don't.

I didn't do this with my first kid, and I wish I had. It kinda sucks that not only did I make my wife carry a child and deliver it, but she had to do all the paper-work of booking tours and bringing up all the various "talks" that we had to have early on.

Read a Pregnancy Book

Difficulty: Medium
Required: In some form, yes

Not assuming all partners are men, but don't get a "dude" book. Not just for pregnancy but in life, if something has "dude" in the title in a non-ironic way. At best it's light on information and heavy on assumptions. At worst it might be misogynisticly awful. At the end of the day reading anything is better than nothing, but quality varies wildly.

The first time around I chose a book that was awful. It had a bullet point list of things that will happen at each phase of pregnancy and while it was written for men, it had little advice on how to help my wife. It was mostly "don't watch too much football".

There are two categories of baby books: "what to expect," and "after the birth event." Many books try to cram in both subjects. I think as someone who has never experienced pregnancy first hand, you want a good book focused on pregnancy. I recommend "From the Hips". Often books tend to lean one heavily towards one philosophy; this book presented many along with science to back them up and different quotes from real pregnant women to give you a very well rounded view.

If you don't like a book, get another one. If you're not retaining information, it's no good.

Birth Classes

Difficulty: Easy
Required: Absolutely

There is as much variety in birthing classes as there are in ways to give birth. While your center may offer a one day "birthing" class, I recommend finding a way to supplement the information. The amount of information and topics is overwhelming. The more spread-out, the better you'll retain it. Do some research, find a few classes around town and ask your partner which ones they would like to check out.

We used a doula (which is another thing you should maybe check out) and as part of the service with the doula collective, Austin Born, there were classes. Some of them were free as a way to meet the doulas, others were a part of our doula experience. We even signed up for an extra one day intensive "what to know after pregnancy" class. Topics varied: for example, a class on breastfeeding, or the different types of medical interventions that might come up while giving birth, and what kinds of rights we had as well as good questions to ask to make decisions.

Whether you hire a doula or not, I do recommend getting some form of birth coach training. Labor lasts a long time, especially for the first pregnancy. If you can support your partner, even a little bit it can help the overall process tremendously. You'll learn things like different labor positions, phrases you can say to coach your partner to breath. Things like holding a partner's hands or rubbing their shoulders can have a big positive effect. Be prepared with food and drinks that are easy to eat in small bites. It's also your job to time contractions and record them (there are apps for this).

Protip: Depending on the place where you're giving birth, some won't let women eat once they are admitted (ask at your location). Even if they can, they might not be in the mood when they are far along, but it's important to consume calories, so they keep up energy. You can buy "mini" ice cube trays from Amazon and fill them with a sugary liquid your partner likes. Keep them in a Ziploc bag in the freezer until it's ready to go to be admitted to your birth place. We made raspberry leaf tea with honey, let it cool and made ice cubes out of those. In the last few hours my wife described them as "a lifesaver". On this note, don't forget to eat and drink yourself. The longer labor goes, the more your partner needs you.

Practice Being your Best Self: Brush and Floss

Difficulty: Hard
Required: Recommended, not required

Close your eyes and imagine what the "ideal" version of you would be. What activity do you know you should be doing that you aren't? While pregnancy may be stressful to you and your partner, having to wake up 8 times a night will push you past mental and emotional limits you didn't know you had. Not to scare you, humans have coped with infants for many generations, and most of our ancestors turned around and signed up to do it multiple times. I'm going back for seconds, but don't kid yourself that it will be easier after the baby comes out. You'll be more successful if you build a plan and start sticking to it now. Don't wait until it's too late.

When you lose sleep, you lose patience, and willpower. Once you find out that your partner is pregnant, you have 9 months to cement all the best habits you wish you had. If it takes willpower to do it before you give birth, it will be virtually impossible after.

One item for me was brushing my teeth. I always hated it. Because of this I intentionally took this time to develop it into a habit. Now I sometimes find myself laying in bed with no memory of brushing my teeth and only a minty fresh breath to prove that I did.

Maybe you already brush your teeth, maybe there's something else you wish you could do. Nick Means gave a great talk about building habits and not needing to rely on willpower at RailsConf.

Wean Coffee and Alcohol

Difficulty: Intermediate
Required: Suggested, not required

One thing I wish I had done in the first pregnancy: limit coffee and alcohol. I love coffee, I love the taste, I love the feeling when the caffeine hits your blood stream early in the morning.

I also love a good craft beer, or a scotch, recently even Texas bourbon. I still love these things. The problem is that when we have a bad day, or when things go south, we are primed to say "geez I need a drink." After the baby comes, you'll not be able to leave the house as much, and it's tempting to turn parenthood into a drinking game in the evening. Baby is crying: have a drink. Baby spit up on itself: have a drink.

The problem is that your body is already being pushed to its limits by a lack of sleep. Want to know what's worse than having to get up in the middle of the night to quiet a screaming infant? Trying to do that with a hangover. Even if you didn't drink that much, you might find your predisposition for brain splitting headaches after a drink has substantially increased. You'll also get even worse sleep than without the booze. But the worse part is that it's a cycle of dependence, the next day you feel even WORSE, so what do you do when you had "a bad day" at the end of the day? Reach for a pint of course. Don't go down that path.

For this pregnancy I'm trying to wean myself off of booze. I'm not going entirely off of it. My wife suggested that I don't drink at home, so that I don't associate everyday tasks, like unwinding and reading a book with also having a beer. I still have a few drinks when I go out. I'm hoping this will make it less tempting to drink when I'm stressed. It's only been a few weeks and I've already lost weight and I feel better, so far so good.

I weaned off coffee because I have a hard time napping. Through the first pregnancy I maybe took 3-4 naps total. It would have been much easier if I could take the standard advice "sleep when your baby sleeps."

If you don't have this napping problem, don't worry about it.

If you want a decaf that is 90% as good as regular, I recommend Third Coast Coffee Roasters. I know it still has some caffeine and also that the process of removing it isn't the best in the world. You do what's right for you and your life. I've lost my regular coffee habbit, you can pry my decaf from my hot, freshly brewed hands.

Side note: drink lots of water. I recommend a Camelbak Podium because you can drink it in bed without having to get up or without spilling. After the birth, you'll be waking up a lot more, and every little thing to make your life a little easier is worth it. Also, get your partner one as they'll be waking up to go to the bathroom well before the big event comes.

Morning Sickness & Cravings

Difficulty: Easy
Required: Absolutely

The biggest misnomer in pregnancy is "morning sickness," because, surprise, it happens all day long. It's usually portrayed as a minor inconvenience; a woman throws up casually as a way to indicate to the audience she is pregnant (squeeeee!!). In reality, they may never throw-up, they may have overwhelming nausea for days at a time, or anywhere in between. While it's different for each woman, my wife once described it as worse than labor when taken as a whole. It's no joke, so you need to be ready to help any way you can.

One way to help is to observe the things that they are sensitive to. Usually women have smell sensitivity. My wife could not stand toothpaste, so we found a salt based toothpaste that was unflavored. With kid number two she is set off by changing diapers. That means all number-twos from kid number-one are on me to change. It's a small thing to do for the woman carrying your child.

Pop culture represents pregnancy as somewhat of a running joke on partners. They're seen running to the store at midnight while a woman in full makeup sheepishly asks (or demands) for pickles and ice-cream. They lug themselves to the store and back, only to find the items are no longer wanted or their partner is peacefully asleep by the time they get back. They slouch into an oversize chair, look a the pickles with a sense of defeat and the studio audience laughs.

The real joke, of course, is expecting Hollywood to be remotely similar to real life. Cravings are different for every woman. Usually, when my wife asked me for some very specific food it was because she was feeling nauseous. We've found that coconut water helps her, so I try to keep it around or ask at virtually every restaurant if they carry it regardless if it's needed yet. Other times, it was just a matter of being flexible. Being willing to get her food while we are both in bed, or making an unexpected stop at a fast food restaurant for something fried.

Gear

Difficulty: Easy
Required: Yes

You'll need a minimum amount of baby gear coming home from the hospital. A car seat, a baby hat, and some clothes. Once you get home you'll also want some swaddles, plenty of fresh diapers (maybe consider cloth if you're the environmental type) and a fair share of onesies. You will also need somewhere for the baby to sleep. There's other things like mittens (so they don't scratch themselves, babies are weird) and bigger items like strollers or swings. You can do research on these things without waiting for your partner. Of course you should talk with them before either of you buy big items, but having some knowledge of the available options will make these conversations much smoother.

Try

Difficulty: Easy
Required: Are you freaking kidding me, of course

The most important part of being a good partner is being there for your baby-momma. You might not get everything right, and there's almost always room for improvement, but the least you can do is make a conscious effort to make the process easy on her and the two of you. While I wrote a lot here, and things can get overwhelming remember that 9 months is a long time. It means that you can take things step by step without having to rush it all. It also means that you have no excuse for skipping out on classes, working towards building better habits, or learning a thing or two about pregnancy. Above all else pay attention to your partner. From my favorite baby delivery joke: it took two of you to get that baby in there, it will take at least two to get out.

If you liked this post join my free mailing list for more. All links to products on Amazon are affiliate links.

Writers Write

Schneems — Tue, 30 May 2017 15:22:45 +0000

I've been writing more recently. One of the biggest reasons is that I've been writing more recently. Writing begets writing; the more I do it, the easier it is to do it more. I've found diet to be similar. When I'm eating fresh fruits and veggies, it's what my body craves. But as soon as I "treat" myself with a bag of chips or a fatty big honking slice of greasy pizza, guess what my body wants? More of the same.

I've written off and on for quite some time. Someone recently pointed out how you could tell how long someone had been in their job by when their last blog post was written. While coders tend to be more comfortable with a text format than many other professions, we're not the best at writing consistently.

I wrote another post about how coders code and thought it was fitting to mention how I've made a plan to publish a blog post once a week. I've been keeping up the streak for awhile now and feel pretty good about it. One trick I'm using is that I've got a few posts written ahead of schedule, so it doesn't feel like I'm down to the wire on deadlines. I'm on a plane right now without WiFi and feeling particularly inspired so this is actually the third post I've written today.

In the past I've traditionally written technically focused articles. I still do that and will keep publishing them. The interesting thing, about forcing myself to sit down and write no matter what, is that I can't wait months for that huge open source pull request I'm working on to drop before I blog about it. I've got to talk about what's on my mind right now, ready or not. Sometimes it ends up being awful and when that happens, since I have some buffer room in my schedule, I can throw those away. Sometimes it ends up taking me places I wouldn't have gone before. When you go through life with an eye out looking for a story, you'll be amazed at how many you find.

I tend to preach when I write. I like to have a message. I like to try to get people motivated or active. I do the same when I give talks. Instead of "look at what I did" I usually also add a "and here's how you can do it too" flare. It's a bit of a crutch at this point as sometimes things can be interesting without having to be instructive. While I don't necessarily recommend blogging as it is tedious and time consuming, I do recommend you write.

As you write you'll find an audience of one. Read your own writing. Take notes. You'll be surprised at what "past you" can teach "future you". Keep writing and maybe you'll find a larger audience. Write wherever possible: Write commit messages, write GitHub comments, write design docs, write READMEs, and method docs. Write. The more you do it, the easier it will be.

Writing is an act of communication. Programmers are really in the communication business. We communicate with machines in an arcane language to tell them how to do our business. We must share our requirements and our efforts with others, co-workers, bosses, co-founders, and customers.

While we often think of writing as a one-to-many medium (books, magazines, blog posts) our most important writings can be the ones that are day-to-day. While your primary job title might not be "writer", writing is a tier-one method of communication.

If you write, I can't guarantee fame or fortune. I can't promise you an audience. However, I will give you my word that writing will bring more writing. Communication will bring more communication. Clarity will bring more clarity.

Write.

If you liked this post join my free mailing list for more.

Coders Code

Schneems — Fri, 26 May 2017 15:15:57 +0000

As truisms go, one of my favorites is "writers write". Many developers walk around pondering whether they are "real coders", or they ask "how can I be more senior". To them, I say "coders code". If someone is writing, then by definition they are a writer. It doesn't matter if they are J.K. Rowling or working a blog post. The act of writing creates a writer. The same is true of coding. If you're in QA or DevOps or Front End or Backend or spend your days hunting down missing semicolons in code-reviews, you're a coder. When you put your fingers to the keyboard at your editor of choice, even if it's not Emacs or Vim or *<latest hot editor here>*, you're still a coder. If your fingers never grace a keyboard and you drive a pair session or dictate text, you're a coder.

What about when you're not in the act of generating code? Do you stop being a coder? I don't know what Stephen King is doing right now, maybe he's brushing his teeth, fixing some food, or going to the bathroom. Because he doesn't have a pen in hand or a typewriter under a finger, does it mean he's no longer a writer? Of course not. The phrase is "writers write" not "writers are only writers when they are actively writing". The same applies to coders - I've heard again and again from just about every programmer under the sun that the longer they spend coding, the less time they physically spend coding. Maybe they pick up a mentorship role or they work on documentation and code reviews. For people who are on a life-long technical track, they tend to be attracted to hard problems, and many are best solved by contemplation and careful thought instead of hours of guess and check at the keyboard. I've switched my showers from first thing in the morning to after I've worked a bit for the day (I work from home). I find when I'm stuck and my wheels are spinning, taking a step back to really think about the issue at hand while in a shower works miracles.

For a deeper dive on why exactly this trick might work check out the book Thinking Fast and Slow. Also, did you know you can check out Kindle books from your local library?

How does a writer get better at writing? They write. Ideally, they have some kind of a feedback loop, maybe a mentor, maybe they go to school and are given assignments, or they join some kind of writing retreat or club. At the end of the day though, the important part isn't the school or the criteria, it's the act of writing. If you're just getting started or want to make it to the next career milestone, might I recommend coding? Coding is the act, and the journey, and the goal. However, you can't just sit down and pound out code.

A book needs a plot. Code needs a problem to solve. Before we can have a problem, we need to have a desire, a want. A problem is a thing that gets in the way of having our desires met. For this, I recommend finding a thing you want to build. As you work towards that goal, the act of coding will come naturally. Forget "what is the best language to learn" or "how do you get better at programming". Find a topic or subject that interests you and an idea that keeps you awake. When you find this, the code will come.

If you don't have a passion project burning a hole in your brain you can cheat to find a muse. Some people are incentivized by a university or a Bootcamp. Maybe prowling a tag on Stack Overflow or signing up for a mailing list might give you inspiration. I run a free service called CodeTriage that helps coders find a muse for Open Source contributions, which is another treasure trove of code just waiting to be written.

If you've been coding for a while and have no lack of opportunities, what then? Code for the job you want. Perhaps you want to specialize in writing Performant code, or at very least get better at it. Performant coders write Performant code. Find other coders who have written fast code, find out the why and the how. Find slow code (shouldn't be too hard) and some benchmarking tools to isolate the slow parts. Then try making it faster. Read performance blog posts and books. Ask questions, go down rabbit holes. When you get stuck, ask yourself what's stopping you from writing code. Remove those blockers, then get back to writing code.

Now ask yourself, are you a coder? If you answered in anything but the affirmative, what's stopping you from claiming the title? You write code, right? That makes you a coder!

What comes next? Get excited, get engaged, get coding!

If you liked this post join my free mailing list for more.

Who Called Git? An Unusual Debugging Story

Schneems — Mon, 28 Nov 2016 00:00:00 +0000