DEV Community: Adam Gordon Bell

What's gotten better in programming languages

Adam Gordon Bell — Tue, 14 Jun 2022 14:08:36 +0000

I have a question, especially for those who’ve been programming for some time or have experience with older languages:

What improvements have you seen in programming languages that aren’t related to syntax or semantics of the language?

I’m thinking about things like the introduction of package managers for pulling in third party dependencies, and the existence of code formatters or linters.

I guess integrated testing into the tooling / language in rust and golang would count as well.

( Another way to frame this would be what do you miss when you are programming in c or c++ or some older language that isn’t the language, but more about the tooling and ecosystem.)

Full Backend Apps on AWS Lambda?

Adam Gordon Bell — Thu, 19 May 2022 12:45:46 +0000

Is anyone hosting a full CRUD type app with routing and persisting as an AWS Lambda?

And if so, how is that going?

Since you can run a container as a lambda, and you get built in scaling up and down it seems like it might be a good fit.

Do you use GRPC?

Adam Gordon Bell — Mon, 31 Jan 2022 22:11:51 +0000

I'm writing an article about GRPC and protobufs. Here is the thing I'd like to know:

What tools to you use for making GRPC requests?

Like if you just want to test that a service is up and taking requests? I'd like a command line one that is simple to use and install and wondering what people use.

Web3 will not replace Contract Law?

Adam Gordon Bell — Fri, 28 Jan 2022 22:11:41 +0000

The law is code that is interpreted by Judges. My wife is a lawyer and tells me this. In this view, amendments or reinterpretations of laws by higher courts are diffs.

I'm not really up on Web3, but this take is wrong somehow.
// Detect dark theme var iframe = document.getElementById('tweet-1486698174869909506-976'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1486698174869909506&theme=dark" }

And not just in the funny way that many languages run in an interpreter:
// Detect dark theme var iframe = document.getElementById('tweet-1486700389822910469-326'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1486700389822910469&theme=dark" }

But in the fact that laws being interpreted by a person is the point of the legal system. It's why it works.

The fact that a judge gets to consider the specifics of your particular case and make a decision that takes those into account makes the legal system work.

In cases where the law is interpreted absolutely, like three-strikes-you're-out laws or mandatory minimums, sentences are usually the places where things turn out badly.

Explicit coded rules are bad at handling exceptional cases, and the real world is messy. People can enforce the spirit of the law and not just the letter of the law.

On the other hand, the existing legal system is slow and expensive ( but almost no contracts are actually resolved in court, it's usually only the messy cases with the exceptions that end up in court).

So, where do coded contracts make sense (Any ethereum people around) ?

Knowing I'm Dumb Is the First Step

Adam Gordon Bell — Thu, 27 Jan 2022 22:09:13 +0000

The Dunning-Kruger Effect is my favorite.

This effect has been the darling of journalists who want to explain why dumb people don’t know they’re dumb.

It's where 'dumb' people don't know that they are 'dumb' and think they are better than more skilled people.

So maybe it's false?

// Detect dark theme var iframe = document.getElementById('tweet-1485319874738114564-592'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1485319874738114564&theme=dark" }

But I've found it helpful as a model of learning. Learning where you are ignorant is an essential step in understanding.

When you don't know anything about software development, and you want to learn, it seems like it will either be easy or impossibly hard. It's just a giant skill to acquire.

But as you learn, you start to recognize all the different gradations of learning. So this one big thing is actually a bunch of skills, and there are levels of each of them.

And the more you learn about, say, backend development, the more you realize there are other areas you don't understand, like not just frontend, but networking programming, phone development, OS development, and writing firmware. Every little area is fractal with so much knowledge in it.

Part of learning is just learning how much you don't know, so it makes sense that "dumb people don't know how dumb they are" because knowing how dumb you are is something you have to learn.

What do you think?

Incident Management Metrics and Key Performance Indicators

Adam Gordon Bell — Thu, 27 Jan 2022 20:53:33 +0000

In 2008, I got my first job at a software-as-a-service company. We built learning management software and ran it on servers in the small data center connected to our office.

We released new software onto these production servers monthly and measured quality by counting bugs per release. We also had account managers who kept us informed of how many large clients seemed upset about the last release.

Occasionally, when something went wrong, we would do a stability release and spend a month only fixing bugs. Testing was not a part of our build process but a part of our team: every feature team had quality assurance people who tested each feature before it was released.

This wasn't that long ago, but cloud software development has matured a lot since this time. Incident management has become standard practice, and many great metrics and Key Performance Indicators (KPIs) exist for measuring release quality. Let's review some of them.

Mean Time Between Failures (MTBF)

When software is being released only once a month, on a fixed timeline, with extensive manual testing, counting the number of bugs might work. But once you start releasing many times per week or per day, this won't work, and another way to measure software quality is required.

Mean time between failures is a metric from the field of reliability engineering. Calculating it is simple: it is time over the number of failures that occurred during that time. If in the last 30 days you have had two production incidents, then the mean time between failure is 15 days.

Calculating MTBF

Incidents in last 30 days
#1	Jan 3rd
#2	Jan 25

Mean Time Between Failures =

: 30 days / 2 Incidents = 15 days

Mean Time to Recovery (MTTR)

Something funny happens when you start releasing more frequently. You may end up with a higher count of issues in production, but resolving them will happen much faster. If each change is released separately using a continuous delivery model, then recovering gets easier -- often, all that is required is hitting a rollback button.

If you are measuring MTBF, your software may be getting much better, but your numbers will be getting worse. Enter mean time to recovery. Mean time to recovery is just what it sounds like: you start a timer when the incident begins and stop it when production is healthy again - even a simple rollback counts. Average this number across incidents, and you have MTTR. You now have a metric that captures the health of your incidence response process.

Calculating Mean Time to Recovery

Incident #1
Reported	10:00 am.
Recovered	12:00 pm.
Recovery Time	2 Hours

Incident #2
Reported	10:00 am.
Recovered	2 days later at 10:00 am.
Recovery Time	48 Hours

Mean Time To Recovery =

: 2 hour + 48 hours / 2 failures = 25 hours

Mean Time to Resolve (MTTRe)

ℹ️ Acronyms Collision Alert

Mean Time To Resolve, MTTRe, differs from Mean Time To Recover, MTTR, but some resources use MTTR for both. To avoid confusion, ensure you are using the correct terminology for your metric.

Rolling back to address an incident is a great idea: it's often the quickest way to get things back in a good place. But there are other types of incidents. Imagine your application deadlocks every once in a while, and you have to restart it to unlock. You may have an excellent mean time to recovery, but you've never actually addressed the root cause. This is what MTTRe measures, not the time to get the service back up and running but to resolve the root cause and ensure the problem never happens again.

The never-happens-again part is hard to achieve but vital. If you are responding quickly but never getting to the root cause, you will be living in a stressful world of constant fire fighting. However, if you are resolving the root cause of each incident, then quality will increase over time.

Calculating Mean Time to Resolve

Incident #3
Reported	day 1
Addressed	day 1
Root Cause Analysis	day 2
Root Cause Addressed	day 31
Resolve Time	30 days

Incident #4
Reported	day 1
Addressed	day 1
Root Cause Analysis	day 2
Root Cause Addressed	day 11
Resolve Time	10 days

Mean Time To Resolve =

: 30 days + 10 days / 2 incidents = 20 days

Mean Time to Acknowledge (MTTA)

An essential part of good incident management is an on-call rotation. You need someone around to respond to incidents when they occur. Our previous metrics would be unable to differentiate between an incident that took 3 hours to recover from and one that was recoverable in 5 minutes but took two hours and 55 minutes to be acknowledged.

MTTA highlights this difference. It is a metric for measuring the responsiveness of the on-call person to any alerts.

️Calculating Mean Time to Acknowledge

Incident #5
Reported	10 am
Acknowledged	10: 05 am
Recovered	12:00 pm
Acknowledge Time	5 minutes

Incident #6
Reported	10 am
Acknowledged	11: 55 am
Recovered	12:00 pm
Acknowledge Time	115 minutes

Mean Time To Acknowledge =

: 5 minutes + 115 minutes / 2 incidents = 60 minutes

Summary

There are many ways to measure the quality of your software as a service product. MTBF, MTTR, MTTRe, and MTTA can each offer a different lens for viewing your software release life cycle. As you improve your Software Development Life Cycle, find ways to collect aggregate metrics like these and choose one or two to target for improvement.

Invest in improving these metrics, and you'll make up for it in time saved fighting fires. Also, focusing on aggregate metrics can be an effective way to move the discussion from blame about specific incidents to a higher-level debate around changing the process to better support the company's goals.

If your build pipeline is taking more than 15 minutes and therefore negatively affecting your metrics, then take a look at Earthly's free and open build tool.

Originally published on Earthly's blog

Property-Based Testing In Go

Adam Gordon Bell — Wed, 12 Jan 2022 16:31:08 +0000

Have you ever wanted your unit tests written for you? Property based testing is a powerful testing technique that, in a sense, is just that. You describe the properties you’d like to test, and the specific cases are generated for you.

Property-based testing can be a bit trickier to learn, and not every problem can be well tested in this manner, but it’s a powerful technique that’s well supported by the go std-lib (testing/quick) and that is under-utilized.

Testing `csvquote`

csvquote is a small program that makes it easier to deal with CSV files at the command line. It does this by replacing problematic CSV characters with the controls characters \x1e and \x1f and later removing them.

You can use it like this:

csvquote | head | csvquote -u

I want to improve it a bit, but first, I want to add some testing to ensure I don’t break it. The tests will also help me document how it works.

Here are my initial test cases:

var tests = []struct {
 in string
 out string
}{
 {`"a","b"`, `"a","b"`}, // Simple
 {"\"a,\",\"b\"", "\"a\x1f\",\"b\""}, //Comma
 {"\"a\n\",\"b\"", "\"a\x1e\",\"b\""}, //New Line
}

To test that the in string always results in the out, I do the following:

func TestSubstitute(t *testing.T) {
 f := substituteNonprintingChars(',', '"', '\n')
 for _, tt := range tests {
  out := string([]byte(apply([]byte(tt.in), f)))
  assert.Equal(t, tt.out, out, "input and output should match")
 }
}

substituteNonprintingChars returns a function that does the conversion and apply is a helper for applying that function over a []byte.

Slightly simplified, it looks like this:

func apply(data []byte, f mapper) []byte {
 count := len(data)

 for i := 0; i < count; i++ {
  data[i] = f(data[i])
 }
 return data
}

The second thing I want to test is that I can reverse this operation. That is, when csvquote is run with -u, it should always return the original input.

I can test this by going in reverse from output to input:

func TestRestore(t *testing.T) {
 f := restoreOriginalChars(',', '\n')
 for _, tt := range tests {
  in := string([]byte(apply([]byte(tt.out), f)))
  assert.Equal(t, tt.in, in, "input and output should match")
 }
}

restoreOriginalChars is the function that restores the string to its original form.

Both these tests pass (the full source is on GitHub). However, it’s still possible there are edge cases that work incorrectly. What I need is a way to generate more test cases.

Property-Based Testing

One way I could improve my testing is to combine the two tests. That is, rather than testing that the substituted value matches expectations and that the restored value matches expectations, I can simply test that substituting and restoring a string always results in the original string.

That is for almost all possible values of the string a:

echo a | csvquote | csvquote -u == a

In my go test, I can state that property like this:

func doesIdentityHold(in string) bool {
 substitute := substituteNonprintingChars(',', '"', '\n')
 restore := restoreOriginalChars(',', '\n')
 substituted := apply([]byte(in), substitute)
 restored := string([]byte(apply(substituted, restore)))
 return in == restored
}

That is, for a string in, substituting and then restoring it should equal itself. I’m calling this doesIdentityHold because an identity is any function that returns the original result. csvquote | csvquote -u should be an identity.

Quick Check

Now that we have that function, we can use testing/quick, the property-based testing library, to test it:

func TestIdentity1(t *testing.T) {
 if err := quick.Check(doesIdentityHold, nil); err != nil {
  t.Error(err)
 }
}

By default, this will cause testing/quick to run 100 iterations of my identity test, using random strings as inputs. You can bump that up much higher, like so:

func TestIdentity1(t *testing.T) {
 c := quick.Config{MaxCount: 1000000} // <- changed
  if err := quick.Check(doesIdentityHold, &c); err != nil {
  t.Error(err)
 }
}

And with that number of tests cases, I run into a problem:

--- FAIL: TestIdentity1 (5.17s)
    main_test.go:43: #13072: failed on input "\x1f\U00037f8b\U000732a1\"
FAIL

The failing test is helpfully printed. However, it’s found a problem in my testing method, not with the code under test. You see, csvquote should always be able to reverse it’s input, except in the case that the input contains the non-printable ASCII control characters and that is what this test case has found.

\x1f may not be used much outside of traditional teletype terminals but I asked csvquote to check across all possible strings, and eventually, it found a failing case.

Long term, I should adjust csvquote to exit with an error condition when it receives input it can’t handle, but that is a problem for another day. Instead let’s focus on constraining the input used in the tests.

Writing a Property Testing Generator

As I’ve shown above, testing/quick can generate test values for testing on its own, but it’s often valuable to write your own.

In this case, writing my own CSV file generator will allow me to make a couple of improvements to the test. First of all, I can remove control characters from the test set. But also, I can move to generating strings that are valid CSV files, rather than just random characters. Testing with random characters is easy, but my main concern is that csvquote can handle valid CSV files, so by narrowing in on that case, I’ll have a better chance of catching real-world issues.

First I write a function to generate random strings of size size and from the character set alphabet:

func RandString(r *rand.Rand, size int, alphabet string) string {
 var buffer bytes.Buffer
 for i := 0; i < size; i++ {
  index := r.Intn(len(alphabet))
  buffer.WriteString(string(alphabet[index]))
 }
 return buffer.String()
}

It outputs what I expect:

r := rand.Rand{}
println(RandString(&r, 5, "abc"))
println(RandString(&r, 5, "abc"))

caccb
cabbb

Then I used that to generate a CSV files with random number or lines and rows:

func randCSV(r *rand.Rand) string {
 var sb strings.Builder
 lines := r.Intn(20) + 1
 rows := r.Intn(20) + 1
 for i := 0; i < lines; i++ {
  for j := 0; j < rows; j++ {
   if j != 0 {
    sb.WriteString(`,`)
   }
   sb.WriteString(fmt.Sprintf(`"%s"`, randCSVString(r)))
  }
  sb.WriteString("\n")
 }
 return sb.String()
}

func randCSVString(r *rand.Rand) string {
 s := RandString(r, 20, "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz01233456789,\"")
 return strings.Replace(s, `"`, `""`, -1) // In CSV all double quotes must be doubled up
}

Here is what calling randCSVString will generate:

"7IWvdDEGTxlRdJZrc4Ra","LMHcCcN,D2RVRsfbw0IF",",LrZ3B4SvFEfG3FqO01n","""vY4FmzhbyR4iDgYpr6f","sw83uJkitIc8trzaYmEO"
"fGloXshqwC""O9Bd,eM,L","LDRajiwNgZNitCA0QL11","1Q
nqhMhzu7Mlu92qxK""8","jtj33vTlH5VP4UlbKu2e","sDhVlor4RRg4JrKfdR""W"
"EGWYmpm1TVNdB3rfmU80","Mx0RdqqVUuWiOpKY""IXk","ctprb8
7PrVu8rX8iJqTg","wnmagEjUDJrZCGBdmAYH","EJtuq1ZJqz9jf,j6vzh1"
",88Q7qWLewZVK9pE83ut","1kZJipoz2FOWLS96xMjW","5BkVvtZqZk""S2wpOB3rQ","kYN6aGPDmgSLAEI4CJtu","xOH,9y9wDDtwWUPsjgz7"
"mbexX2,Wl4""cuq3VGekP","uMXK5uLDz2ZS3Wv4wynY","KJBLt0RipsJyEqVHqrDx","glDHHs6Ujyg""piJGv595","rIa4KOxe,B""qS7EmO
nTB"

I can then hook this up to my test method:

func TestIdentity2(t *testing.T) {
 c := quick.Config{MaxCount: 10000,
  Values: func(values []reflect.Value, r *rand.Rand) {
   values[0] = reflect.ValueOf(randCSV(r))
  }}
 if err := quick.Check(doesIdentityHold, &c); err != nil {
  t.Error(err)
 }
}

And now I can test csvquote against ten thousand random CSV files in under a second.

> go test
PASS
ok github.com/adamgordonbell/csvquote/cmd/cvsquote 0.635s

I’ll leave extending this to covering Unicode values and unquoted rows for another day.

Property Testing

Property-based testing is a powerful technique for testing your code. As you can see from this real-world but somewhat contrived scenario, it moves the work of testing away from specific test cases and towards validating properties that hold for all values. Here the property under test was that substituting and restoring are inverses of each other (and therefore form an identity).

For me, the most challenging part of property-based testing is seeing how I can transform a specific test-case into a testable property. It’s not always immediately clear what’s possible to test in this manner and some tests are simply hard to state in this form. But when a problem is well suited for property-based testing, it is a powerful tool!

( The second most challenging thing about property-based testing is writing generators for your problem domain. Although once written, they make testing in this style feel quite natural. )

More Resources

The hardest part of this style of testing is seeing where and how it can apply. The places I’ve found it valuable include:

Verifying serialization and deserialization code
Verifying an optimization (by comparing the results to the un-optimized version)
Anywhere I have more than one representation of some data and need to test the boundaries of those representations.

But others have found more ways to use this paradigm. If you want to learn more about property-based testing, then gopter, the GOlang Property TestER, is worth taking a look at. Amir Saeid, who’s good at this technique, recommends this book full of examples, and this blog.

If you have any tips or suggested resources for property-based testing, please let me know on Twitter (@adamgordonbell).

And if you care about reliable software, take a look at Earthly. Earthly makes continuous integration less flakey and works with your existing workflow.

Migrating Your Open Source Builds Off Of Travis CI

Adam Gordon Bell — Fri, 07 Jan 2022 16:53:01 +0000

Starting in early 2021 and continuing to this day, a movement has been underway to migrate open-source projects off of Travis CI. So what happened, and where should you move your project to?

I bet you can't guess where I recommend?

If you're not familiar with Travis CI, it's a build company that has been powering the continuous integration (CI) of many open source projects since it launched in 2011. It was the first build solution that was free for open source use and that easily integrated into GitHub.

What Happened?

In 2019 Travis was acquired by a private equity group and many engineers were let go.

So apparently Travis CI is being strip-mined immediately after their acquisition by Idera. Sorry, I mean after "joining the Idera family" 🙄 https://t.co/CE5ERp1RsY A bunch of talented people are waking up to termination letters. Absolutely shameful. https://t.co/BbBRPdnswe
— Senior Oops Engineer (@reinh) February 21, 2019

Then, on Nov 2, 2020, Travis CI announced the end of its unlimited support for open-source projects:

For those of you who have been building on public repositories (on travis-ci.com, with no paid subscription), we will upgrade you to our trial (free) plan with a 10K credit allotment.

When your credit allotment runs out - we'd love for you to consider which of our plans will meet your needs. - Travis CI blog post

The reason behind the change is stated to be abuse by crypto-miners:

However, in recent months we have encountered significant abuse of the intention of this offering (increased activity of cryptocurrency miners, TOR nodes operators etc.).

However, many feel the real reason is that the acquirer is aiming for profitability at all costs and supporting the open-source community represents a significant cost.

My previous company was on Travis, and as soon as I saw that Travis was purchased by private equity, I knew the downward spiral had begun and I recommended we move to something else. Not surprised that this is happening a couple of years later...my understanding is that private equity will tend towards slowing/stopping development after acquisition to cut costs/headcount, and then squeeze the remaining value from what's left, so this is in-line with that playbook. - rpdillion on hacker news

Why It Matters

The open source movement runs on the heroic efforts of not enough people doing too much work. They need help. - CLIVE THOMPSON

Many open-source projects are still using Travis and open-source maintainers are notoriously overworked. Time spent migrating builds is time not spent on other things. Large well-maintained projects will likely quickly transition but for many smaller projects, an abrupt change in a service they depend on is a huge challenge.

Where to Move To

If you maintain an open-source project that uses TravisCI and are hoping to get off it, then assuming you have the time to migrate, there are actually many viable options.

Option: Run Your Own Builds

You can find some scattered instructions online for running Travis builds yourself. There are mixed reports on the stability and feasibility of this approach, but if your adventurous, you could try to set up your own Travis CI build executor on your own hardware.

A better option, if you want to run the builds on your own hardware is to look at something like Buildkite or GitLab CI.

Option: Circle CI

A better option is Circle CI , a Travis CI competitor which still offers a free plan.

Circle CI offers 400,000 build credits per month to any open-source public repository. This is their free plan and limits concurrency to 1 job at a time. They also have an easy GitHub integration and no application process.

They also allow use of the free plan with private repositories. This makes it a great choice if your project is not actually open-source.

Best Option: GitHub Actions

An even better option is GitHub Actions, a cloud CI system directly from GitHub. GitHub is at the center of many open source projects and this makes it a natural choice for CI.

GitHub Actions (GHA) is newer than either TravisCI or Circle CI, having launched in late 2018.

GHA offers very generous build credits, 20 concurrent build jobs per project and no limit on build time used. If your pipeline can be run in parallel this concurrency can really be a great enabler. The only limitation I was able to find is that the build may last no longer than 6 hours in total.

If your project is hosted on GitHub and is open source then the GHA open source plan seems like the best bet right now.

But wait, there is an even better solution: being vendor neutral. Before I explain how you become vendor neutral lets review.

Summary of Open Source Plans

Service	Open Source Offering
Travis CI	1000 minutes total with application process for more
Circle CI	1 concurrent build at a time
GitHub Actions	20 concurrent build jobs per project

Don't Let This Happen Again

So GitHub has a generous build plan, but moving your CI process is not easy or free. The more complex your build, the harder porting from one cloud CI to another is going to be. If you move to GHA and then GHA stops being a viable option in the future then this whole effort will have to be repeated.

Neutral Build Specifications

How can you minimize the effort of moving from build platform to another?

My suggestion is to keep as much logic as possible out of the proprietary build definition. Instead, define it in an open-source format that you can execute anywhere.

Makefiles and Dockerfiles

One way to build a CI neutral build definition is to use a Makefile and a dockerfile. The Makefile contains the various steps of your build pipeline and you run it inside a docker container which installs any needed dependencies. QMK is a popular open-source project that uses this approach.

FROM qmkfm/base_container

VOLUME /qmk_firmware
WORKDIR /qmk_firmware
COPY . .

CMD make all:default

QMK Docker File for executing the full build

Earthly

I am an Earthly contributor and this is the Earthly blog, but in my totally biased opinion, it deserves a mention as an neutral format for defining a build. The Elixir web framework Phoenix is a great example to take a look at.

Earthly is like a Makefile where each step is containerized and dependencies are explicitly declared.

FROM golang:1.13-alpine3.11

build:
 COPY main.go .
 RUN go build main.go
 SAVE ARTIFACT main AS LOCAL main

lint: 
 ...

Example build steps for a go application

Other Interesting Options

Easier Migration from Travis to GHA

Migrating your build out of Travis will take a little work. If you aren't interested in a neutral format, this GHA action might make it easier.

This action setups environment variables specified in the .travis.yml file and then runs one of the (potentially) many build jobs within the test build stage.

Serverless Builds

Another interesting option if you are feeling adventurous is using AWS lambda as your build executor. I have no idea how feasible this is, however, the gg project from Stanford looks interesting. It attempts to use AWS lambdas for running builds at the maximum possible parallelism.

Take-Aways

You probably need to move your open-source project's builds off of Travis CI. If you host it on GitHub, GitHub Actions is probably a good choice. There is a risk that the GHA offer will disappear as well. You can protect yourself from that by defining your build in an open format that is easy to move around. All build problems can be solved by another layer of abstraction.

If you are going that route, I think Earthly is a great option, but as I said, I am biased.

Idiots And Maniacs

Adam Gordon Bell — Wed, 05 Jan 2022 20:12:21 +0000

Observability

If you do software-as-a-service development and you have paying customers, you at some point learn about the need for operational monitoring and observability. Personally, I went through a process something like this:

Service has some basic logging and an uptime alert.
Service has a health-check endpoint, is deployed in triplicate behind a load balancer.
Logs are real-time shipped to Splunk / ELK Stack.
Metrics set up in Datadog / Prometheus with paging.
Distributed Tracing set up for debugging across services.
And so on.

Each step requires more work to set up and has some additional benefits. I moved through each step by necessity as my service handled more requests and became more important to customers.

If you had shown step-1-me what a simple REST service looks like in step 5, I would have been shocked. The metrics counters and distributed tracing spans, and various operational concerns make the service more complex. I would have thought that whoever wrote the service was obsessed with operational issues, to the detriment of solving the problems at hand. I would have thought the service author was an observability maniac.

On the other hand, if step-5-me were to get paged because a service written by step-1-me was down, he would not be happy. He would have a hard time figuring out what was wrong, and he'd be pretty sure the service author was an idiot who had never been paged in the middle of the night.

It reminds me a lot of driving when it first snows here in Peterborough.

Driving

In the winter, in Peterborough, we get snow. It's just a fact of life, and people learn how to drive in snowy conditions. But on the first substantial snowfall of the year, people struggle to remember how to drive.

Some will drive way below the speed limit and slow down traffic for everyone. They are idiots in the snow. Others get frustrated by the slow drivers and go too fast for the road conditions. Don't they know there is snow on the road?

Everyone driving slower than me is an idiot, but everyone going faster than me is a maniac.

So it is with software development. Everyone who takes an idea further than I have is a maniac, and people who haven't taken it as far as me are idiots.

Testing

There was a time when I thought all code should have 80% unit test code coverage as a minimum. Anything less was practically unethical, and if you didn't think so, then you hadn't read Clean Code™️ enough times.

On the other hand, Richard Hipp – who tests to 100% code coverage at the machine code level, covering every branch by running billions of tests each release¹ – is a testing maniac.

I hope you see where I'm going. This idiot to maniac gradient feels right, but it makes no sense². How can I be the only the person driving the right speed? Wherever you find yourself along the spectrum is more a reflection of the context in which you work than anything else.

More Examples

Here are some made-up and exaggerated examples:

Typescript Developer: JavaScript developers are idiots. Don't they know how many bugs the type system could have caught for them.
Typescript Developer: Elm developers are obsessed with types. They are maniacs about using types to catch things at compile time.
Go Developer: The JVM is such a heavyweight runtime and uses so much memory.
Go Developer: Rust is so complex! Who wants to manage memory manually. Use a GC, you maniacs!
Kotlin Developer: Java is so verbose and ugly. Welcome to 2021. Kotlin has a lot of sugar and type improvements that make writing correct code simpler.
Kotlin Developer: Scala is for maniacs. There is so much syntactic sugar and type stuff that it's not worth learning about.

I'm not trying to pick on any of these languages - Typescript, Go, and Kotlin are exceptionally well suited to their niche. But contexts vary³, and it takes mental effort to see that people making other trade-offs sometimes have good reasons for it.

That doesn't mean that other people are never wrong, though. People choose the wrong tool for the job all the time. For example, if I were doing SQLite's level of testing for a low-reliability, low-traffic, state-less web service when a couple of integration tests would do, then that'd be a mistake, but you'd have to know the context to make that call.

What do you think? What are you a maniac about?

I'm a maniac about builds. This article was originally published on earthly.dev where we make pretty cool build stuff.

Richard is the creator of SQLite. See my interview with him for a discussion of his testing approach. ↩
Article Update: Apparently I've unintentionally stolen this idiot to maniac spectrum idea from George Carlin. Thanks to tjones21xx on Reddit for pointing this out. All credit for this idea goes to Carlin. ↩
Personal and team familiarity can also be part of the context. PHP may not seem to be the best choice for building a command-line tool, but if it's a small tool used by a team of PHP developers, then it might be the best choice. ↩

Earthly Stickers And Hacktoberfest

Adam Gordon Bell — Wed, 29 Sep 2021 16:13:03 +0000

Earthly is excited about Hacktoberfest and we are looking for contributors.

Last year we saw lots of great contributions from the community and sent out a bunch of Earthly stickers to participants.

What Is Hacktoberfest?

The hosting provider DigitalOcean is once again organizing the Hacktoberfest to celebrate open source. Hacktoberfest invites you to submit pull requests to open-source projects. People who submit four or more pull requests, get a free t-shirt from DigitalOcean.

What Is Earthly?

Earthly is the command-line build tool you didn't know you needed. It combines the isolation of containers with great usability.

Sticker Rewards

// Detect dark theme var iframe = document.getElementById('tweet-1329394070393925635-24'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1329394070393925635&theme=dark" }

In addition to this official event, Earthly would also like to kick in some rewards for people who contribute.

Earthly is a free and open build tool. It lets you build everything using containers.

The first 40 participants who submit at least one pull request that is accepted Earthly's GitHub Repository between October 1, 2021 and November 1, 2021, will get an earthly sticker. We have labeled many issues hacktoberfest to get you started.

You can also qualify for the sticker by adding a working Earthfile file to another open-source project, including your own personal project.

How Do I Claim my Earthly Stickers?

Once your PR has been accepted please fill out this form. All PRs that are contributing value will get an Earthly sticker!

Questions / Help ?

If you have any questions, or if you need any assistance, join our slack channel. We are here to help!

Questions about a specific issue or PR are best left on that issue or PR.

Happy hacking!

Markdown Linting

Adam Gordon Bell — Thu, 19 Aug 2021 15:22:34 +0000

Many linting, code formatting, and static analysis tools exist for code. You can use eslint, gofmt, or many other static analysis tools, combined with a great continuous integration process, and ensure that your code stays in good shape. But what about markdown files and documentation? How do you ensure you aren't committing spelling and grammar mistakes? How do you ensure your files are valid markdown and that the language you are using is clear and correct? You can do this and more with a documentation linter.

Many tools exist for finding problems in text files. You can use this list as a starting point for finding the markdown and prose linting tools that best fit your needs.

Docs as Code

The movement behind testing and linting prose is known as Docs as Code, and the Writing The Docs website is a great place to learn more.

Criteria

For Ease of skimming, I'll rate each tool based on this criteria:

Formatting: The ability to find errors in the formatting of text files (markdown, txt, asciidoc).
Spelling: The ability to find spelling mistakes.
Grammar: The ability to detect grammar errors.
Clarity: The ability to suggest changes that can improve writing clarity.

Additionally, I will rate tools based on their feature set:

Remediation: The ability to fix errors without manual intervention.
Customization: How well the tool can be customized to fit your use case. If you can't exclude a rule or disable a warning, CI usage may be challenging. The most robust tools support custom rules and documentation style guides.
Integrated Developer Environment (IDE) support: Ability to use in common code editors
Continuous Integration (CI) / Command Line Interface (CLI) Usage: Ability to be used at the command line and in a continuous integration environment.

Markdown Lint

markdownlint is a node.js markdown linter that is easy to install and easy to customize. It is based on an earlier Ruby tool, also called markdownlint. Both are great, but the Node.js tool is easy to install and easy to customize.

You can disable specific rules inline (  ) and set up a per-project config in a .markdownlintrc file. It also supports writing custom rules in JavaScript and can remediate many problems itself with the fix option:

markdownlint --fix "./_posts/*.md"

It doesn't handle spelling, grammar, or sentence structure, but it can't be beaten for dealing with markdown structure and it has a great online demo site.

Coverage

Formatting: 5
Spelling: 0
Grammar: 0
Clarity: 0

Features

Ease of Use: 5
Remediation: 5
Customization: 5
IDE support: 5
CI / CLI Support: 5

mdspell

mdspell is a tool specifically for spelling checking markdown documents. Install it like this:

npm i markdown-spellcheck -g

You can run it on markdown files in an interactive mode that builds up a custom dictionary of exceptions. You can then use that list later in a continuous integration process.

mdspell -n -a --en-us  ./blog/_posts/2021-02-11-mitmproxy.md

The downsides of mdspell are that the dictionary will likely complain about lots of words that are quite common. It may take some time to build up a list of exceptions. As a shortcut, you might be able to find some more .spelling files on GitHub.

Coverage

Formatting: 0
Spelling: 5
Grammar: 0
Clarity: 0

Features

Ease of Use: 5
Remediation: 5
Customization: 5
IDE support: 5
CI / CLI Support: 5

alex

alex does one thing: catches insensitive and inconsiderate writing. It supports markdown files, and works via command-line, and has various IDE integrations. The specificity of alex is its strength. For my rubric, I am scoring it under clarity as catching insensitive writing certainly improves clarity.

Coverage

Formatting: 0
Spelling: 0
Grammar: 0
Clarity: 3

Features

Ease of Use: 5
Remediation: 5
Customization: 5
IDE support: 5
CI / CLI Support: 5

`write-good`

write-good is designed for "developers who can't write good and wanna learn to do other stuff good too." The tool's focus is on improving the clarity of writing (and helping developers write well).

Install:

npm install -g write-good

Run:

$ write-good ./blog/_posts/2021-02-11-mitmproxy.md
here are several ways to accomplish this.
                         ^^^^^^^^^^
"accomplish" is wordy or unneeded on line 305 at column 26
-------------
e-ca-certificates` is an excellent proof of concept, but if you want to run a do
                         ^^^^^^^^^
"excellent" is a weasel word on line 367 at column 84

write-good has many exciting suggestions. It will highlight passive voice, cliches, weak adverbs, and much more. Unfortunately, it's not easy to exclude items or configure rules. It might be helpful as a writing suggestion tool, but this lack of configurability means you will have difficulty using it in a continuous integration process.

Coverage

Formatting: 0
Spelling: 0
Grammar: 0
Clarity: 2

Features

Ease of Use: 5
Remediation: 0
Customization: 1
IDE support: 2
CI / CLI Support: 2

textlint

textlint is a pluggable linting tool that supports markdown, plain text, and HTML. The plug-in architecture means that it can offer the features of some of the previous items by wrapping them up as a plug-in. It has a plug-in for alex, write-good, and for many spell checkers and grammar checkers. The downside of this flexibility is that it is a bit harder to set up and configure: you have to install each plug-in separately.

Install:

$ npm install textlint --global
# install each plugin
$ npm install --global textlint-rule-no-todo
....

Run:

textlint "docs/**"

textlint is configurable via an textlintrc and has inline exclude rules (  ) -- which may make it a possible way to use write-good or other tools that lack this functionality.

Coverage

Formatting: 0
Spelling: 3
Grammar: 3
Clarity: 4

Features

Ease of Use: 1
Remediation: 3
Customization: 4
IDE support: 5
CI / CLI Support: 2

proselint

proselint goes deep on writing clarity improvements in the same way the alex goes deep on inclusive writing:

proselint places the world's greatest writers and editors by your side, where they whisper suggestions on how to improve your prose. You'll be guided by advice inspired by Bryan Garner, David Foster Wallace, Chuck Palahniuk, Steve Pinker, Mary Norris, Mark Twain, Elmore Leonard, George Orwell, Matthew Butterick, William Strunk, E. B. White, Philip Corbett, Ernest Gowers, and the editorial staff of the world's finest literary magazines and newspapers, among others. Our goal is to aggregate knowledge about best practices in writing and to make that knowledge immediately accessible to all authors in the form of a linter for prose.

Some of the writing advice included is great:

echo "The very first thing you'll see at the top of every (well-written) bash script " | proselint
<stdin>:1:5: weasel_words.very Substitute 'damn' every time you're inclined to write 'very'; your editor will delete it and the writing will be just as it should be.

echo "Thankfully, not all the advice I received was bad. " | proselint
<stdin>:1:2: skunked_terms.misc 'Thankfully,' is a bit of a skunked term — impossible to use without issue. Find some other way to say it.

echo "it is worth noting that both for CI and CD, the operating principles and coding philosophy are equally as important as the technical aspect of the implementation." | proselint
<stdin>:1:96: after_the_deadline.redundancy Redundancy. Use 'as' instead of 'equally as'.

This one is awesome considering the context of the original article:

echo "thought leaders" | proselint
<stdin>:1:2: cliches.garner 'thought leaders' is cliché.

 echo "One elephant in the room with ngrok is" | proselint
<stdin>:1:5: corporate_speak.misc Minimize your use of corporate catchphrases like this one.

Learning from all the best writers is a very lofty objective, and proselint has accumulated some valuable rules, but it falls short of its goal of collecting all the worlds writing advice in a parsable form. Ignoring and excluding rules are also not fully supported.

Coverage

Formatting: 0
Spelling: 0
Grammar: 0
Clarity: 5

Features

Ease of Use: 5
Remediation: 0
Customization: 0
IDE support: 5
CI / CLI Support: 2

Vale

Vale, created by Joseph Kato, supports spelling, grammar, and clarity checks. It is extendable using a YAML rule format and is designed around the idea of a style guide -- a specific house style that you put together and vale enforces. It has an implementation of most proselint as a style guide, most of write-good, as well as the Microsoft Writing Style Guide and the Google
developer documentation style guide. Vale is targeted directly at the Docs-as-Code community and documentation teams, who take the writing style of documents very seriously.

Vale is fast and configurable but not necessarily easy to get started with. Initially, I couldn't get it to find any problems until I realized that it needs a config file to run:

MinAlertLevel = suggestion

[*]
BasedOnStyles = Vale

.vale.ini

Additionally, to use it effectively, you will need to copy an existing style-guide into your repository. Separating the styles from the tool is Vale's biggest strength. It's also could be a weakness as the rules you build up are specific to your repository. It is easy to write and customize rules but hard to share them back as they need to live in your source code repository.

Besides the official Vale style guides Buildkite, Linode, and Write The Docs have rules online that you can copy into your repo or use as inspiration for your own rules.

If you are taking linting documentation seriously and can take the time to set up a style that works for you, then Vale is the way to go. The rules of most other tools can be implemented inside value, and many already are.

Coverage

Formatting: 2
Spelling: 5
Grammar: 5
Clarity: 5

Features

Ease of Use: 1
Remediation: 0
Customization: 5
IDE support: 5
CI / CLI Support: 5

Vale Styles

Summary

Many tools exist for testing and linting English prose. You can start as simply as just spelling checking your readme before you commit it or go as complex as a full style guide running on every change to your software documentation.

If you are willing to invest the time, then Vale, with its flexible rules, is the clear leader. Combining Vale with markdownlint and running both in a continuous integration build should ensure that documents are spelling correctly, grammatically correct, and written in a properly formatted and exclusive way.

If you're looking for a more accessible place to start or don't need the grammar and clarity suggestions, then mdspell and markdownlint make a great combination.

Once you have decided on what tools will work best for you, make sure you find a way to automate their usage. This blog uses Vale and markdownlint inside an Earthfile that is run every commit. This helps us prevent mistakes from getting into the blog.

Python Concatenate Lists

Adam Gordon Bell — Mon, 16 Aug 2021 19:35:56 +0000

Concatenate Two Lists in Python

Problem: You have two lists and you'd like to join them into a new list.
Solution:

Python 3.8.2
>>> one = ["one","two", "three"]
>>> two = ["four","five"]
>>> one + two
['one', 'two', 'three', 'four', 'five']

📢 TLDR: Use +

In almost all simple situations, using list1 + list2 is the way you want to concatenate lists.

The edge cases below are better in some situations, but + is generally the best choice. All options covered work in Python 2.3, Python 2.7, and all versions of Python 3¹.

Combine Lists in Place in Python

Problem: You have a huge list, and you want to add a smaller list on the end while minimizing memory usage.

In this case, it may be best to append to the existing list, reusing it instead of recreating a new list.

>>>longlist = ["one","two", "three"] * 1000
['one', 'two', 'three', 'one', 'two', 'three', ... ]
>>>shortlist = ["four","five"]
["four","five"]
>>> x.extend(y)
>>> x
['one', 'two', 'three', 'one',  ..., "four","five"]

As with any optimization, you should verify that this reduces memory thrash in your specific case and stick to the simple idiomatic x + y otherwise.

Let's use the timeit module to check some performance numbers.

# Performance Check
>>> setup = """\
x = ["one","two","three"] * 1000 
y = ["four","five","six"]
"""
# x + y with large x
>>> timeit.timeit('x + y', setup=setup, number=1000000)
3.6260274310000113
# x.extend(y) with large x
>>> timeit.timeit('x.extend(y)', setup=setup, number=1000000)
0.06857255800002804

In this example, where x is 3000 elements, extend is around 50x faster.

❗ Concatenating Lists With Large Elements is Fine

If the elements in your list are huge (million character strings), but the list size is less than a thousand elements, the previous solution x + y will work just fine. This is because Python stores references to the values in the list, not the values themselves. Thus, the element size makes no difference to the runtime complexity.

>>> x = ["one" * 1000, "two" * 1000, "three" * 1000]
>>> y = ["four" * 1000, "five" * 1000]
>>> #This is fine
>>> z = x + y
>>> #Performance Testing (extend is slower for large elements)
>>>  setup = """\
x = ["one" * 1000, "two" * 1000, "three" * 1000]
y = ["four" * 1000, "five" * 1000]
"""
>>> timeit.timeit('x + y', setup=setup, number=1000000)
0.05397573999994165
>>> timeit.timeit('x.extend(y)', setup=setup, number=1000000)
0.06511967799997365

In this case, extend does not have an advantage.

Avoid Chain From `itertools` For Two Lists

It is possible to use chain from itertools to create an iterable of two lists.

>>>longlist = ["one","two", "three"] * 1000
['one', 'two', 'three', 'one', 'two', 'three',, .......... ]
>>>shortlist = ["four","five"]
["four","five"]
>>> from itertools import chain
>>> z = list(chain(longlist, shortlist)
['one', 'two', 'three', 'one', , .........., "four","five"]

We can check the performance of using chain:

>>> setup = """\
from itertools import chain
x = ["one","two","three"] * 1000 
y = ["four","five","six"]
"""
# x + y with large x
# x.extend(y) with large x
>>> timeit.timeit('x.extend(y)', setup=setup, number=1000000)
0.06857255800002804
>>> timeit.timeit('list(chain(x, y))', setup=setup, number=1000000)
16.810488051999982

Using chain with two lists is slower in all cases tested, and x + y is easier to understand.

Combining N Lists in Python

If you need to add three or even ten lists together and the lists are statically known, then + for concatenate works great.

>>> one = ["one","two", "three"]
>>> two = ["four","five"]
>>> three = []
>>> z = one + two + three

Flatten a List of Lists in Python

However, if the number of lists is dynamic and unknown until runtime, chain from itertools becomes a great option. Chain takes a list of lists and flattens it into a single list.

>>> l = [["one","two", "three"],["four","five"],[]] * 99
[['one', 'two', 'three'], ['four', 'five'], [], ...
>>> list(chain.from_iterable(l))
['one', 'two', 'three', 'four', 'five', 'one', 'two', ... ]

chain can take anything iterable, making it an excellent choice for combining lists, dictionaries, and other iterable structures.

>>> from itertools import chain
>>> one = [1,2,3]
>>> two = {1,2,3}
>>> list(chain(one, two, one))
[1, 2, 3, 1, 2, 3, 1, 2, 3]

Performance of Flattening a List of Lists

Performance doesn't always matter, but readability always does, and the chain method is a straightforward way to combine lists of lists. That said, let's put readability aside for a moment and try to find the fastest way to flatten lists.

One option is iterating ourselves:

result = []
for nestedlist in l:
    result.extend(nestedlist)

Let's check its performance vs chain:

>>> setup = """\
from itertools import chain
l = [["one","two", "three"],["four","five"],[]] * 99
"""
>>> # Add Nested Lists using chain.from_iterable
>>> timeit.timeit('list(chain.from_iterable(l))', setup=setup, number=100000)
1.0384087909997106
>>> ### Add using our own iteration
>>> run = """\
result = []
for nestedlist in l:
    result.extend(nestedlist)  
"""
>>> timeit.timeit(run, setup=setup, number=100000)
1.8619721710001613

This shows that chain.from_iterable is faster than extend.

Flattening and Merging Lists With One Big List

What about adding a list of lists to an existing and large list? We saw that using extend can be faster with two lists when one is significantly longer than the other so let's test the performance of extend with N lists.

First, we use our standard chain.from_iterable.

>>> # Method 1 - chain.from_iterable 
>>> longlist = ["one","two", "three"] * 1000
>>> nestedlist = [longlist, ["one","two", "three"],["four","five"],[]]
>>> list(chain.from_iterable(nestedlist))

We then test its performance:

>>> setup = """\
from itertools import chain
longlist = ["one","two", "three"] * 1000;
combinedlist = [longlist, ["one","two", "three"],["four","five"],[]]
"""
>>> timeit.timeit('list(chain.from_iterable(combinedlist))', setup=setup, number=100000)
1.8676087710009597

Next, let's try concatenating by adding everything onto the long list:

>>> # Method 2 - extend
>>>  longlist = ["one","two", "three"] * 1000
>>>  nestedlist = [["one","two", "three"],["four","five"],[]]
>>>  for item in nestedlist:
>>>     longlist.extent(item)

Performance Test:

>>> setup = """\
from itertools import chain
longlist = ["one","two", "three"] * 1000;
nestedlist = [["one","two", "three"],["four","five"],[]]
"""
>>> run = """\
for item in nestedlist:
   longlist.extend(item)
   """
>>> timeit.timeit(run, setup=setup, number=100000) 
0.02403609199973289

There we go, extend is much faster when flattening lists or concatenating many lists with one long list. If you encounter this, using extend to add the smaller lists to the long list can decrease the work that has to be done and increase performance.

Summary

These are the main variants of combining lists in python. Use this table to guide you in the future.

Also, if you are looking for a nice way to standardize the processes around your python projects -- running tests, installing dependencies, and linting code -- take a look at Earthly for Repeatable Builds.

Condition	Solution	Performance Optimization²
2 lists	`x + y`	No
1 large list, 1 small list	`x.extend(y)`	Yes
Known number of N lists	`x + y + z`	No
Unknown number of N lists	`list(chain.from_iterable(l))`	No
List of Lists	`list(chain.from_iterable(l))`	No
1 large list, many small lists	`for l1 in l: x.extend(...)`	Yes

I did all the performance testing using Python 3.9.5 on MacOS BigSur. ↩
If you don't have a performance bottleneck, clarity trumps performance, and you should ignore the performance suggestions. ↩

DEV Community: Adam Gordon Bell

What's gotten better in programming languages

Full Backend Apps on AWS Lambda?

Do you use GRPC?

Web3 will not replace Contract Law?

Knowing I'm Dumb Is the First Step

Incident Management Metrics and Key Performance Indicators

Mean Time Between Failures (MTBF)

Calculating MTBF

Mean Time to Recovery (MTTR)

Calculating Mean Time to Recovery

Mean Time to Resolve (MTTRe)

Calculating Mean Time to Resolve

Mean Time to Acknowledge (MTTA)

️Calculating Mean Time to Acknowledge

Summary

Property-Based Testing In Go

Testing csvquote

Property-Based Testing

Quick Check

Writing a Property Testing Generator

Property Testing

More Resources

Migrating Your Open Source Builds Off Of Travis CI

What Happened?

Why It Matters

Where to Move To

Option: Run Your Own Builds

Option: Circle CI

Best Option: GitHub Actions

Summary of Open Source Plans

Don't Let This Happen Again

Neutral Build Specifications

Makefiles and Dockerfiles

Earthly

Other Interesting Options

Easier Migration from Travis to GHA

Serverless Builds

Take-Aways

Idiots And Maniacs

Observability

Driving

Testing

More Examples

Earthly Stickers And Hacktoberfest

What Is Hacktoberfest?

What Is Earthly?

Sticker Rewards

How Do I Claim my Earthly Stickers?

Questions / Help ?

Markdown Linting

Criteria

Markdown Lint

Coverage

Features

mdspell

Coverage

Features

alex

Coverage

Features

write-good

Coverage

Features

textlint

Coverage

Features

proselint

Coverage

Features

Vale

Coverage

Features

Vale Styles

Summary

Python Concatenate Lists

Concatenate Two Lists in Python

Combine Lists in Place in Python

Avoid Chain From itertools For Two Lists

Combining N Lists in Python

Testing `csvquote`

`write-good`

Avoid Chain From `itertools` For Two Lists