DEV Community: Kam

Everyone fails at software they don't build

Kam — Thu, 09 Mar 2023 15:29:51 +0000

There are three types of software systems that any company runs. First, high value software systems that drive customer value. Second, the systems necessary to deliver the product or service, but not sufficient as stand alone products on their own. Third, systems that support the business but are not related to the competitive advantage of the company. The fourth type of software system is one that doesn’t exist. Every company has software they chose not to build. This is "Unbuilt Software". To understand how much value organizations are missing out on, we must first better understand each type of software system and how software investment decisions are made.

Tier 1 - High Value Systems

These systems or services compose the key building blocks for the company. They are the services that set the standard for what other software in the company will comply with. Meaning their requirements for security, performance, or quality set the standards other software at the organization is expected to meet. Since they are the highest value, they dictate what language and frameworks will be used and what architectural and hosting patterns are used by engineers in the company. These systems establish the culture and establish what skills are high status inside of the engineering groups.

Tier 2 - Necessary Systems

The second type of software systems are software systems that are necessary but not sufficient for the company to exist. These are systems that will use the frameworks, languages, testing processes, release methodology, and security frameworks that the high value systems use. These systems are important for the company but do not drive the same outsized value of Tier 1 systems, hence they do not warrant the same level of investment to create new policies, procedures, or technologies.

Tier 3 - Supporting Systems

Companies have other systems that are necessary for the company to exist. If the company was to be formed again they might not even be internal to the company, but at this point it would cost more to migrate to some other provider than to just stay. This is the classic: "We have been running X for years. It works fine. Why do we need to spend money to switch?" These systems are usually one-off ancillary systems handled by a particular department that has a rollup of functionality underneath them; this is often the domain of an IT operations group.

How to fail - Tier 1

You can fail at the first type of system by not truly understanding the constraints in which you are operating. These systems might have very high levels of requirement for performance. For example, If the engineers do not understand the mechanisms and are not schooled in the particular technologies or optimization required to meet those objectives, this can become an existential threat to the company.

How to fail - Tier 2

You can cause the second type of system to fail by overly constraining their ability to be improved. If the security and build processes needed in the first type of systems constrain the second type of system such that they do not and are not upgraded and improved, they will fall out of pace with the industry. This creates an opportunity for competitors to enter the market with improved efficiencies around the second types of systems, which lowers their overall cost basis. Unfortunately, this can then become an existential threat to the company.

How to fail - Tier 3

A way to fail with the third type of system is to not understand the sunk cost fallacy. These systems are necessary, but any additional investment is throwing good money after bad. If they are so critical that they are necessary for the business to operate then they must be considered one of the other types of systems. It seems the primary way to fail with these types of systems is to let them absorb budget and time from the business beyond their business value.

It’s much better to take three days or a week of downtime at your convenience and at known time versus to risk having the system fail completely when you are unprepared and possibly a critical time for your business. Any effort or any investment of time and money should be focused on migrating or eliminating the systems all together. It is much better to have an account with a company whose business it is to provide that service as that company's first priority versus to have it as an afterthought or red-headed stepchild within yours.

How to fail - Tier 4 The software you don’t build

What this analysis leaves out is the types of systems that do not exist. These are the systems that are speculative. When building software, oftentimes, the full value of the system does not become apparent till the system is beyond a prototype. Invention and innovation are created through experimentation. If the constraints of an organization's build and deployment systems limit the range of software that an individual may build, then the organization will limit the value of the software they may build because they are not able to anticipate the value created by the experiments.

Any minimum fixed cost of building software will cause the organization only to build projects that have reasonable certainty of exceeding the cost of building them. Organizations struggle to prioritize building software that is hard to quantify the value of a system. Examples are tools that more quickly surface information from multiple sources for employees, tools that eliminate communication between groups, or tools that reduce delay for individuals.

If the value of software systems is distributed as a power law then to increase value you should take more samples from the distribution. This implies that it is most important to increase the number of potential opportunities to have a runaway success. If the only way to experiment is to pay the tax of complying with Tier 1 processes, you’re sentencing your organization to a decreased amount of innovation.

A way out

One way to increase the number of experiments is to enable the largest number of individuals in your organization to conceive of and implement these experiments. Each organization has some individuals who are able to do so without assistance. A system that enables them to do so with less friction and less difficulty will increase the frequency that they experiment. Lowering the difficulty will also increase the number of individuals in your organization that are able to experiment. This means creating a way for individuals to rapidly prototype new ideas outside of the usual constraints of Tier 1 systems, but in a way that they can evolve into Tier 1 systems. This capability is a strategic advantage to an organization. Having an experimentation system will create long term value for the organization.

Has your software team found a way to succeed at Tier 4? Visit us in our Discord community and share your successes.

Talking About Failure

Kam — Thu, 29 Sep 2022 20:13:54 +0000

When I was a younger I went on several rock climbing and mountaineering expeditions. I was exposed to some instructors and practitioners that took staying safe in the mountains very seriously. One of them carried a book from The American Alpine Club on climbing accidents. The accidents were never the cause of a single bad choice. They were caused by a series of decisions, taken over time, that combined to create the conditions for the accident.

What stuck in my memory, sitting beside rock faces that could very well be the scene for one of the stories, was that in certain cases we might make one of the same choices. That given a different time of day, or different weather, or different climbing partners, the same choice could have different levels of risk. Reading and discussing those stories made me a more aware adventurer and a safer climber.

The public analysis and discussion of the chain of decisions and events that led to accidents helps beginners build experience, experts grow wisdom and fosters a culture of safety. Other industries and sub-cultures, to a have their own ways to learn from accidents. For example: the US military has after action reviews, medicine has accident review boards and aviation has NTSB accident reviews.

I have read root cause analysis reports from outages at hyper scalar cloud hosting providers. Often they read like the plaster foot prints of a long dead animal, the legal and marketing departments sanitizing and scrubing any element of life or learning from them. They are the cargo culting of accident investigation and reporting. All of the motions and affectation with none of the substance.

The prevailing culture in software to errors seems to be "if you just". Where errors are blamed on lack of knowledge. If the operator had "just" known the impact of config change, if the developer had "just" understood the interaction of their code change. It is a culture of expert knowledge. It is based on the unexamined belief that bugs only exist from lack of knowledge or lack of care. Or taken to their offensive extremes: stupidity and laziness.

In the software industry bugs or outages most often do not result in injury or death. However, as society becomes more dependent on the software systems we are building the severity and impact of problems only grows. As our software systems continue to grow in complexity and the reach of problems we write software to solve, we as a society must grapple with changing this culture.

As a step on this path of changing this culture I am talking about failures I have seen or participated in. My goal is to make it easier for others to share where they have failed. This will create a feedback loop that will let us learn faster.

Join me in sharing stories of failure. Here are some talks and posts about failure from myself and Cyclic:

1) How to fail at Serverless (DevOpsDays Boston 2022 Conference video
2) How to Fail at Serverless: Serverless is Stateless (blog post)
3) Its Always Sunny in us-east-1: The gang does business continuity (blog post)
4) AWS S3: Why sometimes you should press the $100k button (blog post)

We would love to hear your stories.

Write something up.

Post it publicly.

Let us know.

We got your back :)

How to Fail at Serverless: Serverless is Stateless

Kam — Mon, 11 Jul 2022 12:14:15 +0000

This series is an extended version of a talk I gave at Serverless Days NYC 2022. The goal is to share ways that a "friend of mine" has failed at serverless to help level up the community. Just as the transportation industry shares accident report analysis with the whole industry to improve safety, we in the software community need to do the same. This is my attempt to do that.

I'm sure there is some product X that has feature Y that fixes issue Z. I either didn't know about it at the time, didn't have the budget, couldn't get it through vendor/security review, product wouldn’t schedule it with our engineers or it didn't comply with enterprise architecture's plan for our software roadmap. As some guy once said: “You go to production with the tools you have, not the tools you want.”

Note:

Names have been changed to protect the innocent and not so innocent.
Only egos (and wallets) were harmed by the failures recounted here.
Originally published at How to Fail at Serverless: Serverless is Stateless

Serverless is Stateless

All the architecture posts talk about serverless as stateless computing. How serverless is so scalable because it scales out to satisfy increased load, and scales in when not needed. This all makes sense. Lambda is like a big thread pool in the sky. Got it.

I can set some parameters to manage the maximum and minimum number of threads and AWS would handle all the hard work of hosting, spinning up, spinning down and networking the instances together. Awesome-sauce.

Well, almost...

Use Case

Background job that receives batches (10's-100's) of PDFs from the client, splits them apart into individual pages (1-1000 per PDF) and does analysis on each page. The arrival rate from the client is lumpy. Primarily US working hours on a 5 min interval. Response is measured in hours. This feels like a perfect HelloWorld use case for StepFunctions doing orchestration and Lambda doing image manipulation.

Lambda logic:

Invoke with S3 path to file
Read 10+mb files from S3 to /tmp
Process file
Write results to S3
Respond with S3 path to transformed file
Serverless FTW!

Everything works great until... inconsistent out of disk space errors, or no space left on device (depending on your OS/language stack).

How to Fail

Problem

Periodically a step function run will fail due to a Lambda invoke failing on no space left on device (or similar depending on os and language). Running the same file in dev/test doesn't reproduce the error. Re-running the step function in production succeeds.

Solution

Well lambda instances have a writable /tmp folder where you can store temporary files. And somewhere in the AWS docs they describe how instances of your Lambda function are reused between invocations.

Some quick googling gets you a solution of clearing /tmp at the start and/or end of each invocation (stackoverflow ftw). Till that fix makes it to production you can just re-execute any failed step function runs.

Nothing quite like shipping code copy-pasted from Stack Overflow. Nice.

Success! Party!

until... wait what? out of memory errors? WTF!

Problem v2

Periodically a step function run fails. The Lambda invoke error is no memory left (or similar depending on os and language). Running the same file in dev/test doesn't reproduce the error. Re-running the step function in production succeeds. Turning on debug or releasing new builds adding additional logging suppress the error rate. WTF (why the failure?) Load tests don't reproduce the error.

The explanation that AWS just fails on 1 in ~100k lambda invokes starts to wear thin with your Ops team that is getting paged to click a button. But we could just automate the re-run!

Deep inside accepting it "just fails" violates some axiom about how computers works. This is knowable and damn-it I shall know. So you strap the lambda down setting max concurrency to 1. Turn up the load test. Let’s do this.

At 12k documents the lambda invoke fails for out of memory. Interesting.

Rerun the load test. At 12k documents the lambda invoke fails for out of memory.

If an error is repeatable it isn't random. 12k is an important number. It also happens to be the maximum number of file descriptors allowed for the linux OS the lambda is running on.

Back to google. Apparently the image manipulation library I am using has a history of bugs with leaving open file descriptors. Looks like each invoke to lambda is leaving another file descriptor open in the OS file descriptor table.

The lambda will then fail when it hits 12k unless it is recycled. The lambda instance can be recycled due to:

a code deployment: new code, new instances
update to environment variable: hence why turning on debugging suppressed errors.
scaling down: a burst in load will scale out new instances and then scale back down. Only some instances survive.

Solution v2

Expand the IAM role permissions to allow the lambda permission to update itself (your security team might give you side eye for this, fair warning, send them this blog post, I'm sure it will convince them).

After completing the work of an invoke, call the lambda api to set an environment variable to a random value. We dubbed this the "self-immolation" solution.

Hacky? Yeah.

Chaos for the lambda team? Probably.

Did it work? Yes.

Stay tuned, for more stories of #fail.

6 CLI Tools All Experts Know

Kam — Mon, 27 Jun 2022 15:13:45 +0000

I work on Mac OSX and Linux command line environments. My tool kit is shaped by the needs I have. This is my tool box of tools that work with almost any command and help me go from just using the command line to being an expert.
Wow, your friends and colleagues. These tools make every other command at the command line more powerful.

Originally published on my personal site: 6 Command Line Tools All Experts Know

`|` (aka pipe)

Pipe is awesome. It lets me connect the output of one command and "pipe" it into the input of another command. This is the universal glue that instantly lets me level up the effectiveness of my command line usage.

Gotchas

There are two outputs from each command, standard out (stdout) and standard error (stderr). Pipe by default connects to stdout.

`-` (aka dash)

Some commands take a file name as an argument. In cases where I want to use stdin or stdout. Dash is the answer.
I use this alot when I want to read a file from s3 and display it to the console or take the output from a command and write the file to s3.
Print file to console: aws s3 cp s3://some-bucket-name/awesome-file.json -

`pbpaste` and `pbcopy`

These are technically commands as well. They interact with the paste buffer.
On mac you can access your paste buffer using the paste-buffer-paste (pbpaste) and paste-buffer-copy (pbcopy) commands. I love how they keep my console clean especially when I have trying to transform a large chunk of text/json
Example:
aws s3 cp s3://some-bucket-name/awesome-file.json - | pbcopy

`xargs`

This is technically a command. It is a command that executes commands.
It executes a command for every line of input it receives. The arguments you pass to xargs form the basis of the command to execute.
Hint: use -J{} to substitute {} (or some other character string) in the middle of the command to execute or multiple times for that matter.
Example: pbpaste | xargs -J{} aws s3 cp {} -

`2>` and `2>&1`

Sometimes I want to send stderr somewhere else other than the console. Using its number is the way.
Send to trash: 2>/dev/null
Send to error log: 2> error-log.txt
Combine with stdout: 2>&1

Ctl-z then `fg`

Suspend the foreground process, run some other commands, then return the suspended process to the foreground.

`man`, `/`, `n`

When I know the command but not the exact syntax my first source of information is the manual. Instead of searching google I search the manual.
Example: man xargs then /utility advance to next found instance with n
‍

How to Debug

Kam — Thu, 09 Jun 2022 15:43:10 +0000

This was originallly posted at https://www.cyclic.sh/posts/how-to-debug

We all hit bugs that feel impossible to diagnose. The hardest ones to debug are intermittent or inconsistent. How do I fix what works sometimes? If a line of code always breaks then the fix is direct. The system is linear. The action leads to a failure.

How do I debug a bug that is intermittent? How do I debug something that works sometimes?

Here are some reminders to myself next time I encounter a baffling debugging session.

Choose Learning

My greatest learnings come from debugging. This bug is an opportunity for me to learn more about this system. If I already knew what was causing the problem then I wouldn't have a bug. The fact that I have a bug that isn’t obvious to me on how to fix, by definition means I have a chance to learn.

The system is giving me the gift of the chance to learn.

Choose to take the opportunity to learn.

Chase the Error

Answer the question: "how do I make this fail every single time?"

Instead of continuing to make the system work. Instead make the system fail. Go straight at the error. Maximize it. Increase the error rate. Trigger the error repeatedly.

Focus on driving the error rate to 100%. Only then can I be sure any success is from my action and not inconsistency.

Make predictions to test my understanding: "The system will fail, now (pushes button)"

Simplify

Once I can create the error on demand, I isolate. I begin simplifying the steps to trigger. Remove steps or hard code responses. Trim excess complexity not needed to cause error.

If it takes 6 steps to reproduce, try turning each one off in order.

Find the minimum degrees of freedom to still reproduce the error.

Assess the Inverse

Reminder: the error is not caused by something I understand. Look explicitly for areas where I assume the system is working.

What are the parts of the system that I am sure are working? What if they weren’t working as I expected? Verify the components are actually working as I expect.

Reminder: It’s not what we don’t know that gets us into trouble, it is what we know for sure that isn't true

Devise a Conclusive Measure

Brainstorm what report, measure, logging or data would definitely tell me what is causing the error. Maybe this won't tell me “why” the error is happening, but point me closer to "where" the error is being caused.

Question: What piece of information would conclusively tell me what the problem was?

Take a Break

I might be making things worse. If I don’t know what causes the error I don’t know if I’m making it worse. If the situation allows for it, take a break.

Get some air. Go for a walk. Breath.

A clear relaxed mind is open for new ideas.

Sleep on it. Pick it up tomorrow.

Explain How to Reproduce

Say it out loud.

Talk to myself.

Explain the steps to reproduce.

Start at the beginning.

Back up three steps and explain again.

Slower this time.

Summary

Hopefully this helps you debug an issue you are struggling with. If you have other techniques or improvements to these let me know: twitter.com/seekayel

For those who jump to the end, here is the tldr:

Choose Learning
Chase the Error
Simplify
Assess the Inverse
Devise a Conclusive Measure
Take a Break
Explain How to Reproduce