DEV Community: Roman Leventov

Enable CORS with custom headers for an AWS Lambda function behind API Gateway in Serverless framework

Roman Leventov — Thu, 09 Apr 2020 06:37:58 +0000

To properly enable CORS with custom headers for a Lambda function deployed behind API Gateway using Serverless framework, you need to do three separate things:

Add `cors` configurations to HTTP points of the function definitions in your `serverless.yml`

This includes CORS headers to preflight OPTIONS requests to your API:

functions:
  getProduct:
    handler: bin/get_product
    events:
      - http:
          path: product/{id}
          method: get
          cors:
            origin: "*"
            headers:
              - Content-Type
              - X-Amz-Date
              - Authorization
              - X-Api-Key
              - X-Amz-Security-Token
              - X-Amz-User-Agent
              - <your-custom-header-goes-here>

Add `resources` section to `serverless.yml` to include CORS headers in API Gateway-level error responses

Such as due to expired authentification token, unauthorized access, or 404:

resources:
  Resources:
    GatewayResponseDefault4XX:
      Type: 'AWS::ApiGateway::GatewayResponse'
      Properties:
        ResponseParameters:
          gatewayresponse.header.Access-Control-Allow-Origin: "'*'"
          gatewayresponse.header.Access-Control-Allow-Headers: "'Content-Type,X-Amz-Date,Authorization,X-Api-Key,X-Amz-Security-Token,X-Amz-User-Agent,<your-custom-header-goes-here>'"
        ResponseType: DEFAULT_4XX
        RestApiId:
          Ref: 'ApiGatewayRestApi'
    GatewayResponseDefault5XX:
      Type: 'AWS::ApiGateway::GatewayResponse'
      Properties:
        ResponseParameters:
          gatewayresponse.header.Access-Control-Allow-Origin: "'*'"
          gatewayresponse.header.Access-Control-Allow-Headers: "'Content-Type,X-Amz-Date,Authorization,X-Api-Key,X-Amz-Security-Token,X-Amz-User-Agent,<your-custom-header-goes-here>'"
        ResponseType: DEFAULT_5XX
        RestApiId:
          Ref: 'ApiGatewayRestApi'

ResponseType fields control what type of responses the corresponding resource section applies to. If you don't want to include CORS headers in every 4XX or 5XX error response, you can find some more specific ResponseTypes here.

Note: GatewayResponseDefault4XX and GatewayResponseDefault5XX just resource names and have no significance.

Include CORS headers in the normal responses in your lambda handler

Example in Go:

func Handler(ctx context.Context, req events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) {
    rsp, err := apiGatewayAdapter.ProxyWithContext(ctx, req)
    rsp.Headers["access-control-allow-origin"] = "*"
    rsp.Headers["access-control-allow-headers"] = "Content-Type,X-Amz-Date,Authorization,X-Api-Key,X-Amz-Security-Token,X-Amz-User-Agent,<your-custom-header-goes-here>"
    return rsp, err
}

Resources:

Engineering Ideas #9

Roman Leventov — Fri, 13 Mar 2020 14:16:17 +0000

Why Discord is switching from Go to Rust

The following statement caught my attention in this post by Jesse Howarth:

Discord has never been afraid of embracing new technologies that look promising. For example, we were early adopters of Elixir, React, React Native, and Scylla. If a piece of technology is promising and gives us an advantage, we do not mind dealing with the inherent difficulties and instability of the bleeding edge. This is one of the ways we’ve quickly reached 250+ million users with less than 50 engineers.

Which sounds almost the direct opposite of the experience at Instagram (citation from “The Effective Engineer” by Edmond Lau):

Whenever they could, the Instagram team picked proven and solid technologies instead of shiny or sexy new ones. “Every single, additional [technology] you add,” Krieger cautions, “is guaranteed mathematically over time to go wrong, and at some point, you’ll have consumed your entire team with operations.” And so, whereas many other startup teams adopted trendy NoSQL data stores and then struggled to manage and operate them, the Instagram team stuck with tried and true options like PostgreSQL, Memcache, and Redis that were stable, easy to manage, and simple to understand. They avoided re-inventing the wheel and writing unnecessary custom software that they would have to maintain. These decisions made it significantly easier for the small team to operate and scale their popular app.

The conclusion that I make from this contradiction is that sticking to proven tech vs. embracing new tech is not really that important for the scalability of engineering resources. Perhaps, instead of trying to explain our successes, we should focus on understanding what helped us to avoid failure. Nassim Taleb’s Via negativa and Will Larson’s Iterative Elimination Tournaments come to mind.

Key Points from NoSQL Distilled

These points from Pramod Sadalage’s and Martin Fowler’s book “NoSQL Distilled” that I found most insightful:

The rise of NoSQL appears to be connected (or enabled by) to the rise of service-oriented architectures:

There is a movement away from using databases as integration points towards encapsulating databases within applications and integrating through services.

One of the main factors to consider when choosing between a relational and a NoSQL database:

Aggregate-oriented databases work best when most data interaction is done with the same aggregate; aggregate-ignorant databases are better when interactions use data organized in many different formations.

On the tradeoff between technology suitability and the toolbox size (connects to the discussion above about variety vs uniformity in the tech stack):

Adding more data storage technologies increases complexity in programming and operations, so the advantages of a good data storage fit need to be weighed against this complexity.

Documenting Architecture Decisions

Michael Nygard presents a simple framework for documenting the changes and additions to the design of the system. The benefits:

The motivation behind previous decisions is visible for everyone, present and future. Nobody is left scratching their heads to understand, "What were they thinking?" and the time to change old decisions will be clear from changes in the project's context.

The architecture decision records (ADRs) could be committed to the project repository under a separate directory, like doc/adr/. The structure of an ADR:

Title. These documents have names that are short noun phrases. For example, "ADR 1: Deployment on Ruby on Rails 3.0.10" or "ADR 9: LDAP for Multitenant Integration"

Context. This section describes the forces at play, including technological, political, social, and project local. These forces are probably in tension, and should be called out as such. The language in this section is value-neutral. It is simply describing facts.

Decision. This section describes our response to these forces.

Status. "Proposed"/"accepted"/"superseded" with a reference to its replacement.

Consequences. This section describes the resulting context, after applying the decision. All consequences should be listed here, not just the "positive" ones. A particular decision may have positive, negative, and neutral consequences, but all of them affect the team and project in the future.

Here are some real examples. In Apache Druid project, “Proposal” label serves some similar purpose.

For foundational decisions, I think it makes sense to use a more elaborate structure, for example, the one described in the following article:

Architecture Decisions: Demystifying Architecture

Jeff Tyree and Art Akerman enumerate five types of objectives that drive architecture decisions: business needs (aka functional requirements), risks , system issues , change cases (which can be addressed by creating options), and non-functional requirements.

When does a decision require heavyweight documentation?

To test a decision’s architectural significance, an architect should ask the following question: does this decision affect one or more system qualities (performance, availability, modifiability, security, and so on)? If so, an architect should make this decision and document it completely.

Authors propose the following sections for a decision document:

Issue (problem, subject matter). Explain the urge for making the decision (rather than deferring it for later, which should be the default strategy): “Describe the architectural design issue you’re addressing, leaving no questions about why you’re addressing this issue now.”
Decision (short statement)
Status
Assumptions
Constraints (boundary conditions): organizational, human, schedule, cost, risk constraints, non-functional requirements, alignment with the business and technology strategy.
Positions (alternatives, options). “This section also helps ensure that you heard others’ opinions; explicitly stating other opinions helps enroll their advocates in your decision.”
Decision: a longer description
Implications of the decision
Related decisions
Related requirements. Show how the decision is driven by the objectives: “Decisions should be business-driven. To show accountability, explicitly map your decisions to the objectives or requirements. […] If a decision doesn’t contribute to meeting a requirement, don’t make that decision. ”
Related principles. “If the enterprise has an agreed-upon set of principles, make sure the decision is consistent with one or more of them. This helps ensure alignment along domains or systems.” I think that industry best practices or the prior experience (within the team, personal, or found online) could also be referenced here.

If that was not enough, I think it makes sense to emphasize a few more things (deriving mostly from The Effective Executive and The Effective Engineer):

What hard data (evidence) supports the decision?
What conflict is baked into the decision (every important decision has it)? This could be a separate section “Drawbacks”, or a sub-section within “Implications”. How the conflict is addressed (drawbacks mitigated)?
What metric of feedback loop will help to verify that the decision is effective in the future?
Who is responsible for carrying out the decision? What is the implementation plan? The specific next step?

Is there anything else that you think is important to check in architecture documents? Please share it in comments!

This repository curated by Joel Parker Henderson has many links for further reading and alternative architecture decision record templates.

How To Read A Technical Article

Steve McConnell defines two main types of reading:

Inspectional: get the most out of a book or article within a given amount of time
Analytical: get the most out of a book or article with an unlimited amount of time

The process of inspectional reading:

Study the table of contents. What subject areas are covered? What is the book’s emphasis? Can you infer the author’s main points from the table of contents?
Read the book’s preface.
Study the book’s index.
Read the introductory text of each chapter.
Dip into the first and last chapters of the book.

McConnell advises reading inspectionally even when we plan to read analytically anyway:

In case you do need to perform Analytical reading, Inspectional reading prepares you for that. In reading a book or article for understanding rather than just for information, you need to both acquire an understanding of the way the author has organized the subject matter and an understanding of the subject matter details. By jumping into the details first, starting at page 1 and reading through to the end, you have to acquire both of these kinds of knowledge simultaneously, which is very difficult. By performing Inspectional reading first, you acquire an understanding of the organizational framework quickly, and you can then fit the details into the framework during your more detailed reading of pages 1 to n.

How to Motivate a Team of Software Developers

Philipp Hauer writes from his experience about nurturing a motivating atmosphere in the team. Below are some ideas that jumped out for me. The first two echo Daniel Coyle’s “The Culture Code”:

We usually hold a meeting when we have to come up with a database schema, architecture, design decisions or bigger refactorings in our software. After my experience, the outcome of those meetings is great and much better than the solution of a single developer. Swarm intelligence impresses me every time!

The above also reminds me of Earl Beede’s take on the three best decision-making strategies, two of them are communal:

Delegation to those who will be implementing the decision
One person decides after a discussion. Everybody feels heard and understood.
Consensus. Check often that everybody is still on board, to avoid small disagreements.

Noncontrolling language. The phrasing is crucial. This applies not only for feedback in code reviews but also for the daily communication within a team. Try to avoid phrases that contain “must” or “should”. Instead, use “think about …” or “have you considered …”. Again, ask questions.

At Spreadshirt, a developer can make an “internship” in other teams for one or two weeks. I can’t highlight enough the value of those internships: It’s so interesting to see how other teams work, what are their processes, what are their technologies and solutions and what are the social dynamics. And the knowledge exchange works in both directions.

The following reminds me of Edmond Lau’s idea about picking the right metric:

A developer put a lot of effort into optimizing the performance of a remote call or an algorithm? It’s so rewarding if there are metrics (e.g. in Grafana) showing the improvement in terms of shorter response times and higher throughput.

Thank you for reading thus far!

You can subscribe to new Engineering Ideas via RSS (e. g. using Feedly) or e-mail.

Engineering Ideas #8

Roman Leventov — Fri, 06 Mar 2020 09:00:40 +0000

ORM dilemma, the curse of Lisp, coping with the complexity of systems, leadership of ICs, software development top mistakes, productivity = prioritisation

ORM Hate

Martin Fowler warns against swinging from using an existing ORM fully back to rolling your own thing:

For much of the 90's I saw project after project deal with the object/relational mapping problem by writing their own framework - it was always much tougher than people imagined. Usually, you'd get enough early success to commit deeply to the framework and only after a while did you realize you were in a quagmire - this is where I sympathize greatly with Ted Neward's famous quote that object-relational mapping is the Vietnam of Computer Science.

A conclusion from the referenced Ted Neward’s writeup:

Just as it’s conceivable that the US could have achieved some measure of “success” in Vietnam had it kept to a clear strategy and understood a more clear relationship between commitment and results (ROI, if you will), it’s conceivable that the object/relational problem can be “won” through careful and judicious application of a strategy that is clearly aware of its own limitations.

Fowler suggests the moderate approach:

This is where the criticism comes that ORM is a leaky abstraction. This is true but isn't necessarily a reason to avoid them. Mapping to a relational database involves lots of repetitive, boilerplate code. A framework that allows me to avoid 80% of that is worthwhile even if it is only 80%. The problem is in me for pretending it's 100% when it isn't. David Heinemeier Hansson, of Active Record fame, has always argued that if you are writing an application backed by a relational database you should damn well know how a relational database works. Active Record is designed with that in mind, it takes care of boring stuff, but provides manholes so you can get down with the SQL when you have to. That's a far better approach to thinking about the role an ORM should play.

On Scaling Mental Models

Hillel Wayne comes with an interesting idea about why codebases in “cool” programming languages don’t scale:

Programming is an expression of how we think. We take the basic instructions and build abstractions over them. The more power you have in building abstractions, the more you can tailor those abstractions to your exact needs and exactly how ** _ **you think _ . […]

This compounds. If you’re working by yourself, you can shape your code and environment to reflect your mental model. This makes it easy to quickly write terse, simple, maintainable code. But it’s hard for other people to work with you. They don’t share your mental model , and they don’t come in with all your initial assumptions. This is somewhat addressable if you all start working on the project together but falls apart when people join on later.

I cannot say I fully agree with this as an explanation of why technology written in these languages gets rebuilt with more mundane languages later (Wayne gives an example of Reddit website being rewritten from Lisp to Python).

There exist huge codebases in Scala with hundreds or even thousands of engineers working on them (Twitter), by all accounts is a very powerful language which allows shaping very advanced abstractions.
The reasons for rewrites from Lisp might be just its poor runtime performance or that it’s hard to find engineers to write in it. These may be related to the “expressivity” factor, but nevertheless are different points. There may be problems with scaling Lisp in production, but they may be more prosaic than “hard to scale mental models”, which may actually be a non-problem.

STELLA: report from the SNAFU-catchers workshop on coping with complexity

By David Woods, summary by Adrian Colyer. (SNAFU stands for “Situation Normal: All Fucked Up”, i. e. an accident, a disruption.)

On the ability to cope with complexity and why one should be really careful adding that to a system:

Woods’ Theorem: As the complexity of a system increases, the accuracy of any single agent’s own model of that system decreases rapidly.

Remember that ‘agent’ here could be a human – trying to form a sufficient mental model to triage an incident for example – or a software agent.

On similarities between failures in complex systems (Sidney Dekker writes almost the same in “Drift into Failure”):

Section 3.4.1 in the paper lists some interesting features that all the anomalies had in common, including:

Arising from unanticipated interactions between system components

Having no single ‘root cause’ (during my holiday reading I discovered by accident that Tolstoy has a lot to say on this topic in ‘War and Peace’ – try reading the first couple of pages from Vol III, Part I, Chapter I for example!).

Having been lurking in the system as ‘accidents waiting to happen’ for some time

Being triggered by only slight differences from normal operating conditions

Intially being buffered by system mechanisms designed to cope with partial failures etc., but eventually exhausting those mechanisms

How Can I Prepare to Eventually Move into Engineering Management?

Another great post by Gergely Orosz, providing a mental map about how to transition into management, or, I would say, grow as an individual contributor as well.

It’s hard to cite anything specific, I’d rather suggest you read the whole post. Here are some of the topics:

Lead a project and focus on completing it.
Look for bottlenecks the team is facing that slows work down and fix them.
Pick up ungrateful tasks that no one else wants to do.
Give credit to others, selflessly.
Start mentoring other developers.
Give candid feedback and don't shy away from difficult conversations.
Network with other professionals within or outside the company. Connect with non-engineers, learning about their work.
Observe and learn from your managers.

The last point reminds me of a Babak Nivi’s idea about leverage from here:

Look at who is getting leverage off of the work that you’re doing. Look up the value chain—at who’s above you and who’s above them—and see how they are taking advantage of the time and work you’re doing and how they’re applying leverage.

Software Development’s Classic Mistakes

Construx Software, Steve McConnell’s company, conducted a survey about the most frequent mistakes on software projects. Here are the top 14 most damaging mistakes by exposure (frequency multiplied by severity):

Unrealistic expectations

Overly optimistic schedules

Shortchanged quality assurance

Wishful thinking

Confusing estimates with targets

Excessive multi-tasking

Feature creep

Noisy, crowded offices

Abandoning planning under pressure

Insufficient risk management

Heroics

Shortchanged upstream activities

Inadequate design

Lack of user involvement

The common themes that I see here:

Wishful thinking : mistakes 1, 2, 4, 5, and 10.
Underprioritized planning and quality : mistakes 2, 3, 5, 9, 10, 11, 12, and 13.
Requirements and product mistakes : 1, 7, and 14.
Developers unable to focus : mistakes 6 and 8; cf. “Deep Work” by Cal Newport.

The white paper includes more elaborate descriptions of mistakes which are also interesting.

On our to-do list: an interview with Google’s productivity expert

Laura Mae Martin goes straight to the point:

— What’s one thing people should start doing to manage their workload more efficiently?

— Determine your top priorities for the quarter, and write them on a note on your desk. If you’re asked to do something that doesn’t align with one of those priorities, say no. The more you say no, the more chances you have to say yes to something that really matters.

Thank you for reading thus far!

You can subscribe to new Engineering Ideas via RSS (e. g. using Feedly) or e-mail.

Drift Into Failure by Sidney Dekker – notes on the book

Roman Leventov — Fri, 28 Feb 2020 14:55:49 +0000

I decided to read “Drift Into Failure” by Sidney Dekker because of an appraisal by Lorin Hochstein (thanks to him for curating his reading notes). I thought I would learn something about software systems reliability from this book. Apart from that, the book actually gave me more: it introduced me to a number of fascinating philosophical ideas and the complexity and systems theory with which I was unfamiliar before.

I highly recommend this book , despite it is a bit self-repetitive. It might be that you can learn the same concepts from some other books or papers which are shorter than “Drift Into Failure”, including some of the newer Dekker’s books.

Below are my notes on the book. All citations are from the book, © Sidney Dekker.

Drift into failure (DIF) is a gradual, incremental decline into disaster driven by environmental pressure, unruly technology, and social processes that normalize growing risk. It's an inevitable byproduct of normally functioning processes.

Competition and scarcity dictate that organisation borrows more and more from near the safety boundaries, takes greater risks.

The traditional model claims that for accidents to happen something must break, give, malfunction: component, part, person. But organisations drift into failure precisely because they are doing well.

Cartesian-Newtonian epistemology = rationalism, reductionism, belief that there is the best (optimal) decision, solution, theory, explanation of events.

Optimal, rational decision-maker:

Completely informed about alternatives, the consequences of decisions, probability of events.
Capable of full objective logical analysis.
Sees the finest differences between alternatives.
Rationally ranks alternatives based on the priorities.
(Doesn’t exist :)

Before the 1970s, accidents were seen as truly accidental or sent by god. After—as failures in risk management or the result of deliberate amoral choices or neglect. We also connect the moral gravity to the outcomes of the accident.

The growth of complexity in our society has got ahead of our understanding of how complex systems (CS) work and fail.

Simplicity and linearity remain the defining characteristic of the stories and theories that we use to explain bad events that emerge from the complexity of the world.

In complex systems, we can predict only probabilities, not results. (In Antifragile, Nassim Taleb argues that we cannot estimate narrow probabilities with any good precision, too.)

Complexity is not designed. It happens or grows (against our will).

In 2008 debt crisis, the absence of liability of asset managers skewed the incentives from loan quality to loan quality. More was always better.

Knowledge is inherently subjective. Direct, unmediated, objective knowledge of how a whole complex system works is impossible to get.

Local rationality principle (LRP): people are doing what makes sense to them given situational indictions, organisational pressures, and operational norms that exist at the time. Rational decision making, however, requires massive cognitive resources and all the time in the world.

In ambiguity and uncertainty of complex systems, options that appear to work are better than perfect options that never get computed. However, decisions that look nice locally can fail globally.

In complex systems, local actions can have global results.

Five characteristics of drift

DIF is caused by resource scarcity and competition.
DIF occurs in small steps. In CS, past success is not a guarantee of future success or safety of the same operation.
CSs are sensitive to small changes in input conditions, early decisions. The potential to DIF may be baked in a very small event when the system was much simpler.
CSs that can drift into failure are characterized by unruly (unproven, delusional) technology that creates uncertainty.
Protective (regulatory) structure between CS and the environment which is supposed to protect CS from failure may contribute to DIF because it is subject to the same pressures as the operator.

Philosophy of complex systems

It takes more than one error to push a CS into failure. Theories explaining failure by mistake or broken component (Cartesian-Newtonian) are overapplied and overdeveloped.

Complexity doesn't allow to think in unidirectional terms along which “progress” or “regress” could be plotted.

Though DIF invites the idea of regress: smaller margins, less adaptive capacity, declining standards, CS always simply adapts to whatever happens now or just happened. Any larger historical story is ours, not the system's.

DIF is a construct in retrospect, or, at the most, it is a construct that can be applied to a CS from the outside. From the inside, drift is invisible.

Evolution has no visible direction when you are in the middle of it. Evolution, progress, regress in CSs (including society) are perhaps just illusion, our phycological imposition.

The idea that there is a vector of evolution, a “hand” behind it, a cause for its speed and direction is Newtonian. (Vectors of forces.) (The “hand” also made me think about the connection of this idea to religion.)

Exegesis is an assumption that the essence of the story is already in the story, ready formed, waiting to be discovered. All we need to do is to read the story well, apply the right method, use the correct analytic tools, put things in context, and the essence will be revealed to us. Such structuralism sees the truth as being within or behind the text.

Eisegesis is reading something into a story. Any directions in adaptations are of our own making. Post-structuralism stresses the relationship between the reader and the text as the engine of truth. Reading is not passive consumption of what is already there, provided by somebody who possessed the truth, passing it on. Reading is a creative act. Readers generate meanings out of their own experience and history with text.

Nietzsche, and post-structuralism in general, don't believe that it is a single world that we are interpreting differently. That we could in principle reach agreement when we put all different pictures together. More perspectives don't mean a greater representation of some underlying truth.

In CSs, more perspectives mean more conflicts. CSs can never be understood or exhaustively described. If they could, they would either be not complex, or the entity understanding them would have to be as complex as the whole system. This contradicts the local rationality principle.

Events that we want to study, just like any CS, can never be fixed, tied down, circumscribed with a conclusive perimeter telling us what is part of it and what is not.

By reading drift into a particular failure we will probably learn some interesting things, but we will surely miss or misconstrue other things.

Whether there is a drift into failure depends not on what in the story, but on us, what we bring into the story, how far we read, how deeply, what else we read. It depends on the assumptions and expectations that we have about knowledge, cause, and, ultimately, about morality.

Newton’s and Descartes’ model, like any model, prevents us from seeing the world in other ways, from learning things about the world that we didn't even know how to ask.

Galileo Galilei insisted that we should not worry too much about the things that we cannot quantity, like how something tastes or smells, because he thought that is a subjective mental projection and not a material property of the world that can be measured, and, since you cannot measure it, it's not very scientific. That legacy lives with us today. An obsession with metrics and quantifications and downplaying the things that not only help determine the outcome of many phenomena, like our values or feelings, motives, intentions, but also make us human.

Francis Bacon: “Science should torture nature’s secrets from her. Nature has to be hounded in her wanderings, bound into service, put into constraint and made a slave.”

Analysis of failure

Take the context surrounding component failures seriously. Look for sources of trouble in the organisational, administrative, and regulatory levels of the system, not just the operational or engineering sharp end.

Systems thinking is about

— relationships, not parts

— the complexity of the whole, not the simplicity of the carved out bits

— nonlinearity and dynamics, not linear cause-effect

— accidents that are more than a sum of broken parts

— understanding how accidents can happen when no parts are broken, or no parts are seen as broken.

The work in complex systems is bounded by three types of constraints:

— economic boundary , beyond which the system cannot sustain itself financially.

— workload boundary : people or technology not able to perform the task.

— safety boundary beyond which the system will functionally fail.

Finance departments and competition push organisations into workload/safety boundary corner.

In making tradeoffs for efficiency there is a feedback imbalance. Information on whether a tradeoff is cost-efficient is easy to get. How much was borrowed from safety to achieve this benefit is much harder to quantify.

Operation rules must not be set in stone. They should emerge from production experience and data.

In practice, practices don’t follow the rules. Rules follow emerging practices.

Declaring that CS has a safety hole doesn’t help to find when did the hole occur and why.

The banality of accidents: incidents don’t precede accidents. Normal work does. Accidents result from a combination of factors none of which in isolation cause an accident. These combinations are not usually taken into account in the safety analysis. For this reason, reporting is very ineffective at predicting major disasters.

The etiology of accidents is fundamentally different from that of incidents: hidden in residual risks of doing normal work under pressure of scarcity and competition.

The signals now seen as worthy of reporting, or organisational decisions now seen as bad (even though they looked good at the time) are not big, risky events or order-of-magnitude steps. Rather, there is a succession of weak signals and decisions along a steady progression of decremental steps. Each has an empirical success and no obvious sacrifice to safety.

We need to raise awareness through a language that enables a more functional account. A story of living processes run by people with real jobs and real constraints but no ill intentions, whose understanding co-evolve with the system and the environmental conditions as they try to stay up-to-date with them.

Looking for organisational, latent causes of incidents is still very Newtonian (cause-effect): broken parts, failures of higher management.

Newtonian idea: data (metrics) help to understand something real, and some data may be better than other data.

More redundancy (which itself comes with a fixation on parts) creates greater opacity and complexity. The system can become more difficult to manage and understand.

Individualism was in part fueled by Cartesian dualistic phycology. It’s unnatural in individualistic worldview to think that it may take a team, an organisation, or an entire industry to be responsible for a failure.

The conceptualisation of risk/failure as build-up and release of energy (physical metaphor) is not necessarily well-suited to explain the organisational and sociotechnical factors behind system breakdown, nor equip with a language that can meaningfully handle processes of gradual adaptation or the social processes of risk management and human decision making.

High-reliability theory

Four components of a high-reliability organisation (HRO) (according to high-reliability theory):

— Leadership commitment

— Redundancy and layers of defence

— Entrusting people at the sharp edge to do what they think is needed for safety

— Incremental organisational learning through trial and error, analysis, reflection, simulation. Should not be organised centrally, can (should) be distributed locally as well as decision making.

HRO needs to keep expecting a surprise. The past should not grow a belief in sustaining future safety. Tactics:

— Preoccupation with failure

— Reluctance to simplify. Paying close attention to context and contingencies of events. More differentiation, mindsets, worldviews. Diversity of interpretation helps to anticipate what might go wrong in the future.

— Sensitivity to actual operations, rather than predefined procedures and conclusions.

— Deference to expertise. Listen to the minority opinion.

— Commitment to resilience

Deference to expertise might be still slippery. Experts cannot ultimately escape the push towards margins in their expert judgement based on an incomplete understanding of the data. From within an organisation, it might be impossible to transcend the routine and political stuff and notice an emerging problem. What we don’t believe we cannot see. Thus, relying on human rationality is unreasonable.

Conflicting goals are not an exception, they are a rule. They are the essence of most operational systems. Systems don’t exist to be safe, they exist to provide a service.

Most important goal conflicts are not explicit, they emerge from multiple, irreconcilable directives from different levels and sources, saddle and tacit pressures, from management’s or customers’ reactions to particular tradeoffs.

Practitioners see their ability to reconcile the irreconcilable as a source of professional pride and a sign of their expertise.

Normalisation of deviance is the mechanism of how production pressures incubate failures.

Redundancies, the presence of extraordinary competence, or the use of proven technology can all add to the impression that nothing could go wrong.

A solution to risk, if any, is for an organisation to continually reflect on and challenging its own definition of “normal”. To prioritize chronic safety concerns over acute production pressures.

Large organisations cannot act as closed, rational mechanisms. They themselves have local rationality precisely because they are made of people put together. Problems are ill-defined, often unrecognised. Decision-makers have limited information, shifting allegiances, and uncertain intentions. Solutions may be laying around actively searching for problems to attach themselves to.

According to control theory, degradation of safety control structures can be due to asynchronous evolution: one part of the system changes without related charges in the other parts.

Changes to subsystems may be carefully planned, but their role in overall safety or effect on each other may be neglected or viewed inadequately.

Systems thinking

Mechanistic thinking about failures means going down and into individual “broken” components. Systems thinking means going up and out: understanding comes from seeing how the system is configured in a larger network of systems, tracing the relationships with those, and how those spread out to affect and be affected by factors laying far away in time and space from things went wrong.

At the subatomic level, the interrelations and interactions between the parts of the whole are more fundamental than the parts themselves. There is motion, but, ultimately, no moving objects. There is activity, but no actors.

In complex systems, just like in physics, there are no definite predictions, only probabilities.

Ironically, as natural and social sciences both shift towards complexity and systems thinking, they become closer again. Natural sciences become “softer”, with an emphasis on unpredictability, relationships, irreducibility, non-linearity, time irreversibility, adaptivity, self-organisation emergence, all sort of things that have always been better suited to capture the social order.

Systems thinking predates the scientific revolution that gave birth to the Newtonian-Cartesian paradigm. Leonardo da Vinci was the first systems thinker and complexity theorist. He was a relentless empiricist and inventor, a humanist fusion of art and science. He embraced the profound interconnectedness of ideas from multiple fields. His goal was to combine, advance, investigate, and understand processes in the natural world through the interdisciplinary view.

Complexity and systems theory

Foundations of the complexity and systems theory:

Complex system is open, affecting and affected by the environment
Each component is ignorant of the behaviour of the system as a whole and doesn’t know the full effect of its actions. Components respond locally to the information presented to them there and then. Complexity arises from the huge web of relations between components and their local actions.
Complexity is a feature of the system, not the components.
CS operates far from equilibrium. Components need to get inputs constantly to keep functioning. Without it, CS won’t survive in a changing environment. The performance of a CS is typically optimised at the edge of chaos, just before the system’s behaviour can become unrecognisably turbulent.
CS has a history, a path dependence. The past is co-responsible for its present behaviour. Descriptions of complexity have to take history into account. Synchronous snapshot never represents the CS, only the full diachronic lineage can.
Interactions in CS are non-linear. Small events can produce large results.

Complicated systems consist of a lot of parts and interactions but are closed, unlike complex systems which are open.

Paradox: complexity should support resilience because CS can adapt to the environment and survive. How can complexity lead to failure? Complexity opens a path to a particular kind of brittleness:

Drift into failure cannot be seen synchronically. Only a diachronic study can reveal where the system is headed.
Non-linearity can amplify small events into a failure.
Local reactions and reasoning support decremental steps toward failure through normalisation of deviance.
Unruly technology introduces and sustains uncertainties about how and when things may fail.

Technology can be a source of complexity. Even if parts can be modelled exhaustively (thus are merely complicated), their operation with each other in a dynamic environment generates unpredictabilities and unruliness of complexity.

Emergence

Emergence: a system has an emergent property if what it produces can not be explained by the properties of the parts.

Organized complexity is emergent, bottom-up, a result of local interactions and a cumulative effect of simple rules.

Accidents are emergent properties of CS.

Beware of renaming things that renegotiates the perceived risk down from what it was before, because it may have far-reaching consequences (phase transition).

Exploration-exploitation tradeoff

The optimum (maximum) behaviour is determined in the relationship between the CS and the environment.

A tradeoff between exploration and exploitation (by local agents) lies behind much of the system’s adaptation. This is also why adaptation is sometimes unsuccessful.

Exploration can produce big failures (e. g. trying a new technology in production can produce an outage because the technology fails under the load). This is the reason why the optimum balance between the exploration and exploitation puts the system near the edge of chaos.

Arriving at the edge of chaos is the logical endpoint of DIF. The system has tuned itself into maximum capability.

In a CS, an action controls almost nothing but influences almost everything.

Diversity

Diversity is critical for resilience. One way to ensure diversity is to push authority about safety decisions down to people closest to the problem.

High-reliability theory also recommends attuning to minority opinions, which helps diversity.

Greater diversity of agents in a CS leads to richer emergent patterns (including for good qualities). It typically produces greater adaptive capacity.

=> Diversity is good for an organisation.

Diversity of people will likely enhance creativity on a team. More stories get told about the things we should be afraid of. More hypothesis might be generated about how to improve the system.

Diversity of opinion in a CS should be seen as a strength, not a weakness. The more angles and perspectives, the more opportunities to learn.

Tactics to manage the complexity of drift (“drift into success”)

1. Resource/safety tradeoff:

Full optimization of CS is undesirable because the lack of slack and margins can turn small perturbations into large events. Diversity of opinion helps to prevent exhaustive optimization.

2. Small steps offer two levers for preventing drift:

Small steps happen all the time, providing a lot of opportunity for reflection. What this step can mean for safety? Ask people in different parts of the org (since the step can reverberate to there).
Small steps don’t generate defensive posturing when they are questioned, and cheap to rollback.

Hiring people from outside provides an opportunity to calibrate the practices and correct deviance. Rotation between teams also works. An additional benefit of people rotation is constant ongoing training, including of the safety procedures.

3. Unruly technology:

We may have traditionally seen unruly technology as a pest that needs further tweaking before it becomes optimal. This is a Newtonian commitment. Complexity theory suggests that we see unruly technology not as a partially finished product but as a tool for discovery. Turn small steps into experiments. If results surprise us, it’s an opportunity to learn from about the system, or about ourselves.

Why did this happen?
Why are we surprised that this happened?

But consider the post-structural view, and that the insights that we could possibly have are limited by our local rationality and prior experience. This is where diversity matters, again.

4. Protective structure:

The idea of safety “oversight” is problematic. Oversight implies a “big picture”. A big picture in the sense of a complete description is impossible to achieve over a CS. CS evolve and adapt all the time. Nailing its description down at any one moment means very little to how it will look in the next moment.

If oversight implies sensitivity to the features of complexity and drift, it might work. Oversight may try to explore complex properties:

Interconnectedness, interdependence: less is better
Diversity, rates of learning: more is better
Selection (e. g. for groupthink culture)

Protective structure should adopt the co-evolving or counter-evolving mindset. When inspecting parts, it should relentlessly follow dependencies and connections and explore interactions.

Rather than working on the safety strategy or the accident prevention program, we should create preconditions that can give rise to innovation, to new emergent orders and ideas, without necessarily our own careful attending and crafting.

There are thousands of small steps that can lead to failure. No prevention strategy tinkered by one designer can foresee them all.

Complexity and accountability

In a CS, it’s impossible to determine whose view is right and whose view is wrong, since the agents effectively live in different environments.

Blaming someone, beware the limited knowledge and selection bias and post-structural nature of this act.

There may almost always be several authentic stories of an incident.

Complexity theory doesn’t provide the answer about who is responsible for DIF, but it at least dispels the belief that there exists an easy answer.

Thanks for reading thus far!

You can subscribe to new Engineering Ideas via RSS (e. g. using Feedly) or e-mail.

Single Responsibility Principle Unpacked

Roman Leventov — Tue, 25 Feb 2020 18:00:00 +0000

This article explains the Single Responsibility Principle (SRP): what does it practically mean? And when and how should we apply it?

What does the Single Responsibility Principle say?
- Types of responsibilities
- How small or large should a responsibility be?
The impact of the Single Responsibility Principle on different software qualities
- Understandability and Learning Curve
- Flexibility
- Reusability
- Testability
- Debuggability
- Observability and Operability
- Reliability
- Code Size
- Performance
Summary

What Does the Single Responsibility Principle Say?

The Single Responsibility Principle may feel a bit vague at first. Let's try to deconstruct it and look at what it actually means.

The Single Responsibility Principle applies to the software that we develop on different levels: methods, classes, modules, and services (collectively, I'll call all these things components later in this article). So, the SRP states that each component should have a single responsibility.

This phrase is a little more concrete, but it still doesn't explain what a responsibility is and how small or large a responsibility should be for each particular method, class, module, or service.

Types Of Responsibilities

Instead of defining a responsibility in abstract terms, it may be more intuitive to list the actual types of responsibilities. Here are some examples (they are derived from Adam Warski's classification of objects in applications which he distilled in his thought-provoking post about dependency injection in Scala):

Business Logic

For example, extracting a phone number from text, converting an XML document into JSON, or classifying a money transaction as fraud. On the level of classes and above, a business logic responsibility is knowing how to do (or encapsulating) the business function: for example, a class knowing how to convert XML documents into JSON, or a service encapsulating the detection of fraud transactions.

External Integration

On the lowest level, this can be an integration between modules within the application, such as putting a message into a queue which is processed by another subsystem. Then, there are integrations with the system, such as logging or checking the system time (System.currentTimeMillis()). Finally, there are integrations with external systems, such as database transactions, reading from or writing to a distributed message queue such as Kafka, or RPC calls to other services.

On the level of classes, modules, and services, an external integration responsibility is knowing how to integrate (or encapsulating integration with) the external part: for example, a class knowing how to read the system time (which is exactly what java.time.Clock is), or a service encapsulating talking with an external API.

Data

A profile of a person on a website, a JSON document, a message. Embodying a piece of data can only be a responsibility of a class (object), but not of a method, module, or service. A specific kind of data is configuration: a collection of parameters for some other method, class, or system.

Control Flow

A piece of an application's control flow, execution, or data flow. An example of this responsibility is a method that orchestrates calls to components that each have other responsibilities:

void processTransaction(Transaction t) {
  if (isFraud(t)) { // Business logic
    // External integration: logging
    logger.log("Detected fraud transaction {}", t);
    // Integration with external service
    alertingService.sendAlert(new FraudTransactionAlert(t));
  }
}

On the level of classes, an example of a data flow responsibility may be a BufferedLogger class which buffers logging statements in memory and manages a separate background thread that takes statements from the buffer and writes them to actual external logger:

class BufferedLogger implements Logger {
  private final Logger delegate;
  private final ExecutorService backgroundWorker;
  private final BlockingQueue<Statement> buffer;

  BufferedLogger(Logger delegate) {
    this.delegate = delegate;
    this.backgroundWorker = newSingleThreadExecutor();
    this.buffer = new ArrayBlockingQueue<>(100);
    backgroundWorker.execute(this::writeStatementsInBackground);
  }

  @Override public void log(Statement s) {
    putUninterruptibly(buffer, s);
  }

  private void writeStatementsInBackground() {
    while (true) {
      Statement s = takeUninterruptibly(buffer);
      delegate.log(s);
    }
  }
}

Method writeStatementsInBackground() itself has a control flow responsibility.

In a distributed system, examples of services with a control or data flow responsibility could be a proxy, a load balancer, or a service transparently caching responses from or buffering requests to some other service.

How Small or Large Should a Responsibility Be?

I hope the examples above give some more grounded sense of what responsibility of a method, class, module, or service could be. However, they still provide no actionable guidance on how finely we should chop responsibilities between the components of your system. For example:

Should conversion from XML to JSON be a responsibility of a single method (or a class), or should we split it between two methods? One translates XML into a tree, and another serializes a tree into JSON? Or should these be separate methods belonging to a single class?
Should individual types of interactions with an external service (such as different RPC operations) be responsibilities of separate classes, or should they all belong to a single class? Or, perhaps, should interactions be grouped, such as read operations going to one class and write operations going to another?
How should we split responsibilities across (micro)services?

Uncle Bob Martin (who first proposed the Single Responsibility Principle) suggests that components should be broken down until each one has only one reason to change. However, to me, this criterion still doesn't feel very instructive. Consider the processTransaction method above. There may be many reasons to change it:

Increment counters of normal and fraudulent transactions to gather statistics.
Enrich or reformat the logging statement.
Wrap sending an alert into error-handling try-catch and log a failure to send an alert.

Does this mean that the processTransaction() method is too large and we should split it further into smaller methods? According to Uncle Bob, we probably should, but many other people may think that processTransaction is already small enough.

Let's return to the purpose of using the Single Responsibility Principle. Obviously, it's to improve the overall quality of the codebase and of its production behavior (Carlo Pescio calls these two domains artifact space and runtime space, respectively).

So, what will ultimately help us to apply the Single Responsibility Principle effectively is making clearer for ourselves how SRP affects the quality of the code and the running application. The optimal scope of the responsibility for a component highly depends on the context:

The responsibility itself (i. e. what the component actually does)
The non-functional requirements to the application or the component we're developing
How long we plan to support the code in the future
How many people will work with this code
Etc.

However, this shouldn't intimidate us. We should just split (or merge) components while we see that the software qualities we're interested in keep improving.

Thus, the next step is to analyze how the Single Responsibility Principle affects the specific software qualities.

The Impact Of the Single Responsibility Principle On Different Software Qualities

Understandability and Learning Curve

When we split responsibilities between smaller methods and classes, usually the system becomes easier to learn overall. We can learn bite-sized components one at a time, iteratively. When we jump into a new codebase, we can learn fine-grained components as we need them, ignoring the internals of the other components which are not yet relevant for us.

If you have ever worked with code in which the Single Responsibility Principle was not regarded much, you probably remember the frustration when you stumble upon a three-hundred-line method or a thousand-line class about which you need to understand something (probably a little thing), but to figure that out, you are forced to read through the whole method or the class. This not only takes a lot of time and mental energy, but also fills the "memory cache" of your brain with junk information that is completely irrelevant at the moment.

However, it's possible to take the separation of concerns so far that it might actually become harder to understand the logic. Returning to the processTransaction() example, consider the following way of implementing it:

class TransactionProcessor {
  private final TransactionInstrumentation instrumentation;

  ...

  void processTransaction(Transaction t) {
    if (isFraud(t)) {
      instrumentation.detectedFraud(t);
    }
  }
}

class TransactionInstrumentation {
  private final Logger logger;
  private final AlertingService alertingService;

  ...

  void detectedFraud(Transaction t) {
    logger.log("Detected fraud transaction {}", t);
    alertingService.sendAlert(new FraudTransactionAlert(t));
  }
}

We extracted the observation part of the logic into a separate TransactionInstrumentation class. This approach is not unreasonable. Compared to the original version, it aids the flexibility and the testability of the code, as we will discuss below in this article. (In fact, I took the idea directly from the excellent article about domain-oriented observability by Pete Hodgson.)

On the other hand, it smears the logic so thin across multiple classes and methods that it would take longer to learn it than the original, at least for me.

Extracting responsibilities into separate modules or services (rather than just classes) doesn't help to further improve understandability per se, however, it may help with other qualities related the learning curve: the discoverability of the functionality (for example, through service API discovery) and the observability of the system, which we will discuss below.

Understandability itself is somewhat less important when we work on the code alone, rather than in a team. But don't abuse this - we tend to underestimate how quickly we forget the details of the code on which we worked just a little while ago and how hard it is to relearn its purpose :).

Flexibility

We can easily combine independent components (via separate control flow components) in different ways for different purposes or depending on configuration. Let's take TransactionProcessor again:

class TransactionProcessor {
  private final AlertingService alertingService;

  ...

  void processTransaction(Transaction t) {
    if (isFraud(t)) {
      logger.log("Detected fraud transaction {}", t);
      alertingService.sendAlert(new FraudTransactionAlert(t));
    }
  }

  private boolean isFraud(Transaction t) { ... }
}

To allow the operators of the system to disable alerting, we can create a NoOpAlertingService and make it configurable for TransactionProcessor via dependency injection. On the other hand, if the sendAlert() responsibility was not separated into the AlertingService interface, but rather was just a method in TransactionProcessor, to make alerting configurable we would have to add a boolean field sendAlerts to the class.

Imagine now that we want to analyze historical transactions in a batch process. Since the isFraud() method (that is, the fraud detection responsibility) is a part of TransactionProcessor, this method is called during batch processing. If online and batch processing require different initialization logic, TransactionProcessor has to provide a different constructor for each use case. On the other hand, if fraud detection was a concern of a separate FraudDetection class, we could prevent TransactionProcessor from swelling.

We can notice a pattern: it's still possible to support different use cases and configuration for a component with multiple responsibilities, but only by increasing the size and the complexity of the component itself, like adding flags and conditional logic. Little by little, this is how big ball of mud systems (aka monoliths) and runaway methods and classes emerge. When each component has a single responsibility, we can keep the complexity of any single one of them limited.

What about the "lean" approach of splitting responsibilities only when we actually need to make them configurable? I think this is a good strategy if applied with moderation. It is similar to Martin Fowler's idea of preparatory refactoring. Keep in mind, however, that if we don't keep responsibilities separate from early on, the code for them may grow to have many subtle interdependencies, so it might take much more effort to split them apart further down the road. And to do this, we might also need to spend time relearning the workings of the code in more detail than we would like to.

Reusability

It becomes possible to reuse components when they have a single, narrow responsibility. The FraudDetection class from the previous section is an example of this: we could reuse it in online processing and batch processing components. To do this in the artifact space, we could pull it into a shared library. Another direction is to move fraud detection into a separate microservice: we can think about this as reusability in the runtime space. The FraudDetection class within our application will then turn from having business logic responsibility to do external integration with the new service.

Most methods with a narrow responsibility shouldn't have side effects and shouldn't depend on the state of the class, which enables sharing and calling them from any place. In other words, the Single Responsibility Principle nudges us toward a functional programming style.

Pro tip: thinking about responsibilities helps to notice unrelated subproblems hiding in our methods and classes. When we extract them, we can then see opportunities to reuse them in other places. Moving unrelated subproblems out of the way keeps a component at a single level of abstraction, which makes easier to understand the logic of the component.

Testability

It's easier to write and maintain tests for methods and classes with focused, independent concerns. This is what the Humble Object pattern is all about. Let's continue playing with TransactionProcessor:

class TransactionProcessor {

  void processTransaction(Transaction t) {
    boolean isFraud;
    // Some logic detecting that the transaction is fraud,
    // many lines of code omitted
    ...

    if (isFraud) {
      logger.log("Detected fraud transaction {}", t);
      alertingService.sendAlert(new FraudTransactionAlert(t));
    }
  }
}

In this variant, there is no separate isFraud() method. processTransaction() combines fraud detection and the reporting logic.

Then, to test the fraud detection, we may need to mock the alertingService, which pollutes the test code with boilerplate. Not only does it take effort to set up mocks in the first place, but mock-based tests also tend to break every time we change anything in the production code. Such tests become a permanent maintenance burden.

Alternatively, to test the fraud detection logic in the example above, we could intercept and check the logging output. However, this is also cumbersome, and it hinders the ability to execute tests in parallel.

It's simpler to test a separate isFraud() method. But we would still need to construct a TransactionProcessor object and to pass some dummy Logger and AlertingService objects into it.

So, it's even easier to test the variant with the separate FraudDetection class. Notice that to test the intermediate version without a separate FraudDetection class, we often find ourselves changing the visibility of the method under test (isFraud(), in this example) from private to default (package-private).

Changing visibility of a method and the @VisibleForTesting annotation are clues to think about whether it's better to split the responsibilities of the enclosing class.

Pete Hodgson also explains how extracting observability like the alerting feature into a separate class (like TransactionInstrumentation) enables clearer, more focused tests.

In contrast to methods and classes, smaller (micro)services complicate the local setup for integration testing. Docker Compose is a godsend, but it doesn't solve the problem fully.

Debuggability

When methods and classes are focused on a single concern, we can write equally focused tests for them. If tests covering only a single production method or class fail we immediately know where the bug is and thus we don't need to debug. Sometimes, debugging may become a large portion of the development process: for example, Michael Malis reports that for him, it used to take as much as a quarter of the total time.

When we still have to debug, we can accelerate the debugging loop by testing isolated pieces of functionality without building large graphs of objects through dependency injection or spinning up databases in Testcontainers.

However, keep in mind that many bugs are due to one component incorrectly using another. Mistakes happen exactly in the integration of real components. So, it's important to have both narrowly focused unit tests to quickly fix certain types of errors without lengthy debugging, and more integration-like tests to check that components use each other properly.

Observability and Operability

Having methods with single responsibilities also helps to quickly pinpoint performance problems because the results of profiling become more informative. At the top of a profiler's output, we can see the methods that perform badly and will know what exact responsibilities they have.

When components (not only methods and classes, but also modules and distributed services) are connected with queues (either in-memory, in-process Queues, or distributed message brokers such as Kafka), we can easily monitor the sizes of the backlogs in the pipeline. Matt Welsh, the engineer who proposed the staged event-driven architecture (SEDA), regarded this observability of load and resource bottlenecks as the most important contribution of SEDA.

Decoupled services could be scaled up and down independently in response to the changing load, without overuse of resources. Within an application, we can control the distribution of CPU resources between method, class, and module responsibilities by sizing the corresponding thread pools. ThreadPoolExecutor even supports dynamic reconfiguration in runtime via the setCorePoolSize() method.

When microservices have focused responsibilities, it also helps to investigate incidents. If we monitor the request success rates and health status of each service and see that one service which connects to a particular database is failing or unavailable, we may assume that the root problem lies in this database rather than any other part of the system.

However, despite the advantages of finer-grained monitoring and scaling, splitting responsibilities between smaller services generally increases the burden of operating the system. Smaller services mean more work:

Setting up and operating intermediate message queues (like Kafka) between the services.
DevOps: setting up and managing separate delivery pipelines, monitoring, configuration, machine and container images.
Deployment and orchestration: Kubernetes doesn't fully alleviate it.
To ensure rollback safety, the deployments should be multi-phase, shared state and messages sent between services should be versioned.

Reliability

Reliability is the first software quality in the list that we mostly hurt, not aid when we split smaller responsibilities between the components.

If engineered properly (an important caveat!), a microservice architecture can increase reliability: when one service is sick, others can still serve something for the users. However, the inherent fallibility of distributed systems hits harder: more remote communications between services mean more ways of how things could go wrong, including network partition or degradation.

Discussing the pros and the cons of microservices is not the main goal of this article, but there are plenty of good materials on this topic, the reliability aspect in particular: 1, 2, 3.

Code Size

Smaller responsibility of each component means that there are more components in total in the system.

Each method needs a signature declaration. Each class needs constructors, static factory methods, field declarations, and other ceremony. Each module needs a separate configuration class and a dependency injection setup. Each service needs separate configuration files, startup scripts, CI/CD/orchestration infrastructure, and so on.

Therefore, the more focused responsibilities of the components we make, the more code we will need to write. This impacts the long-term maintainability much less than all the qualities discussed above: understandability, flexibility, reusability, etc. However, it means that it takes more time and effort to develop the first version of the application with finely separated responsibilities than with larger components.

Performance

This shouldn't be a concern normally, but for the sake of completeness, we should note that a large number of smaller classes may impact the application startup time. An entry in the Spring blog has a nice chart illustrating this:

Having lots of small methods taxes the application performance through method calls and returns. This is not a problem at hotspots thanks to method inlining, but in applications with a "flat" performance profile (no obvious hotspots), an excessive number of method calls might considerably affect the cumulative throughput.

On a higher level, the size of services might significantly impact the efficiency of the distributed system due to the costs of RPC calls and message serialization.

Summary

The Single Responsibility Principle applies to software components on all levels: methods, classes, modules, and distributed services.

The Single Responsibility Principle itself doesn't include guidance about how large or small a responsibility for a component should be. The optimal size depends on the specific component, the type of the application, the current development priorities, and other context.

We should analyze how making the responsibilities of components smaller or larger affects the qualities of the code and the system that we are developing.

If we are writing proof-of-concept or throwaway code, or the relative cost of time to market/penalty for missing some deadline is super high, it's important to keep in mind that following the Single Responsibility Principle "properly" requires more effort and therefore may delay the delivery time.

In other cases, we should split up responsibilities into separate methods and classes as long as the flexibility, reusability, testability, debuggability, and observability of the software keep improving, and while the code doesn't bloat too much and we still see the "forest" of the logic behind the "trees" of small methods and classes (in more formal language, the understandability of the code doesn't begin to deteriorate).

This may sound overwhelming, but of course, this analysis shouldn't be done for each and every method and class in separation, but instead done infrequently to establish a guideline on the project, or just to train our intuition.

On the level of the distributed system, the trade-off is much less in favor of extracting (micro)services with more narrow responsibilities: discoverability, flexibility, reusability, and observability improve, but testability, operability, reliability, and performance mostly decline. On the other hand, the Single Responsibility Principle probably shouldn't be the first thing to consider when sizing microservices. Most people in the industry think that it's more important to follow the team boundaries, bounded contexts, and aggregates (the last two are concepts from Domain-Driven Design).

P. S.: I explore the idea of analyzing software design practices and principles through the lenses of distinct software qualities such as understandability, testability, performance, and so on in the Software Design project on Wikiversity.

P. P. S.: I publish weekly Engineering Ideas where I share articles and papers about software development, cloud architecture, data engineering, reliability, operations, and team culture which I find insightful.

Engineering Ideas #7

Roman Leventov — Fri, 21 Feb 2020 14:03:08 +0000

Synthesizing data structure transformations from input-output examples

John Feser, Swarat Chaudhuri, Isil Dillig have created a program synthesizer capable of recreating pretty advanced functional algorithms:

We have used λ2 to synthesize a program originally invented by Barron and Strachey. Danvy and Spivey call this program “arrestingly beautiful” and believe it to be “the world’s first functional pearl”.

The program is required to produce the Cartesian product of a set of lists.

Barron and Strachey’s pearl to implement this transformation looks like this:

cprod xss = foldr f [[]] xss where f xs yss = foldr g [] xs where g x zss = foldr h zss yss where h ys qss = cons(cons(x, ys), qss)

Give the examples below, synthesises exactly this program in ~84 seconds.

[] -> [[]][[]] -> [][[], []] -> [][[1,2,3], [5,6]] -> [[1,5],[1,6],[2,5],[2,6],[3,5],[3,6]]

You may say that this is very far from generating useful real-world programs. But think how long would it take you to write the algorithm above yourself? Is it less than 84 seconds? I think the machine power to apply brute force shouldn’t be discounted. And it’s easy to imagine the above synthesizer could be used to generate small utility functions in real projects.

There is no escape from the adaptive universe

By Lorin Hochstein

In his 2018 paper titled The theory of graceful extensibility: basic rules that govern adaptive systems, Woods describes the two assumptions of the adaptive universe:

Resources are always finite.

Change is ongoing.

At first glance, the assumptions sound banal. Nobody believes in infinite resources! Nobody believes that things will stop changing! Yet, when we design our systems, it’s remarkable how often we don’t take these into account.

Monolith First

A refreshing perspective from Martin Fowler:

I've noticed a common pattern.

Almost all the successful microservice stories have started with a monolith that got too big and was broken up

Almost all the cases where I've heard of a system that was built as a microservice system from scratch, it has ended up in serious trouble.

This pattern has led many of my colleagues to argue that you shouldn't start a new project with microservices, even if you're sure your application will be big enough to make it worthwhile.

However, he himself acknowledged back in 2015 that this a very nuanced question and the opposite approach, as to start building a brand new system of microservices, might sometimes be a better alternative. At least, we shouldn’t choose it blindly.

The counter-argument says that starting with microservices allows you to get used to the rhythm of developing in a microservice environment. It takes a lot, perhaps too much, discipline to build a monolith in a sufficiently modular way that it can be broken down into microservices easily. By starting with microservices you get everyone used to developing in separate small teams from the beginning, and having teams separated by service boundaries makes it much easier to scale up the development effort when you need to. This is especially viable for system replacements where you have a better chance of coming up with stable-enough boundaries early.

On Pair Programming

This is a practical overview of pair programming setup, benefits, and challenges by Birgitta Böckeler and Nina Siessegger. What I found most evocative though in this article is the conclusion, which echoes some of the ideas from Daniel Coyle’s The Culture Code:

We talked a lot about the benefits of pair programming, but even more about its challenges. Pairing requires a lot of different skills to get it right, and might even influence other processes in the team. So why bother? Is it really worth the hassle?

For a team to be comfortable with and successful at pair programming, they will have to work on all the skills helpful to overcome its challenges: Concentration and focus, task organisation, time management, communication, giving and receiving feedback, empathy, vulnerability - and these are actually all skills that help immensely to become a well-functioning, collaborative and effective team. Pairing gives everybody on the team a chance to work on these skills together.

Consider this headline from Harvard Business Review: "Diverse Teams Feel Less Comfortable - and That's Why They Perform Better". The authors are making the point that "Homogeneous teams feel easier - but easy is bad for performance. (...) this idea goes against many people's intuitions".

In this talk, Pia Nilsson describes measures her team at Spotify took to get over the uncomfortable friction initially caused by introducing practices like pair programming. Among other things, she mentions feedback culture, non-violent communication, psychological safety, humility, and having a sense of purpose.

The core values and practices for building software at ThoughtWorks

Evan Bottcher distils the software design and development strategy to four core values:

Fast feedback : unit tests, CI, pair programming
Repeatability : automation, infra as code, containers, data immutability
Simplicity : software design, TDD
Clean code : design, refactoring, collective code ownership

These principles apply to mobile and web development, backend development, data engineering, and Cloud and Dev Ops alike.

A simple answer to whether to practice TDD or not (or any other XP/Agile practice):

So what if you don’t use TDD? Then, I’d be asking what you have in place to ensure your design is simple and you have good, fast feedback. If you can convince me that the thing you’ve replaced TDD with is working (or that you’ll know when it’s not), then I’ll be happy. Same with pair programming, or any others of our core practices.

Don’t Fuck Up the Culture

Brian Chesky (CEO of Airbnb) explains what culture is, in essence:

Culture is simply a shared way of doing something with passion.

And why it is important:

Why is culture so important to a business? Here is a simple way to frame it. The stronger the culture, the less corporate process a company needs. When the culture is strong, you can trust everyone to do the right thing. People can be independent and autonomous. They can be entrepreneurial. And if we have a company that is entrepreneurial in spirit, we will be able to take our next “(wo)man on the moon” leap.

I see parallels here with the idea from Ron Davison’s The Fourth Economy (see summary by Taylor Pearson) that one of the drivers of growth in the 21st century will be entrepreneurship.

You can subscribe to new Engineering Ideas via RSS (e. g. using Feedly) or e-mail.

Engineering Ideas #6

Roman Leventov — Fri, 14 Feb 2020 13:53:13 +0000

Will Project Loom obliterate Java Futures?

Another great post by Adam Warski discussing the tradeoffs of different models of concurrent programming: futures, green threads, coroutines, IO monad, actors.

Warski makes an analogy of RPC to demonstrate that boilerplate around async calls (which Project Loom aims to eliminate) is not necessarily a bad thing:

RPCs, and any network calls in general, have a significantly different characteristic than a normal function call. They are unpredictable: can arbitrarily fail , regardless of the value of input parameters. They can take an arbitrary amount of time to complete, while a normal function completes, fails or loops. And even if a network call seems to have failed , it might have still succeeded from the viewpoint of the other system. […] The fact that a normal function call is also syntactically distinct from an RPC call, might be an advantage to readability.

Martin Thompson brings insight into why Loom Fibers won’t magically solve all performance problems with our concurrent applications:

Martin Thompson

@mjpt777

@pressron @rafaelcodes Physics is teaching that the only way to advance IO operations is asynchronous & batching, IO APIs are evolving to this. Fibers encourage using synchronous imperative code, i.e. old APIs. It is the delusion of RPC all over again.

07:31 AM - 24 Sep 2019

0 4

Warski distills a persuasive strategy about picking up a concurrency model for the task:

When doing a simple service , small application or a quick script, I don’t want to deal with any kind of wrappers, be it Future or IO. Fibers and their “codes like sync, works like async” model will make my life much easier.

When writing a business application , I might want to use the synchronous-like API that Loom Fibers enable to express the business logic, using well-known constructs to express the control flow within a business process. However, to orchestrate concurrently running computations, handle errors and allocate resources, I’ll use an IO-like abstraction.

Finally, for a high-performance asynchronous system, I’ll probably take the fully-asynchronous approach, working with state machines, callbacks or Futures.

I want to contrast the above discourse with a quote from this blog post by Brad Fitzpatrick:

The Go runtime is relatively complex internally but it permits simple APIs and programming models for users who then don't need to worry about memory management, thread management, blocking, the color of their functions, etc.

By not having Futures, indeed, Go avoids a lot of complexity which we have with concurrency in Java.

Avoiding fallback in distributed systems

Jacob Gabrielson shares many non-intuitive observations and actionable tactics for improving reliability.

At Amazon, we avoid fallback in our systems because it’s difficult to prove and its effectiveness is hard to test. Fallback strategies introduce an operational mode that a system enters only in the most chaotic moments where things begin to break, and switching to this mode only increases the chaos.

Pre-allocate:

For critical single-machine applications that must work in case of memory allocation failures, one solution is to pre-allocate all heap memory on startup and never rely on malloc again, even under error conditions.

In “Failure Modes and Continuous Resilience”, Adrian Cockcroft mentions a similar idea in the context of failover to a different availability zone: at a critical moment, the control plane service may fail, too (or appear to not work due to problems in code or configuration, that was not noticed earlier because it is not usually exercised), so instances/databases/networks could instead be pre-allocated in the secondary region in a cold standby fashion.

Replace pull with push:

The IAM service needs to provide signed, rotated credentials to code running on EC2 instances. To avoid ever needing to fall back, the credentials are proactively pushed to every instance and remain valid for many hours. This means that IAM role-related requests keep working in the unlikely event of a disruption in the push mechanism.

Convert fallback into failover, which falls into the pattern of constant work :

A service must run both the fallback and the non-fallback logic continuously. It must not merely run the fallback case but also treat it as an equally valid source of data. For example, a service might randomly choose between the fallback and non-fallback responses (when it gets both back) to make sure they're both working.

Do you monitor retries in your system already?

We maintain metrics that monitor overall retry rates and alarms that alert our teams if retries are happening frequently.

Where to Start

Keavy McMinn on how to start a new project: talk with people. Customers, colleagues, industry peers.

A great question I learnt more recently, while sitting in on a colleague interviewing a customer, was “What question do you wish I’d asked you?”.

My experiences have taught me that if you want to produce the right thing, that has rich and lasting impact, this starting point of finding people, talking to and learning from them is fundamental. For me it’s the precursor before the technical research and experiments can truly begin.

What makes engineers productive?

From an HN comment by Jonathan Tang:

Jeff & Sanjay have a reputation as rockstars because they can apply this productivity to things that really matter; they're able to pick out the really important parts of the problem and then focus their efforts there so that the end result ends up being much more impactful than what the SWE3 wrote. The SWE3 may spend his time writing a bunch of unit tests that catch bugs that wouldn't really have happened anyway, or migrating from one system to another that isn't really a large improvement, or going down an architectural dead-end that'll just have to be rewritten later.

Jeff or Sanjay will spend their time running a proposed API by clients to ensure it meets their needs, or measuring the performance of subsystems so they fully understand their building blocks, or mentally simulating the operation of the system before building it so they rapidly test out alternatives. They don't actually write more code than a junior developer (oftentimes, they write less), but the code they do write gives them more information, which makes them ensure that they write the right code.

… these developers rapidly become 1x developers (or worse) if you don't let them make their own architectural choices - the reason they're excellent in the first place is because they know how to determine if certain work is going to be useless and avoid doing it in the first place.

How to build a PaaS for 1500 engineers

Galo Navarro highlights the most important question every platform infrastructure team should ask themselves:

A Platform team should really avoid competing against AWS, Google, or any commercial company. It doesn’t matter if their homegrown CI system is superior, today, to $commercial_ci_system. The market will catch up, faster than expected, and make that team redundant. Every Platform team should be asking themselves: what is our differentiator? What do we offer than makes it worthwhile for our company to invest in our team, rather than throwing those engineers at the product?

The main value we provide is in the joints, the articulation, the glue. In how we integrate together all these systems. […] We focus on what is specific for our company, tailoring off-the-shelf solutions to our needs.

How I learned to program

Dan Luu on the benefits of having an up-front design phase on project with the invention and implementation phases more challenging than the discovery phase (see Grady Booch’s phases):

Working through a design collaboratively teaches everyone on the team everyone else's tricks. It's a lot like the kind of skill transfer you get with pair programming, but applied to design.

The iteration speed is much faster in the design phase, where throwing away a design just means erasing a whiteboard. Once you start coding, iterating on the design can mean throwing away code; for infrastructure projects, that can easily be person-years or even tens of persons-years of work. […] I've seen teams on projects of similar scope insist on getting "working" code as soon as possible. In every single case, that resulted in massive delays as huge chunks of code had to be re-written , and in a few cases the project was fundamentally flawed in a way that required the team had to start over from scratch.

I get that on product-y projects, where you can't tell how much traction you're going to get from something, you might want to get an MVP out the door and iterate, but for pure infrastructure, it's often possible to predict how useful something will be in the design phase.

He also confirms the importance of design docs, which lines up with the practices at Uber shared by Gergely Orosz:

I noticed a curiously strong correlation between the quality of initial design docs and the success of projects.

You can subscribe to new Engineering Ideas via RSS (e. g. using Feedly) or e-mail.

The Culture Code by Daniel Coyle - notes on the book

Roman Leventov — Fri, 24 Jan 2020 16:39:20 +0000

I’ve listened to Daniel Coyle’s The Culture Code after reading the excerpt which resonated with me greatly. Thanks to Subbu Allamaraju who pointed to this book in his post about Status Management.

The book has changed the way I think about work profoundly: even more than The Effective Executive by Peter Drucker and Principles: Life and Work by Ray Dalio, and comparably to Cal Newport’s Deep Work. I would say “The Culture Code” is a must-read for everyone who works in a team. These “Engineering Ideas” are my notes on the book. Many of them are direct citations from the book, © Daniel Coyle.

Culture is not something you are, it’s something you do.

Safety

Small features of groups with chemistry (belonging queues):

Close physical proximity, often in circles
A profuse amount of eye-contact
Physical touch, handshakes, fist bumps, hugs
Lots of short energetic exchanges, no long speeches
High level of mixing, everyone talks to everyone
Few interruptions
Lots of questions
Intensive, active listening
Humour
Laughter
Small attentive courtesies, thank yous, opening doors

Qualities of belonging queues:

Energy: they invest in the exchange that is occurring.
Individualization: they treat the person as unique and valued.
Future-orientation: they signal the relationship will continue.

Belonging queues translate into the message: “You are safe here”.

Believe firmly in what you are saying if you want others to believe in it.

Team performance is driven by five measurable factors:

Everyone in the group talks and listens in roughly equal measure, keeping contribution short.

Members maintain high levels of eye contact, and their conversations and gestures are energetic.

Members communicate directly with one another, not just with the team leader.

Members carry on back-channel or side conversations within the team.

Members periodically break, go explore outside the team and bring back what they have found to share information with the others.

Words are noise. Group performance is determined by behaviors that translate one powerful and overarching idea: “We are safe and connected”.

Three models in startups:

Star model: hire the best.
Professional model: build groups around specific skillsets.
Commitment model: build a group with shared values and strong emotional bonds. This model leads to a higher rate of success.

Give people a signal that you care about them to boost their motivation.

Thinking about our connections boosts our sense of autonomy.

Belonging needs to be committed and reinforced constantly.

Connect when interviewing to see if you can become friends with these people.

Connect on something that is bigger than the profession with colleagues. Eat together.

Not achieving happiness, but solving hard problems. Look if people are oriented at that.

Magic feedback: “ I’m giving this feedback because I have very high expectations, and I know that you can reach it.” There are three queues in this phrase: “You are part of this group”, “This place is special”, “I believe in you.”

Cooperation and Vulnerability

Look into persons’ faces when they speak. Nod. Ask “What do you mean by that?”, “Could you tell more about this?”

Listen with eyes wide open, still, slightly leaning towards the speaker, eyebrows slightly up, generating a constant stream of affirmations.

It’s important not to interrupt the speaker. Top salespeople hardly ever interrupt others. Distinguish, however, between interruptions from mutual excitement and due to lack of concern.

Spotlight your fallibility early on, especially if you are a leader. Show your mistakes. Invite input with phrases like:

These are just my two cents.
Of course, I could be wrong here.
What am I missing?
What do you think?

Ask questions. Listen keenly. Radiate humility.

Say something emotional, like “I’m terrified of”, to invite a deeper connection because it sparks a response in the listener “How can I help you?”

Proactively invite feedback. It’s hard for people to raise a hand and say “I have something tentative to say.”

Thank people over the top. It ignites cooperation, even with very different persons. Thank the least powerful person in the group.

Ensure everyone has a voice.

Find ways to show equality and do small favors to the group, e. g. pick up trash. These actions send a signal: “We are all here together.”

Pay attention to the day of joining the group. Note this special moment: “We are together, now.”

Separate positive and negative feedback into separate processes.

Handle negative feedback as a dialogue:

Ask if a person wants feedback.
Have a learning focus to a two-way conversation about the needed growth.
Note positives through ultra-clear bursts of recognition and praise.

Embrace fun. Laughter is the most fundamental sign of safety and connection.

Vulnerability doesn’t come after trust, it precedes it.

When it comes to creating cooperation, vulnerability is not a risk, but a phycological requirement.

Leaving yourself wide open, so that everybody knows who you are, if done right, can build the level of trust times higher that you can get any other way.

One person telling other people what to do is not a reliable way to make good decisions.

Try to avoid authority bias. You have to go around those barriers, but they never go away.

When giving your opinion, be careful to attach phrases like “Let’s see if someone can poke holes in this”, “Tell me what’s wrong with this idea?”

Steer away from giving orders, and instead ask a lot of questions. “Anybody have any ideas?”

Tell subordinates explicitly that they must speak up whenever it seems to them that something is wrong with your decision or conduct.

Spending time outside work together helps to avoid authority bias. Hard challenges work even better (like going through an obstacle race under a rain).

Design truth-telling sessions, reflections/retrospectives on past projects.

“I skewed that up” might be the most important thing any leader can say.

Good retrospectives follow a template. Go over the project chronologically and ask a lot of “why” questions.

The goal of retrospectives is to build shared mental models that could be applied in future projects. Eventually, everybody should be able to see what’s really happening, not just a small piece of it.

Make sure people understand how their actions affect others.

“Backbone of humility” is the tone of a good retrospective. The relentless will to see the truth and take ownership.

Combining openness with discipline is not easy, but does pay off.

Real courage is seeing the truth and speaking the truth to each other. Nobody wants to say “Wait a second, what’s really going on here?”

Combine warmth and curiosity. Capture what someone is doing, throw some new ideas at them and ask “Why don’t you try that?”

Make sure the leader is vulnerable first and often.

Deliver the message “I’m scared” with steadiness, confidence, and comfort that underline the deeper message: “It’s safe to tell the truth here.”

Laszlo Bock, head of People Analytics at Google, recommends that leaders ask their people three questions:

– What is one thing that I currently do that you’d like me to continue to do?

– What is one thing that I don’t currently do frequently enough that you think I should do more often?

– What can I do to make you more effective?

“The key is to ask not for five or ten things but just one,” Bock says. “That way it’s easier for people to answer. And when a leader asks for feedback in this way, it makes it safe for the people who work with them to do the same. It can get contagious.”

Overcommunicate expectations. The successful groups I’ve visited did not presume that cooperation would happen on its own. Instead, they were explicit and persistent about sending big, clear signals that established expectations, modeled cooperation, and aligned language and roles to maximize helpful behavior.

The more complex the problem, the more help you need to solve it. Explicitly define helpful roles and generate vulnerabilities. Fill chats with help requests.

Collaborate. Make others successful. Going out of your way to help others is a secret sauce.

Deliver the Negative Stuff in Person

This was an informal rule that I encountered at several groups I researched. It goes like this: if you have negative news or feedback to give someone—even as small as a rejected item on an expense report—you are obligated to deliver that news face to face. This rule is not easy to follow (it’s far more comfortable for both the sender and receiver to communicate electronically), but it works because it deals with tension in an up-front, honest way that avoids misunderstandings and creates shared clarity and connection.

Two critical moments in forming of a group:

The first vulnerability
The first disagreement

These are doorways to paths:

Appearing “strong” and winning interactions.
Exploring the landscape and learning together.

Good listening is not just nodding attentively. It’s about adding insight and creating moments of mutual discovery.

Effective listeners

Interact in a way that makes the other person feel safe and supported.

Take a helpful, cooperative stance.

Occasionally ask questions that gently and constructively challenge old assumptions.

Make occasional suggestions to open up alternative paths.

Effective listeners are active responders, absorbing what the other person gives, supporting them, and adding energy to help the conversation gain velocity.

When asking questions, don’t stop at the first response. Find different ways to explore the area of tension, ask secondary “whys”. The first response is usually not the answer, it’s just the first response.

In conversations, resist the temptation to reflexively add value. Vulnerability is often created by what you do _ not _ say. Don’t interrupt with “quick ideas” or your experiences.

Candour, not brutal honesty. Make feedback smaller, more targeted, less personal, less judgemental.

Flash mentorship: shadow every teammate for a few hours.

Purpose

Repeat your mission every day during a standup.

Mental contrasting:

Think of your goal and imagine you’ve achieved it.
Picture the obstacles vividly.

Motivation is not a possession, but rather a result of a two-part process of channeling your attention: where you are and where you want to go.

Stories are the best thing ever created for delivering mental models that drive behavior.

Effective learning culture:

Conceptualize learning as something that will help end goal (or customers) rather than as an add-on to existing practices.
Explicitly tell why each role on the team is important for ultimate success.
Don’t just dive into action. Design deliberate learning practice with post-analysis and talking about communication.
Encourage people to speak up if they see any problem and use active coaching.
Use retrospectives.

Response to setbacks with energy to fix things (love problems). Help colleagues. The number one job is to take care of each other.

If somebody behaves poorly, you should avoid judging them and instead give them the benefit of the doubt.

Don’t spread negative energy in the workplace (e. g. if you have a bad mood).

You have priorities, whether you name them or not. If you want to grow, you would better name them explicitly, as well as the behaviors that support these priorities.

Create engagement around a clear, simple set of priorities.

Creating purpose in a creative group is about building systems that can churn through lots of ideas in order to help unearth the right choices.

Non-catchy phrases for teams of people who need to discover what to do for themselves (creativity):

Hire people smarter than you.
Fail early, fail often.
Listen to everyone’s ideas.
Face toward the problems.
It’s more important to invest in good people than in good ideas.

Building creative purpose isn’t about creativity. It’s about building ownership, providing support, and aligning the group’s energy towards an arduous, error-filled, ultimately fulfilling journey of making something new.

(This description reminds me of the hero’s journey.)

Successful cultures use crises to crystallize their purpose. Be grateful when reflecting on failures. They are crucial to discover what you (or the team) can be.

Name and Rank Your Group’s Priorities

Obvious but true: in order to move toward a target, you must first have a target. Listing your priorities, which means wrestling with the choices that define your identity, is the first step. Most successful groups end up with a small handful of priorities (five or fewer), and many, not coincidentally, end up placing their in-group relationships—how they treat one another—at the top of the list. This reflects the truth that many successful groups realize: their greatest project is building and sustaining the group itself. If they get their own relationships right, everything else will follow.

Be ten times as clear about your priorities as you think you should be. It’s necessary to drastically overcommunicate priorities.

Regularly challenge the company’s values and purpose. Create conversations that encourage people to grapple with the big questions. What are we about? Where are we headed?

Figure out where your group aims for proficiency, and where it aims for creativity.

Proficiency: building purpose to perform these skills is like building a vivid map. Spotlight the goal and provide crystal-clear directions to the checkpoints along the way.

Fill the group’s windshield with clear, accessible models of excellence.
Provide high-repetition, high-feedback training.
Build vivid, memorable rules of thumb: if X, then Y.
Spotlight and honor the fundamentals of the skill.

Creativity: empowering the group to do the hard work of doing something that has never existed before. “Supplying an expedition”: provide support, fuel, and tools.

Keenly attend the team’s composition and dynamics.
Define, reinforce, and relentlessly protect the team’s creative autonomy.
Make it safe to fail and to give feedback.
Celebrate hugely when the group takes initiative.

You can subscribe to new Engineering Ideas via RSS (e. g. using Feedly) or e-mail.

DEV Community: Roman Leventov

Enable CORS with custom headers for an AWS Lambda function behind API Gateway in Serverless framework

Add cors configurations to HTTP points of the function definitions in your serverless.yml

Add resources section to serverless.yml to include CORS headers in API Gateway-level error responses

Include CORS headers in the normal responses in your lambda handler

Engineering Ideas #9

Engineering Ideas #8

Drift Into Failure by Sidney Dekker – notes on the book

Five characteristics of drift

Philosophy of complex systems

Analysis of failure

High-reliability theory

Systems thinking

Complexity and systems theory

Emergence

Exploration-exploitation tradeoff

Diversity

Tactics to manage the complexity of drift (“drift into success”)

Complexity and accountability

Single Responsibility Principle Unpacked

What Does the Single Responsibility Principle Say?

Types Of Responsibilities

Business Logic

External Integration

Data

Control Flow

How Small or Large Should a Responsibility Be?

The Impact Of the Single Responsibility Principle On Different Software Qualities

Understandability and Learning Curve

Flexibility

Reusability

Testability

Debuggability

Observability and Operability

Reliability

Code Size

Performance

Summary

Engineering Ideas #7

Engineering Ideas #6

What makes engineers productive?

The Culture Code by Daniel Coyle - notes on the book

Safety

Cooperation and Vulnerability

Purpose

Add `cors` configurations to HTTP points of the function definitions in your `serverless.yml`

Add `resources` section to `serverless.yml` to include CORS headers in API Gateway-level error responses