DEV Community: Piet van Dongen

Resolving the paradox of cloud-native solutions: more agility without total control

Piet van Dongen — Tue, 31 Jan 2023 08:20:55 +0000

Being cloud-native enables organizations to implement changes at the speed of light, but leveraging cloud technology also means giving away control. That sounds like a paradox, but it resolves once you realize that you don’t need to give away control completely. But what do you keep in-house, and what power do you give to your cloud technology and third-party software vendors? Sneak peek: focus your efforts on your core competitive edge while smartly integrating with the non-differentiating stuff.

Don’t feel like reading? Watch this post (and more) as a video

Last week, I joined an XPLUS Breakfast Session to talk about how organizations can leverage cloud technology to shorten their time-to-value. Our client Ctac, personified by Ivo van der Raad, started the webcast by presenting their cloud plans for their XV Retail Suite. Exciting stuff, so feel free to come back here later and watch the session first.

I used my ten minutes to look at this typical retail challenge from a technical point of view. This blog post is an extended transcript of that presentation, with the upside that you don't have to look at my face the whole time. Win-win.

Now, back to resolving the paradox.

The need for speed

Before I get into the technical stuff, let me briefly explain the problem in business terms first.

Consumer-facing software is moving faster than ever, with customers expecting increasingly safer, better, and faster solutions. Evolving technology is a significant enabler and driving force behind this acceleration. Successful solution providers combine tailor-made software, third-party packages, and SaaS offerings to craft the stuff their customers crave. That's impressive! But it also sounds like a magic trick. An illusion, if you will. How is it possible to hand over so much control to external parties and maintain the agility needed to compete in the market?

Smart integration, that's how.

The integration challenge: from central management to distributed control

Three focus areas emerge if we compare this current reality to the world of a decade or more ago.

The first is a shift from centrally managed, self-hosted application landscapes to distributed solutions. Let's say an organization has a few systems containing customer data and many custom-made services and clients that integrate with them. A decade ago, their IT department would have self-hosted this custom and third-party software combination, employing a variety of service buses, API endpoints, and direct messaging solutions. They would have slowly evolved the landscape whenever they had to build new functionality or add a new system.

Thanks to the advent of Software as a Service, IaaS, PaaS, and high-level cloud services, organizations need to move much faster to keep up. Their landscapes are fragmenting. Keeping expertise in-house is costly and reduces innovation budgets.

Here emerges the second challenge. The technology market's shift to SaaS and managed cloud services, combined with the never-ending desire for better customer functionality, forces companies to gain traction and move more dynamically.

That can only mean, whether they want to or not, that builders need to let go of some of the control they used to have. But they still need to integrate everything. They must learn how to leverage new ways of integrating their building blocks, and I propose they use those giant vendors' shoulders and shift a lot of the heavy lifting onto the public cloud and SaaS providers' offerings.

The cloud-native integration paradox: more agility, less control

All right, now we are looking at what seems to be a paradox. Organizations need to be in total control, but at the same time, they need to hand over some control to external parties. What gives?

Organizations need to be able to move fast to respond to the changing needs of their customers. They must deliver changes into production quickly, see what works, and continually adapt based on what they learn. To do so, they require the technology and skills to precisely determine what their customers want and give it to them. They must practice deploying changes rapidly but safely. And they need to develop a software delivery process that is solid and improving over time. In other words, organizations need to be in complete control.

However, I suggest that they hand over some control to external parties. That means they no longer have total power over all the underlying infrastructure or the services and software they need to combine into a solution.

So here we have the paradox of delivering value, end-to-end, through software using the cloud and software as a service: the need for more agility while simultaneously leveraging the offerings of external vendors.

Luckily, this is not an actual paradox. It is possible to increase agility while giving up control, but the trick is to be smart about it. Let me explain.

Build, buy, outsource, integrate? All of the above (but: pick your battles)

Let's look at the choices the fictional organization behind this highly successful (but fictional, like I said) app has made.

In the image above, you see an example architecture of an AI-powered photo retouching app, broken down into its components and plotted on a Wardley Map. The y-axis shows us how valuable elements of the solution are to end users. The higher, the more visible a component is to them, and the more valuable they deem it: a user generally does not care if you use platform X or Y as long as their app works reliably. The x-axis is about the evolutionary stage of the components. Just-discovered technology, for example, will appear on the left, but the older it gets, the more it will shift to the right. Over a decade ago, container platforms were relatively new, so they would have been plotted much farther left than they are now.

At the top left, we see the mobile application users interact with daily. It uses custom algorithms running somewhere in the backend or the app itself. This functionality is by far the most customer-visible and valuable thing they own; that’s why it’s so high on the map. So, they want complete control: they design, develop, and deploy the app and algorithms themselves.

To further evolve the algorithms, their data scientists and business stakeholders need to gain continuous insight into their users' behavior. The blue dots are the systems and services that help them collect, combine, transform, and finally visualize their user's behavioral data using managed services, third-party tooling, and a hosting platform. They don't need complete control here, so they opt to hand off its management to service providers. However, they do need control over integrating this part of their landscape.

Then there is the so-called boring stuff, at least from a software creator's point of view, at the bottom right in orange: the critical infrastructure. Their customers do not care one bit if they manage this themselves. But like water and electricity, it is fundamental for everything built on top to work. So, they want to exert less control here: they don't want to spend costly resources managing all this highly available, virtually infinitely scalable stuff. They want to leverage the years of experience of a party that knows how to run critical infrastructure while paying only for what they use. So they outsource this.

Here we have the first part of the answer to the so-called paradox of less control and more speed. Now let's solve the integration part of this puzzle.

Integrating the solution: less control = looser coupling

Here is the same solution from an integration perspective.

We are still looking at a mix of self-hosted applications integrated with managed, third-party services and externally hosted SaaS. At the infrastructure level, there is a mixture of public cloud, private cloud, or a private data center, and the magical SaaS infrastructure.

The (still fictional) AI-powered photo retouching app company smartly combines integration solutions to make something of these mixed ingredients. The general rule of thumb here is simple: the more control they have over solution lifecycles, the tighter they can couple them. And vice versa: less control, looser coupling.

On the left-hand side, there is all the stuff they control entirely. They integrate using direct connections, push notification services, and message queues.

In the middle are the managed services they employ. They use a central event bus, so they decouple the integration slightly more than on the left by introducing a highly configurable, flexible, but fast integration solution in the middle. It could be a serverless event router like Amazon Eventbridge or Azure Event Grid.

At the far end, on the right-hand side, in orange, are the external parts of our landscape: SaaS offerings, maybe another cloud, or externally hosted service. Since the photo app company has no control over them, they decouple these elements using data transfer services or an event bus at the edge of their specific hosting platforms. This way, the organization can still evolve its end-to-end solution, and the SaaS and external service providers can continuously update their offerings independently. In the case of an API or contract change, the AI photo app developers only need to change the data transformation logic, while the rest can keep running as before.

Now the speed-up-with-less-control paradox is resolved. Again, we need to realize that our landscape consists of groups of subsystems, grouped by their properties — in this case, the amount of evolutionary control the organization has over each solution piece. From this perspective, they can then integrate them using fit-for-purpose integration solutions. They still control their total solution, just not all its internals.

Excellent, but the story may not convince you entirely yet — the company, its users, and its success are fictional, after all. Let me try to convince you with a real-world example where I was closely involved.

OHRA: cloud-native development and integration from the trenches

OHRA, one of the Netherlands' biggest directly writing insurance companies, asked Luminis to help migrate their application landscape from their on-premise data center to the AWS cloud. Their application landscape was large, with almost 1,000 integrations between internal and external services, clients, and databases. Did I mention the fixed deadline? About a year from the start. We helped them make it, partly thanks to the abovementioned integration strategy.

Most of OHRA's moving parts are under their control, so they are integrated using a mix of direct messaging and message queues. When moving to AWS, we helped them mostly replatform the applications from application servers running Java applications to a managed Kubernetes cluster running containers. Much of OHRA's data flows through third-party packages spread over their application landscape. So we employed the tactic we saw on the previous slide: loose coupling for SaaS integrations using data transfer logic at the edge and asynchronous communication with third-party apps installed in their application landscape.

A big help in making OHRA's cloud migration deadline was their existing service-oriented architecture, an excellent stepping stone towards a more event-driven and, thus, cloud-native architecture.

Further reducing complexity and costs, improving agility

After migrating, the cloud provided OHRA with new opportunities to evolve its solutions. One example is the modernization of batch applications.

Previously, scheduled jobs were deployed on self-managed servers, running 24/7, even though most of them only actively did something for a few minutes to a couple of hours a day. In the public cloud, you should pay only for what you use, so I helped OHRA envision a more cloud-native and cost-effective way of creating short-running jobs.

During two weeks, I led a team that re-architected an existing solution and delivered a modern, lean batch application: a distributed data processor made from a combination of serverless building blocks like Amazon EventBridge rules, Amazon SQS queues, and AWS Lambda Functions. As you can see in the image above, this drastically reduced the amount of code needed (and thus its complexity), the resources used, and the costs accrued.

OHRA can use this cloud-native way of working moving forward and enable their teams to free up time and resources, which they can then spend on stuff that will help the business grow its competitive edge.

One more time: the paradox resolved

So, now you have a complete, high-overview answer to the faux paradox of gaining speed while not having complete control. Allow me to recap.

First off, from a market perspective, companies are forced to move away from centrally controlled, neatly contained solutions running in their data centers towards integrating their locally managed custom building blocks with third-party software and externally hosted SaaS solutions.

There is a two-part solution to this problem.

The first part concerns being smart about spending valuable time and resources. Organizations require complete control of the software development and deployment lifecycle for the user-facing features they develop. Everything that powers this end-to-end solution — but is less visible or even completely invisible to the customer — can be hired as a service or bought as a product from a cloud provider.

The other half of the answer concerns being smart about integrating a fragmented landscape. I identified three groups, from fully controllable and evolvable elements to managed services and non-transparent SaaS solutions. Tight coupling is a good solution for distributed applications under a company’s control, but managed services and SaaS need looser coupling. Organizations can implement this solution by leveraging a combination of message queues, event buses, and data transfer services.

And thus, the paradox is dissolved!

The Evolution of AWS from a Cloud-Native Development Perspective: Serverless, Event-Driven, Developer-Friendly, Sustainable

Piet van Dongen — Fri, 18 Mar 2022 13:51:34 +0000

Authors: Auke Noppe and Piet van Dongen

Recent AWS announcements feature fewer service additions and instead underline the efforts the cloud giant is undertaking to increase the strength of its massive portfolio of building blocks. AWS CEO Adam Selipsky has mapped out a course that intensifies AWS’s focus on serverless technologies, event-driven architectures, an improved developer experience, and sustainability.

In this blog post, we outline the direction cloud-native engineering is heading in, how you can learn to leverage AWS’s portfolio of managed services and primitives to your advantage, and how cloud technology is reshaping the field of software engineering.

If you prefer watching a video, we got you covered! We recently broadcasted a cloud-native development focussed AWS re:Invent re:Cap featuring most of this article’s themes. You can find the recording at the end of this post.

What AWS is telling us

With so many services in its portfolio already, this year’s AWS re:Invent didn’t hold any massively surprising announcements. With the yearly conference reaching its 10-year milestone — and AWS being around for 15 years — the AWS platform is maturing while at the same time evolving to meet market demands.

AWS leadership’s re:Invent keynotes

Freshly at the helm of AWS, Adam Selipsky held his first re:Invent Keynote. Selipsky’s primary message: AWS is shifting from merely offering infrastructure primitives to embracing the idea of being a so-called platform of platforms. Therefore, AWS’s offerings will increasingly become an interesting piece of organizational value chains.

While Amazon CTO Werner Vogels took the stage after an introduction referencing Fear and Loathing in Las Vegas, his keynote wasn’t as spectacular. Dr. Vogels took the time to make a case for Well-Architected applications, which we interpret as a good signal for builders. Other highlights were the release of Construct Hub and the CDK Patterns library.

Primitives, not frameworks

An important theme these last few months was the expansion and refinement of AWS as a cloud-native platform. You can view AWS as a data center containing compute resources and seemingly limitless storage at the other end of the internet, or maybe as a big bunch of infrastructural primitives. The way we look at the platform is by grouping its services into three categories:

Infrastructural primitives: storage, networking, and compute (a.k.a. Infrastructure as a Service, IaaS).
Middleware-oriented services: web servers, container orchestrators, databases, message queues (a.k.a. Platform as a Service, PaaS).
Serverless: take what you need, pay-as-you-go.

Building something on a platform containing 200+ services might seem a bit overwhelming. Therefore, from the perspective of cloud-native development, we use this rule of thumb: start building from the most modern collection of services first and, if needed, complement the solution using more primitive building blocks.

Frequently, you will end up with a serverless solution that is rapidly and almost infinitely scalable and highly cost-effective. However, sometimes there are missing pieces of the puzzle, and you can’t complete it with serverless services. We can then turn to platform primitives or infrastructure services like containers, RDS, virtual machines, or even complete SaaS solutions. Don’t forget to update your solutions from time to time, as the platform never stops evolving.

Reduce risk, improve performance, grow revenue, increase efficiency

Effectively adopting the cloud means making the cloud work for your money. It’s not just a virtualized data center, or at least: that’s not the mindset that enables you to accelerate your organization towards its goals. Cloud-native development requires understanding cloud-native best practices in terms of performance, resiliency, cost optimization, and security. Luckily, AWS keeps investing in adoption frameworks, its Well-Architected Framework, and certification and training programs.

The ongoing evolution of serverless: commoditize all the things

You can immediately tell the serverless category from above is something special. Its services allow you to focus on functional value without worrying about what is needed to run it. You can compare a serverless service to a commodity, like electricity: you use it anytime you need it, without much second thought. For a while, serverless was synonymous with functions (FaaS) or Lambda, but its reach rapidly extends beyond compute to data stores and application integration.

Serverless technology keeps gaining traction, as we can see in its continuous evolution. Let’s look at AWS’s current serverless portfolio and see how it evolves.

Three categories of building blocks

Roughly speaking, we can divide AWS’s serverless building blocks into three categories:

Compute: running code without provisioning servers (Lambda, Fargate).
Data Storage: storing objects, documents, relational and other data (Simple Storage Service (S3), DynamoDB, Relational Database Service Proxy, Aurora Serverless).
(Application) integration: EventBridge, Step Functions, Simple Queue Service (SQS), Simple Notification Service (SNS), API Gateway, AppSync.

Combining these building blocks, you can quickly create valuable solutions at scale.

Serverless expansion areas

While SaaS is powerful and often offers excellent value for money, it has significant constraints in terms of flexibility. To win in the marketplace, we need more flexibility for our differentiating technology. These are the lines along which serverless is evolving. Its journey started at compute, but the serverless philosophy quickly moved into other AWS categories.

Big data and streaming

Amazon MSK will get a serverless counterpart, which is great news for organizations running Kafka workloads. The same goes for AWS’s big data platform Amazon EMR and its data warehouse offering Amazon Redshift. Amazon Kinesis is also evolving its data streaming service, exemplified by the new Data Streams On-Demand feature. Last but certainly not least: AWS Data Exchange for APIs makes it a breeze to use third-party data while leveraging AWS-native authentication and governance.

Application integration

Apart from getting extra ephemeral storage, AWS Lambda now supports partial batch responses for SQS and event source filtering for Kinesis, DynamoDB, and SQS. EventBridge added Amazon S3 Notifications. On the surface, that might sound like a small change. But as Matt Coulter puts it: code is a liability, not an asset. Anything you can push to the cloud vendor gains you some security and an increased ability to focus on value. You can now control Amazon Athena workflows using AWS Step Functions in that same vein.

Are applications, infrastructure, and integration separable?

Now that serverless is moving beyond compute into the data and integration layers, we might ask ourselves: are applications, infrastructure, and integration separable? Is programming still mostly about functions and logic, and where does it bleed into the domain of operations? The advent of serverless is pushing boundaries everywhere and blurring previously clear and stable lines.

That’s excellent news, in our opinion: the more control and flexibility our teams have, the faster we can innovate. Gregor Hohpe, an Enterprise Strategist at AWS, says: “I firmly believe that we are just starting to realize the true potential of cloud automation. Modern cloud automation isn't just about reducing toil. (…) it can help us blur the line between application and automation code.” We agree wholeheartedly.

The future (of cloud) is event-driven

Another direction cloud technology developments are pointing towards is that of event-driven architectures. Not only is the cloud itself highly event-driven, but applications built on cloud platforms also tend to be fitting most naturally when using the power of events. Why is that?

Agility in architecture

Market forces are driving organizations to become more agile. As Jeff Bezos puts it: “The only sustainable advantage you can have over others is agility, that’s it.” Organizations must learn to leverage technology to respond to and drive change to fulfill this business requirement. Event-driven architectures present the necessary characteristics:

Loosely coupled: heterogeneous services and share information effectively while hiding implementation details.
Scalable: multiple types of consumers can work in parallel, massively if needed.
Independent: evolution, scaling, and failing can happen in isolation, especially when using buffers like event routers.
Auditable: if events are routed and stored centrally, we can quickly implement auditing, policy enforcement, and information access control.
Frugal: if we push changes when available, we don’t need to waste resources continuously polling for changes.

Event-driven systems enable us to focus on system behavior and temporality within a business domain, freeing us from the chains of rigid structures incapable of dealing with changing needs and requirements. There is a trade-off (of course): agile systems are complex and thus force us to learn to manage — seeming, but not actual — chaos.

AWS building blocks for event-driven architectures

Now, how is AWS helping us compose these systems? Whether you are building event-sourced systems, leverage CQRS, or are just notifying components, AWS has the building blocks you need:

Producers: most AWS primitives, especially serverless ones, produce events. For example, we can feed Lambda with events from API Gateway, CloudWatch, DynamoDB, S3, Kinesis, and many more services. Besides that, we can use custom apps, SaaS solutions, and microservices as event producers.
Consumers: notable consumers are AWS Lambda, Amazon SQS, Amazon Kinesis Data Firehose, and Amazon SNS. We can also use APIs from SaaS offerings and custom apps as event consumers.
Routers: we can fulfill most integration needs using Amazon EventBridge and Amazon SNS.
Well-Architected: the AWS Well-Achitected Framework describes best practices, which we can assess and monitor using the AWS Well-Architected Tool.

Combining these AWS services and using them to integrate SaaS and self-built products gives us the leverage we miss when only using off-the-shelf products.

How are event-driven cloud architectures evolving?

The answer to this question is short and straightforward: along the same lines as serverless. Last December, AWS mostly announced incremental improvements. But they quickly add up, accumulating in an ever-more powerful platform to build and evolve our solutions on. And the announcements don’t stop when re:Invent is done: AWS is working to improve its event-driven primitives year-round.

Amazon Eventbridge is an excellent example of this continuous investment. It was introduced in mid-2019 and has gained a lot of features since then: a schema registry, archiving and replaying of events, and a long list of event destinations. Its evolving event pattern matching, filtering, and routing options embody the ‘code is liability’ philosophy, enabling us to focus on value delivery. Recently, S3 Event Notifications were added to EventBridge’s features, giving us more direct, reliable, and developer-friendly options when responding to S3 object changes. For the web-scale organizations among us, AWS introduced cross-region event routing.

We mentioned Lambda’s new message filtering features above but want to highlight them again since they underline AWS’s continued investment in this area. Lastly, AWS IoT TwinMaker deserves some attention: it utilizes events as data sources and enables developers to create digital twins of real-world systems, which is nothing less than fantastic.

Improving the developer experience

AWS provides us with many building blocks, but using them effectively is not straightforward. Luckily for us, AWS seems to understand this problem and is steadily improving in that area.

Programmable everything: APIs, SDKs, and the AWS CDK

In 2002, Jeff Bezos's so-called API Mandate forced all Amazon teams to expose their data and functionality through service interfaces. Amazon has built AWS around the same principles: every service is programmatically controllable, from starting a virtual machine to accessing a satellite. While this is an essential property of an effective cloud platform, it is not necessarily developer-friendly.

By now, AWS has incrementally and significantly improved in this space. Besides using their APIs, we can control services and infrastructure with a unified Command Line Interface (CLI), Software Development Kits (SDK), CloudFormation, and, since July 2019, the AWS Cloud Development Kit (CDK).

The CDK had a massive impact on developer productivity and satisfaction. While teams could already access and control services and infrastructure using the AWS SDK and their favorite programming language, infrastructure was primarily defined using incredible amounts of mostly punctuation marks and whitespace, also known as YAML or JSON. CDK — initially only flavored TypeScript and Python — finally gave developers an AWS-native means to define Infrastructure as actual Code. Since its introduction, AWS has added support for more languages, like Java, C#, and Go.

Metrics, toolsets, and communities

The lines between infrastructure and logical code have thus been blurred, creating more opportunities to increase business agility. And AWS provides more tools to increase development productivity, like metrics and purpose-built toolsets. New developments in this space are:

Community support platforms like the newly announced re:Post and Construct Hub.
Monitoring improvements like CloudWatch Real-User Monitoring and Metrics Insights.
A Secrets Detector addition to the software engineer’s best non-human programming buddy, Amazon CodeGuru.

Reducing complexity

Another way to increase (business) agility is by reducing complexity. AWS seems very aware of the complexity we add when developing cloud-native solutions. It has released several offerings and improvements to its platform over the last period in reaction to this.

Less handcrafted code

We can reduce the amount of hand-crafted code we deploy in several areas (once more: code is a liability!). Business analysts can kickstart machine learning workloads using Amazon SageMaker Canvas, while others can visually create full-stack apps using Amplify Studio. More incremental improvements are enhanced dead-letter queue management for SQS and the Lambda and S3 enhancements mentioned earlier.

Feature flags and dark launches

Reducing operational complexity is a way to increase productivity. Modern organizations have learned to leverage feature flags and dark launches to test changes without introducing much operational overhead. Our colleague Nico Krijnen took the time to write down how Amazon CloudWatch Evidently and AWS AppConfig Feature Flags can help us in this regard.

Migration automation

Lastly, a powerful — but often not so straightforward — way to reduce complexity is to migrate existing workloads from the old data center to the cloud. AWS has been investing a lot of effort in this space and continues to do so, as evidenced by the recent releases of AWS Migration Hub Refactor Spaces, AWS Mainframe Modernisation, and AWS MicroService Extractor for .NET.

Sustainability of the cloud, sustainability in the cloud

Software engineering has more dimensions than speed and complexity. With the advent of DevOps, FinOps, and the increased awareness of IT infrastructure's impact on our environment, teams are increasingly responsible for more than just developing and deploying code.

Adrian Cockcroft, Amazon’s newly appointed VP of Sustainability Architecture, recently headed a very insightful talk on architecting for sustainability. Amazon has committed to net-zero carbon by 2040, and AWS aims to use 100% renewable energy by 2025. AWS needs their customer’s help to decrease their footprint, so AWS has introduced several sustainability tools and improvements to their platform.

The Customer Carbon Footprint Tool was released earlier this month, enabling AWS users to calculate the carbon emissions their workloads produce now and in the future. Reducing them is the next logical step, which is why AWS added a new Sustainability Pillar to their Well-Architected Framework. More concretely, teams optimize resource usage by choosing several new CPU instances for EC2 or Lambda or by analyzing the enhanced AWS Compute Optimizer infrastructure metrics and acting accordingly.

Survival of the fittest

AWS’s cloud platform is becoming more mature, stable, developer-friendly, and sustainable while at the same time evolving rapidly to meet emerging business needs. By closely following the needs of its users and experimenting and growing with them, AWS keeps delivering the platform of the future.

For us cloud-native developers, that’s excellent news. Increasing business agility by creating and evolving event-driven, serverless, sustainable systems that can turn on a dime in response to customer needs: it’s what we need now and going forward.

Silver Linings, No Silver Bullets: 3 Perspectives on Innovating with Cloud

Piet van Dongen — Wed, 15 Dec 2021 09:48:08 +0000

Authors: Auke Noppe and Piet van Dongen

We’ve heard the phrase, seen the cynical stickers: “There is no cloud; it’s just someone else’s computer.” That might have been true 15 years ago, but the thing we call the cloud has evolved significantly since. We live in times of accelerating change; organizations that can leverage the cloud effectively will survive them, even thrive in them. How well prepared are you?

Taking great advantage of the cloud requires a shift in thinking: technology no longer belongs to the IT department (if it ever did). To become cloud natives, organizations need to learn to streamline their effort around everything the cloud offers.

Let’s look at this paradigm shift from several perspectives: operations, development, and leadership. Each of these perspectives starts with describing the ineffective way of working. We use the term cloud naive (as in: inexperienced or lacking knowledge). Or as the State of DevOps report classifies it: the way of the low performers. Then, we cross the chasm and look at it from the cloud-native perspective, as used by companies that use cloud technology to accelerate their business continuously.

Operations: embrace speed to reduce risk

Low performance: cloud naive operations

In simpler times, developers delivered complete software packages to operations people, who in turn installed and ran them on the appropriate infrastructure. Preferably not too often and with as little change needed as possible. A similar rule determined a snail’s pace of change for platforms: standardization before innovation. Why? Because every change introduced risk and operations were held responsible for it, often having to clean up the mess when things went wrong.

This model led operations departments to push back hard on changes and reduce the speed of change—developers adjusted by increasing release sizes and time between releases.

The result: slow change, little cooperation, suffering users.

Elite performance: cloud-native operations

The way out of this standoff is well-known by now but not always well understood or effectively implemented: DevOps. Making agile teams responsible for business outcomes forces down the classical barriers between the business, developers, and operations, leading to better customer outcomes. Cloud platforms have evolved by adding layer upon layer of abstraction onto the classic data center, leading to hundreds of managed services that teams can continuously leverage to decrease their time-to-value. Perhaps counterintuitively, this increased rate of change enables teams to manage risk better than ever.

Think about it: if time-to-value is the most critical metric for success, change size logically also gets smaller. Minor changes are easier to manage in case of failure, and deploying multiple times a day yields teams totally in control of their technological solutions.

This cloud-native way of doing operations yields precisely what these teams aimed for in the first place: less risk, more control.

Development: focus on outcomes, not technology

Low performance: cloud naive development

Before the emergence of the cloud, servers had to be installed by hand, as did the software running on them. Installing and testing releases was a time-consuming process, mostly done by other teams. So, developers made sure their sparsely released work contained as many features as possible. Any infrastructural requirements and other dependencies had to be carefully communicated and planned with IT operations, dictating long lead times between innovative ideas and customer value.

As a result, developers preferred working with stable, minimally variable environments simulated on their development machines. Their products were massive monoliths often written using a single programming language, running on single application container instances. Development feedback might have been fast; customer feedback certainly wasn’t. Developers were often kept totally in the dark about the impact of their changes in production until nightly calls from frustrated IT operations, who spoke in an entirely different IT dialect.

Again: slow change, little cooperation, suffering users.

Elite performance: cloud-native development

The step-by-step virtualization and containerization of servers, platforms, and runtime environments have enabled development teams to master what was previously the sole job of operations. And more importantly, the marriage of Dev and Ops has given birth to beautiful things like infrastructure as code, CI/CD, and SRE. Managing complex systems this way has opened up opportunities for creating massively scalable but well-observable systems. The old monoliths have been split into small, modular services or even single functions that typically communicate asynchronously using a mind-boggling number of events.

Now the customer wins. Changing, replacing, or radically rewriting a single, small module and testing and deploying it in the blink of an eye has become a reality. Experimenting — seeing if an idea for a new job to be done holds up in production — has become cheap as chips using the pay-as-you-go managed services as provided by cloud platforms like AWS, Azure, and GCP.

Leadership: get out of the way of innovation

Low performance: cloud naive leadership

During less volatile times, markets were predictable, transformations were carefully planned and budgeted, and users were okay with incremental changes every so often. Software, and the hardware it ran on, were complex and expensive beasts, mastered only by tech masochists stuffed away in cost centers called IT departments. Nothing got built until it was very meticulously specified in requirements documents. Even then, the delivered result was often wildly different from what the market specialists and innovators defined months or years before, if not already outdated.

Once more: slow change, little cooperation, suffering users.

Elite performance: cloud-native leadership

Software has been eating the world for a while now. Organizations need to adapt or accept a slow and painful decline into irrelevance. Users now expect to get what they need before realizing they do: a new Netflix series, a recommended product, or even an improved driving experience. Understanding users is key to market success and requires continuous experimentation and software development mastery. The cloud is a massive enabler: it enables organizations to focus on user needs by handing the management of undifferentiated technology over to the cloud provider.

But while technical skill development is essential, it is just one part of the innovation puzzle. Another is culture: leaders need to focus everyone’s attention on the importance of time-to-value and agility as the metrics for success. To create empowered, self-sufficient, and effective teams, people need to come together and shatter the walls previously separating them. Lastly, leaders need to take the long view and realize that short-term risk aversion can be a major innovation blocker.

To boldly go, together

Transforming an organization from cloud naiveness into cloud-native isn’t for the faint of heart, but very necessary nonetheless. Luckily, Luminis has traveled this path many times before and is a very effective guide on the path to cloud-native. For example, we helped OHRA, one of the largest Dutch digital insurers, migrate its complete data center to the cloud. Another example is the digital transformation of The Learning Network (TLN) we helped shape and implement.

Get in touch and let us help you kickstart your cloud-native transformation.

What is resilience engineering? A lightning talk with background information

Piet van Dongen — Sat, 15 May 2021 13:00:30 +0000

The coronavirus pandemic has taught us two things: all online systems will fail eventually, and they will fail in unpredictable ways. But have no fear; resilience engineering is here!

Building and maintaining event-driven, highly scalable, and quickly adaptable systems are a pain in the neck. Resilience engineering embraces this complexity and helps you manage it.

In the TEQnation lightning talk below, I will explain what resilience engineering is, why every software person should know about it, and how you can start practicing it. You can find the transcript below.

More on resilience engineering

Hungry for more? Great! This might help:

Resilience engineering: where do I start?

The Resilience Engineering Association (REA) has created a lovely introductory guide on resilience engineering. After reading or scanning it, you should have a pretty good idea about what resilience engineering is, where it came from and who to follow if you want to learn more.

Safety science

Resilience engineering takes a lot of inspiration from safety science. In short, safety science aims to help people die less and in less horrible ways. A prominent researcher is Erik Hollnagel, who introduced the terms Safety I and Safety II. The former focuses on preventing error, the latter on increasing success.

Complex systems

The term system has special meaning in the context of resilience engineering. It is about both humans as well as technology, thus the term socio-technical system. When resilience engineers talk about complex systems and how they fail, they focus on the whole, from broken machine parts to overworked maintenance workers and increasing market pressures.

Richard Cook has written a surprisingly readable paper on how complex systems fail. Another great resource is the book Drift into Failure by Sidney Dekker, which contains examples ranging from economic collapse to airplanes crashing into the sea.

Here’s Richard Cook again, this time using the human body to explain what resilience engineering is all about:

Blameless post mortems

An essential element of resilience engineering is the post mortem. To emerge from failure more strongly than before — a critical property of a resilient system — engineers need to learn how to think about finding and fixing causes of failure. That‘s why you can often hear resilience engineering advocates (confusingly) proclaim: there is no root cause.

John Allspaw can explain really well what is needed to dig into failure and learn from it:

DevOps

Resilience engineering has an ongoing and major influence on the DevOps movement. Disciplines like chaos engineering and site reliability engineering are heavily leaning on resilience engineering principles and research.

Gene Kim and John Willis extensively talk about the origins of this side of DevOps (and much more) in their audio series Beyond the Phoenix Project.

Blog series

Learning about resilience engineering is an ongoing effort here at Luminis. That‘s why we decided to start a blog series on resilience engineering. We will continue to add to it over the next few years, and I invite you to follow us and give us your feedback. I believe that writing, reading, and reflecting are excellent (and fun!) ways to become experts on complex subjects like resilience engineering.

Thanks again for your attention!

Video transcript

Welcome

Hi everyone! And welcome to my TEQnation lightning talk titled: What is resilience engineering?

About me

Before we dive into the details, I want to talk about my favorite subject: me! I am Piet van Dongen. I am 37 years old. I work at Luminis, a super nice Dutch company that is all about technology leadership, software craftsmanship, and sharing knowledge. Check us out at www.luminis.eu!

I once started my career as a primary school teacher, then moved to web design, front end engineering, back end engineering, and now I am a Cloud Consultant.

I am really, really excited about cloud technology. And I think understanding resilience engineering is essential for anyone working with technology — especially cloud technology, which enables us to build and evolve super-complex systems.

I’d love to tell you more about me, but this is a lightning talk. Let’s get cracking!

Overview

What will you learn in the next 10 minutes? This is a talk about resilience engineering, so we’ll start off with resilience.

What is resilience exactly? This will give you an idea of what resilience is and is not.

Then, the engineer parts comes into play. I’ll talk about safety science, systems thinking and the socio-technical system.

Finally, I’ll give you some pointers. So you can do a resilience engineering deep dive yourself.

Resilience

Now, what is resilience?

To understand resilience, you need to think about robustness first. We — you know, software people — are really good at building robust systems. Systems that are highly available, that can handle a little peak load, a little stress. To do so, we use techniques like:

Redundancy.
Fallbacks.
Retries.
Failovers.

Sound familiar? Great.

We do this because we tend to think about what might go wrong. You know:

Servers might fail.
Networks might become unreliable.
Things go brrrrr.

We don’t want that. So we design around all the things we know can go wrong. Resilience engineering people sometimes call this: the known unknowns.

Robustness is nice, but it does not protect us when things go south in the most unexpected ways. Redundancy won’t help when the Retro Encabulator’s hydrocoptic marzel vanes go from horizontal to vertical fumbling. We call these: unknown unknowns. Stuff we don’t know we don’t know, to quote Ronald Rumsfeld.

That’s the stuff resilient systems can handle. And robust systems can not.

Take this glass, for example. I can do this (Piet taps the glass), even this (Piet taps the concrete floor with the glass). It’s designed to withstand all that. It’s robust. But it is not very resilient. If I do this (Piet smashes the glass with a hammer), it breaks.

But, smashing glasses with a hammer is not a good example of resilience engineering. Smashing with a hammer is not an unknown unknown for glassware designers. It’s just not a requirement.

Hmm, lemme think…

Yes, I have a better example!

(Piet takes a hammer. And smashes fingers. He screams loudly. Fade to black. Elevator music plays. Fade back. He now has bandages on his fingers.)

Piet, now with a trembling, high-pitched voice:

Yes, a much better example. The human body is an excellent example of a resilient system. You see, even though I smashed my fingers with a hammer, my body still works. Even better: in a few weeks, these bones will be stronger than before. Great.

Now that I’ve explained the difference between robust and resilient systems let’s dive into the engineering part of resilience engineering.

Safety science

Let’s go back to my example about known unknowns. The things we know can go wrong. If you think about things that can go wrong a lot, you tend to focus on preventing error.

And how do we prevent error?

By minimizing variance. For example, by not allowing certain input.
By writing a LOT of documentation. The Boeing 747 flight crew operating manual is 3,000 pages long.
By enforcing things. Like making it impossible to shift to reverse while driving your car.

That’s all fine and dandy, but it only helps us prevent the known unknowns. And there are a LOT more unknown unknowns.

Have you seen the HBO series Chernobyl? A great example of disaster caused by unknown unknowns:

Graphite tips.
Bad management.
Cover up by the government.

Just to name a few. So how do you prevent that? That’s what the field of safety science is all about, the birthplace of resilience engineering.

In trying to create a safer world, safety scientists are trying to shift the focus from avoiding things that go wrong, sometimes, to making sure things go right, most of the time. They do this by focusing not only on technology but on humans as well.

Socio-technical systems

When resilience engineering advocates talk about systems, they do not mean components or computers, or networks. They are talking about the so-called socio-technical system. Socio as in: the people that interact with the technology. This perspective is known as systems thinking.

You see, whether you like it or not, humans are always part of the systems we create. Take the Chernobyl example again. What was ‘the system’ there? Just the nuclear power plant? The control room, the reactor, the steam turbines. Or the people as well? The operators, the managers, the government.

When thinking in socio-technical systems, the humans are just as important as the machines.

So what? How does this prevent error? What’s the difference? How do you engineer resilience, then? Surely, you can only create systems, not the humans as well? That’s where the engineering part of resilience engineering finally comes into play.

Resilience engineering

When things go wrong, they do not go wrong suddenly and all at once. Again, Chernobyl. The explosion of the reactor core was the climax, for sure. But what caused it?

The operators in the control room pushing a shutdown button? Maybe, but they did not anticipate the disastrous outcome. They didn’t even completely understand what was happening at the time, even though they were surrounded by tons of data. Also, they were under a lot of pressure from their superiors. Who were under pressure themselves as well. Plus, nobody imagined that a shutdown would cause an explosion. What happened was radically different from what people imagined would happen.

As you see, a lot of factors are at play here. Both human factors as well as technical factors. Socio. Technical. And I think that is the central theme of resilience engineering: to make sure things go right, most of the time, using the engineer skills we have.

How do we do that? By helping the human part of our systems as much as possible.

One way to do that is by creating transparent systems. Systems that can tell you what is happening, what state it is in at any point in time. Systems with clear controls, interfaces that enable operators to control the system when they decide they need to. That way, we enable humans to imagine systems as close as possible as to their actual state, which is, of course, impossible.

Another way to practice resilience engineering is by creating easily extensible systems. Systems that can be improved over time. So write clear, modular code. Implement architectures that can evolve. Make sure you are clear about what your code does, or at least: should and should not do.

Lastly, when things go wrong, and they will go wrong, keep an open mind. Don’t focus on a single root cause because it does not exist, and it only limits your chances of success. Don’t blame each other, but use the opportunity to create an even better system. A system that increases the chances of success over time.

That is what resilience engineering is all about.

Conclusion

So in conclusion:

Resilience is not robustness.
Resilience engineering is about creating success, not eliminating error.
Resilience engineering is about humans and technology.

I hope you have learned something today. This talk only scratches the surface, of course. And the same goes for me: the more I learn about resilience engineering, the more I realize there is much more to learn. That’s why I’ve created a blog post that will help you dive deeper. You can find it here.

Thank you very much for listening to me. I hope you have a great day!

The new normal demands resilience engineering

Piet van Dongen — Tue, 12 Jan 2021 21:40:16 +0000

Understanding resilience engineering is essential for organizations that want to survive as we accelerate into the online future. Let me tell you what resilience engineering is and how it can help you roll with the punches that come with success.

The only way to keep the Dutch at home is by grounding their offspring, as our government has discovered twice over. So, after spending two weeks dreading the end of the holiday season, last week marked the start of the second homeschooling season in a year.

Thanks mostly to my very structured wife and non-male children (no coincidence there), our household was prepared. We set up two desks in the living room, made schedules, and brought laptops, tablets, headphones, pencils, paper. But when it was time for the digital school gates to open, they… didn’t.

Watskeburt?

In our case, the school’s digital learning and communication platforms choked on the peak load all the homeschoolers generated. Luckily, Netflix/YouTube/the bookcase/outside was still online, so we just postponed the inevitable for a bit. By lunchtime, everything was already forgotten, replaced by the regular remote schooling joys and frustrations.

After dinner and coffee and ice cream, and some delicious white chocolate, I, ehm… Where was I? Oh yeah, I started to get curious about that morning’s outage. And I found an interesting ongoing status report by the Parro team. Apparently, they already anticipated the surge and had taken what they believed would be enough measures to survive it:

“To be fair, we expected that Parro, with all the measures taken, would be able to handle the new extreme peak traffic.”

After what I imagine must have been a stressful few hours, the Parro team fixed all issues and had a smoothly running system by the end of the day again. Kudos to them. That’s what mattered most to their customers, the teachers and parents of our small kingdom.

But their work is not done, as the updates on the status page show. Where is the next bottleneck? When will it become a problem? How will they fix it? Will they ever be done?

They won’t. This is the new normal. But it doesn’t have to be a negative thing. There is a way to use these weird times to engineer ourselves into the future.

Let‘s examine how.

Peaks are the new normal

Now that we are accelerating into the online future, unpredictable massive peak usage will become just another truth. Scary? Sure. Exciting? I think so. But a reality, nonetheless.

This will lead to two types of problems to deal with:

Known unknowns: a problem that hasn’t happened yet, but is not surprising once it happens. Server failures, network troubles, that kind of stuff.
Unknown unknowns: something that comes as a complete surprise, shocking systems into failure. Or, as Donald Rumsfeld called them: “the ones we don’t know we don’t know.”

The first type is tough enough to handle as it is. But a well designed, robust system can withstand these known unknowns. Redundancy, retries, fallbacks, failovers: when designing highly available systems, these terms are familiar. We use them to deflect what we know can go wrong, like a server going down or a request to a service timing out.

But what about a global pandemic leading to sudden, massive demand on a system designed for a different reality? Surely, you can’t handle all the ways a system might break down under these unpredictable conditions?

But you can. It’s called resilience engineering.

Handling the unknown unknowns

Resilience engineering is a field still very much inside the scientific domain. It is closely related to the areas of human error, cognitive systems engineering, and safety. You know, plane crashes, oil spills, and nuclear disasters, that type of stuff.

Websites and mobile apps going down rarely lead to environmental disaster or death. That doesn’t mean we can’t learn anything from the tons of research that went into making our world a little safer. Resilience engineering takes all the good stuff that prevents 747s and powerplants from going down and applies it to our young (software) engineering field.

Old thinking, new thinking

Like my colleague Hans Bossenbroek so eloquently put: as the need for agility and speed increases, we need to move towards event thinking. The solution: event-driven, modular, highly scalable systems. But also: super complex architectures resembling chaotic spider webs. And a gazillion events flowing through at breakneck speed.

Of course, that’s not something you design and deploy at the end of the first sprint. And it might be a long way off from your massive monolith. But since we live in web-scale-or-die-times, it’s the logical solution to real-world problems. As the technology evolves, so should the way you manage it. You have to fight the urge to control the new system the way you did your old one.

Instead of focusing on the things that might go wrong (known unknowns!), try instead to look at the things that go right and do those things more often. This will help you and your teams:

Understand what leads to the right thing and do that more often.
Increase the chance of success instead of nervously awaiting the next breakdown.
Be proactive instead of reactive.
Drive towards continuous improvement.

You’ve read that right; it’s about humans as much as it is about hardware and software!

Systems thinking

You see, systems encompass both people and technology. Think about it: when was the last time you heard about a developer accidentally deleting a production database? Or the other way around: some Sherlock Holmes type saving the day by tweaking a setting even the most experienced colleague was unaware of?

And that is where resilience comes from: humans doing the right things. We do that because we just know what is right. We have a knack for it, some kind of intuition. It’s what keeps complex systems from falling apart.

And when it does fall apart, like last week, something must have tipped the scales. Despite all the hard work by the humans and the computers, systems go down from time to time. And when they do, they have to be brought back up as soon as possible.

Now, stop and think: if production goes down, what do you do? Maybe something like:

Find the root cause.
Fix it.
Make sure that never ever happens again.

If you want to practice resilience engineering: wrong answer.

There is no root cause!

Stop again. Think about the complex system. We already saw that keeping a system running is a balancing act. Finding a root cause would imply that there is a single thing that caused the error. If good behavior is explained by complexity, how can a single cause explain faulty behavior?

It can’t. It‘s a logical fallacy. There is no root cause.

John Allspaw

@allspaw

There is no root cause. bit.ly/wIvsrQ #devops

16:01 PM - 10 Feb 2012

This also means that a proper postmortem should not be about explaining the hunt for a single point of failure. An excellent postmortem is about showing that you understand that complexity sometimes results in unwanted behavior. And that you realize, in hindsight, that many things were working together until they didn’t. But that you have all learned from it. That you now know how to do even more things right.

In other words: good postmortems are blameless, balancing team safety and accountability.

Going resiliently into the future

So, to err is… complex. Faults are a given. Resilience engineering is not about preventing them. It‘s about doing ever more things right so that you can increase complexity while maintaining control. So that when the next unknown unknown strikes, you will be prepared.

So get to it! Learn:

What is resilience engineering? (You already know!)
How do I start practicing it?
Whom should I involve?
How do we get better over time?

Need help? Keep an eye on our blog series on resilience engineering to find out the answers to these questions!

Observations on the resilience of bone

In the meantime, here is an excellent talk by Dr. Richard Cook about resilience engineering and bones. Bones? Yes, you know, you have them inside your body most of the time! And they are a great vehicle for explaining two types of resilience engineering. Watch and learn:

Migrating to the cloud: 6 strategies for future success

Piet van Dongen — Wed, 16 Dec 2020 09:59:44 +0000

Let’s say you’re convinced (and rightly so!) that the future of your organization is in the cloud. How will you get there? Many roads lead to Rome, some more complex and promising than others. In this blog, I will share six strategies for getting into the cloud successfully. Which one fits your organization best?

Choosing a migration strategy strongly depends on the goals you wish to achieve. Some approaches offer you many strategic opportunities, but they are often a bit more complex to implement. Some options are relatively straightforward but come with higher costs in the long run. Therefore, be aware of the why of your migration and choose the strategy that best suits your needs.

Strategy 1: Lift & Shift

Also called rehosting, the idea is simple: you move your systems to the cloud platform with as little change as possible. You basically move your whole data center to the cloud.

Rehosting may be an attractive choice: you’re done relatively quickly, and little risk is involved. However, this strategy has a significant disadvantage: operational costs will be relatively high, especially in the long run. Furthermore, it will take some time to set up your new environment and connect your existing CI/CD solutions.

Do you need to move out of your data center sooner rather than later? This strategy might be a good fit. But beware: the clock is ticking. Once the migration is done, you need to start changing your applications and architecture for a better cloud fit. Monitor your costs meticulously, so you know which applications are good candidates for the next round of refactoring.

Another option is to do targeted lifts and shifts. There’s a good chance that a handful of older systems has to move to the cloud as well. You can move them at the end of the migration, then phase them out or replace them at a later stage.

In short: this can be a useful strategy, but use it wisely and sparingly. Lifting and shifting is a way to quickly get into the cloud, but it comes at the cost of having to do more work afterward.

Strategy 2: Replatforming

A.k.a. lift-tinker-and-shift. You’re still not changing your systems’ functionalities, but the underlying platform gets an upgrade.

Do you have a chunky database server with an expensive license running somewhere? Then this is a great option. A database-as-a-service is a considerable improvement in several ways: no more managing systems and paying for idle time.

Another example: a colossal application server running lots of deployments on expensive hardware. Setting up and maintaining clusters of those is very time consuming and complex. Moving these applications to Docker containers is certainly an attractive option in that perspective.

Beware the disadvantages, though, which mostly lie in the less visible parts. The details of the cloud’s underlying platform differ subtly from your on-prem’s. Keep the fallacies of distributed computing in mind and practice resiliency.

Not only does this strategy provide you with the chance to shift your IT spending to a radically different model, but it can also greatly improve your organization’s agility. So: lots of cloud gold, without touching your architecture. But there’s more to gain when you move towards cloud-native.

Strategy 3: Refactoring / rearchitecting

Refactoring (also called: rearchitecting) is the most far-reaching strategy in terms of architectural and applicative change, but it’s bursting with potential. By most effectively using what the cloud offers, things that were impossible on-prem are within reach when taking the rearchitecting route. Think unprecedented scalability for minimal costs and using services that would conventionally mean huge upfront material and human capital investments. Not to mention time-to-value improvements.

The impact of this strategy is mostly dependent on the current state of your system landscape. How tightly coupled are your applications? How modular is your architecture? If you’re already reasonably service-oriented, you’re halfway there. A step towards microservices — even better: serverless — is not a giant leap from there. It‘ll put you right at the cutting edge.

Fortunately, again, this isn’t an all-or-nothing strategy. It’s an excellent fit for migrating applications that exhibit cloud-native characteristics. But it is equally interesting to view rearchitecting from a business value perspective.

Suppose you could do magic and instantly accelerate your idea-to-production time. Deliver multiple times a day. With fewer bugs. Without having to compromise on security and stability. Which applications would yield the best results then? That part of your portfolio probably benefits the most from the refactoring / rearchitecting strategy.

Strategy 4: Keep

Rest assured, migrations are not all about doing buzzword bingo heavy work. Not all systems have to be migrated. Some are doing just fine in the data center. You might migrate them later on, or maybe you can switch them off entirely in the future.

During a cloud migration, you will acquire tremendous amounts of technical know-how. But a cloud migration is not a strictly technical affair. Slowly but surely, your organization will start shifting towards a new way of thinking and working (together). Sometimes, systems turn out to be not such a good fit anymore. Sure, they can last for a little while longer, but they need to go sooner or later. Bringing them to the cloud might not be worth the hassle.

Maybe you’ve just gone through a big systems upgrade project. Or have other reasons just to leave some parts be. It might be a valid option.

Strategy 5: Phase out

It’s not always clear from the get-go, or maybe you just didn’t expect it. But during a migration, you always bump into stuff that can be retired.

For example, when we helped OHRA plan and execute their migration from their data center to the AWS cloud, they eventually switched off about 20% of their applications. That saved them a lot of migration work.

Strategy 6: Replace (with SaaS)

This one is aimed at systems that cost a lot but yield little. Replacing them with SaaS solutions is a viable strategy here. A cluster full of mail servers and file servers filled with spreadsheets might be a candidate for replacement with Office 365 or Google Workspace. Salesforce can replace a CRM system, and an on-prem content management system has plenty of SaaS alternatives.

Developing software is costly enough as it is. Save time, energy, and money to build something that sets you apart from the competition.

Which cloud migration strategy is right for your organization?

There is no such thing as a one-size-fits-all cloud migration strategy. Every organization has a different set of goals and ambitions. Pick a strategy that best fits yours. But keep this in mind: you will reap the most cloud benefits if you try to be as cloud-native as possible. The more you stick to the old data center centric model, the more you will see this reflected in your operational costs.

With the right combination of knowledge, experience, and crew, you can make your cloud migration a success story. We’d be happy to help!

Want to know more? Download our cloud migration white paper

My colleague Bert Ertman and I have written a white paper about cloud migrations. In it, we answer the following questions:

What promises does the cloud hold? What are its pitfalls?
Which migration*strategies* can you use? (You already know)
Which people do you need and when?
How do you plan a cloud migration?
When can you start with the first migrations?
How do you gain migration velocity?
When and how can you complete a migration successfully?
What’s next?

You can download the cloud migration white paper for free. Have fun reading!

Soundbyte 390: Crystalline

Piet van Dongen — Sun, 14 Apr 2019 20:44:19 +0000

Whenever something that I have been looking forward to is over, I feel so sad and empty inside. The bigger the event, the worse the feeling. I can vividly remember falling into the black hole after my first birthday-disco-party was over and my last friend left. I just wanted it to not be over, I wanted the fun and freedom to last forever and ever.

This week the feeling hit me again, twice. The last few weeks I had been rehearsing a play for my daughters' school camp and last Thursday we performed it. It was so much fun to get together with the other parents, to anticipate playing the scenes and to finally see the looks on all the little faces when doing the play. But the sadness afterwards, aaaargh, I could really have done without it.

Still, the ups more than make up for the downs, I wouldn't have missed it for the world. Quite literally, because that Thursday also marked our 5th Luminis DevCon. Luckily I had my phone with me so I could ask my friends what I was missing out on. A lot of good stuff, so they have told me.

I was especially proud of my colleague Sam van Geijn, who spoke about his super funny Dutch Rap Generator. Sam was still a student last year and now he has done a talk in front of an actual live audience with actual human beings in it. And a very good talk at that, a few friends of mine let me know afterwards. I could write a whole post about what I love about Sam and his talk, which I might actually do later.

The same goes for the DevCon itself. I love that we have a yearly event about the Art of Software Engineering (not just programming!), and that we do almost everything ourselves. From writing papers to creating a high-quality program, from the brainstormins to the organisation of the day itself, it's all done by the colleagues we work with every day. And it shows, the feedback from visitors has always been very positive. This I could go on and on about as well, but I won't.

I want to get back to that feeling of sadness after big events I started this post off with. You see, my personal news bubble this week was filled with the same image:

I was watching the Event Horizon Telescope press conference and when they finally showed the image the whole world seemed to have been waiting for years for, I got that feeling again. That was it, I thought, these people have poured their hearts and souls into this project and now it is over, done, finished. What in the world must they feel like now? Somehow that thought stayed with me the whole week.

And then this morning, I was a watching a two year old talk by Katie Bouman, explaining how software was going to help her team capture the image of the black hole:

Watching that made me so happy for some reason. This is so awesome, I thought. These people dreamed of creating a telescope the size of the earth so they could take a picture of something Einstein dreamed up more than a hundred years ago. And then they actually went ahead and did it! How can you be sad about that? That's awesome!

I get the same feeling from my job, I really do. I don't really care so much about the software I am writing. Don't get me wrong, I am proud of my code, but what I produce has no value outside of its context. The value comes from all those other people that are part of the work we do: the directors and clients with their wild imaginations and forward thinking, the designers and their creative solutions to the seemingly impossible problems, the project managers and Scrum masters that keep our eyes on the prize and all the other stakeholders that make sure we build what people didn't know they needed.

On a huge scale, the achievement of the Event Horizon Telescope project is an example of this. No one human can imagine a black hole all by themselves, let alone take a picture of it. But the whole of humanity surely can. Pushing boundaries is a team effort.

Now for the music. After this week's shitty controversy regarding Katie Bouman's work I was reminded of this Björk quote from a 2015 Pitchfork interview:

I want to support young girls who are in their 20s now and tell them: You’re not just imagining things. It’s tough. Everything that a guy says once, you have to say five times.

So let me do the work of five women at once: Björk was right. Do something about it. Start with her awesome song Crystalline from the topically space-and-nature-themed album Biophilia and work from there. The outro alone is worth the listen.

Have a nice week!

Pure Bliss With Pure Functions in Java

Piet van Dongen — Fri, 15 Feb 2019 21:45:24 +0000

Writing software is hard.
Writing good software is harder.
Writing good, simple software is the hardest.
Writing good, simple software as a team is the hardest… est.

You shouldn't need several hours to understand what a method, class, or package does. If you need a lot of mental effort to even start programming, you will run out of energy before you can produce quality. Reduce the cognitive load of your code and you will reduce its amount of bugs.

“This is puristic nonsense!” you say. “Intellectual masturbation! Software should be modeled after real-world, mutable objects!”

I'm not saying you should go on a vision quest and return as a hardcore functional programmer. That wouldn't be very productive. Functional and object-oriented programming can complement each other greatly.

I will show you how.

But first, let's get to the what and why of it.

What?

A pure function is a function that:

…always returns the same result, given the same input.
…does not have any side effects.

Put simply: When calling a pure function with input A, it always returns B, no matter how often, when and where you call it. Also, it does nothing else.

In Java, a pure function might look like this:

public static int sum(int a, int b) {
    return a + b;
}

If a is 2 and b is 3, the result is always5, no matter how often or how fast you call it, even if done so concurrently.

As a counterexample, this would be an impure function:

public static int sum(int a, int b) {
    return new Random().nextInt() + a + b;
}

Its output could be anything, no matter the input. It violates the first rule. If this would be part of a program, the program would become nigh impossible to reason about.

An example of the second rule, a function that has side effects:

public static int sum(int a, int b) {
    writeSomethingToFile();
    return a + b;
}

Even though this method is static and always returns the same value, it is impure. It does more than it advertises. Even worse, it does so secretly.

These examples may look simple and harmless. But impurity adds up quickly.

Why?

Let's say you have a decent-sized program you want to add a new feature to. Before you can do so, you'll have to know what it does. You will probably start by looking at the code, reading documentation, following method calls, et cetera.

While you are doing this, you are creating a mental model of the program. You are trying to fit every possible flow of the data inside your mind. A human brain is not optimised for this task. After a few state changes here, some conditionals there, you start to get somewhat foggy.

Now we have a breeding ground for bugs.

So how can we prevent this? By reducing the complexity of our program, for starters. Pure functions can be of great help here.

How?

A great way to clarify complex code is breaking it up into smaller, more manageable pieces. Ideally, every bit has its own responsibility. This makes them easier to reason about, especially if we can test them in isolation.

While untangling the spaghetti, you will notice you are now able to free up some of your intellectual capabilities. You are slowly getting at the core of the problem. Every piece that can stand on its own can be moved and you are left with the essence of your program.

Finding that bug is much easier now. So is adding a cool new feature. To make things more concrete, here are some guidelines.

Partition

Firstly, partition the bigger pieces by their functionality. Think about the various parts of the solution and how they interconnect.

Break Up

Break up those parts into composable units. Define a function that can be used to filter items from a list, or an action that can be re-used for every single item. Maybe add helper functions that encapsulate logic that would otherwise end up inside deeply nested code.

Document

Write documentation when you feel you are ‘done’ implementing a unit. This will help you see the logic from a different perspective and reveals unforeseen edge cases. If necessary, add unit tests to further define the program's intents.

Rinse and repeat.

The definition of done is personal and can change over time. Don't overdo it. If it's still not clear enough, you will find out later. If not during a code review, maybe when adding a new feature to the project.

So far, so good, right? Theory always sounds great. But we need some convincing in the form of a practical example.

An Example

Let's say we have a client that sells IoT devices. Customers can install these devices in their home and connect them to the internet. They can control these devices with an app on their mobile phone.

Our client wants their customers to receive a push notification if their device has been offline for a while, so they have a chance to reconnect the device. If it stays offline, they want to push notifications periodically, to remind the customer to reconnect the device. But not too often, since they don't want unstable devices to cause an endless stream of warnings.

Also, our customer wants to be able to adjust the thresholds for sending these messages. At runtime, without having to pay us to do it for them. There should be at least one threshold, but there could be any number of them.

Using our backend software, we can detect the online status of these IoT devices. If they go offline, we store the moment they went offline.

Naive Implementation

How can we implement this? This feature looks quite simple on paper, so we could just start and see how far we'd get.

After doing some scaffolding we quickly get to the core of the problem: determining whether notifications should be sent for each offline device. Sounds simple, right?

public void run() {
    // @todo Send notifications for offline devices
}

We add some for loops here.

for (Map.Entry<Device, Instant> offlineDevice : offlineDevices.entrySet()) {
    for (Duration threshold : thresholds) {
        // ...
    }
}

Some if statements there.

if (firstThresholdWasPassed) {
    // ...
}

Looking good!

Wait, there's a special case for the last threshold.

if (i == thresholds.size()) {
    // ...
}

Oh, and one for a single threshold.

if (thresholds.size() == 1) {
    // ...
}

Crap, we forgot to check if the notification for each threshold was already sent. In 3 places.

if (!lastOfflineNotificationInstant.isPresent()) {
    // ...
}

And what if it was sent before the device went offline?

if (Duration.between(disconnectInstant, lastOfflineNotificationInstant.get()).isNegative()) {
    // ...
}

Oh my.

SonarLint is foaming at the mouth, telling us to reduce the cognitive complexity of 69, where 15 is allowed. The unit tests are getting quite complex as well, having to use an external Clock an whatnot.

We cannot let our future selves see this, they'd kill us!

Luckily, we have our three-step program. Let's begin at 1.

Partitioning by Functionality

Now, to untangle this mess. Our first step is to partition the bigger pieces by functionality, to think about the various parts of the solution and how they interconnect.

We could visualize our problem by drawing a timeline, starting at the instant a device went offline:

On the timeline we can plot the various thresholds:

Every time our job is running, we have arrived at a certain point on the timeline. If we have not yet passed a threshold, nothing needs to be done:

If we do pass a threshold, we should send a notification and remember the time we sent it:

Next time we run our job, we should take the sent notification into account. If we have passed a threshold that the user was already notified about, we do nothing:

Only when passing another threshold, we can send another notification:

And so on and so forth.

How do we express this in code? It seems we need to know:

Which threshold (if any) we passed last.
Which threshold (if any) we sent the last notification for.

That's it, basically. If we know these, we can determine if we should send a notification. All the special cases and complexity are captured by these two core parts of our solution.

Now we need to see if we can simplify our solution any further by breaking it up into composable parts.

Breaking It Up Into Composable Units

The two above statements look awfully familiar, no? That's because they are. They both define a threshold or nothing. They both need similar input:

The moment the device went offline.
The threshold(s).
The point in time we are currently evaluating.

Now that we know the input and output of our unit, we can define its signature. Using the fancy java.time API, we can express it as such:

Optional calculateLastPassedThreshold(Instant start, Instant current, Duration[] thresholds);

We can now use this function to get both thresholds we needed. Did I say function? Let's make that a pure function: declare it static and make sure it's deterministic and doesn't create any side effects:

static Optional<Duration> calculateLastPassedThreshold(Instant start, Instant current, List<Duration> thresholds) {
    Duration timePassed = Duration.between(start, current);

    if (timePassed.compareTo(thresholds.get(0)) <= 0) {
        return Optional.empty();
    }

    for (int i = 0; i < thresholds.size(); i++) {
        if (timePassed.compareTo(thresholds.get(i)) <= 0) {
            return Optional.of(thresholds.get(i - 1));
        }
    }

    return Optional.of(thresholds.get(thresholds.size() - 1));
}

There! Aside from its breathtaking purity, it has another pro: It can be unit tested very easily. You don't need mocks nor external clocks to test this part of the code.

Can you feel the amount of space your brain just regained? When programming the rest of our solution, we can just trust that this function will do what it says.

So what's left?

We can now determine, for each device, if we should send a notification. Again, we can isolate this part of our program, make it pure and test it thoroughly:

static boolean shouldSendNotification(Instant jobStart, Instant deviceOffline, Instant lastNotification, List<Duration> thresholds) {
    Optional<Duration> lastPassedThreshold = calculateLastPassedThreshold(deviceOffline, jobStart, thresholds);

    if (!lastPassedThreshold.isPresent()) {
        return false;
    }

    if (lastNotification.isBefore(deviceOffline)) {
        return true;
    }

    Optional<Duration> lastPassedThresholdNotifiedAbout = calculateLastPassedThreshold(deviceOffline, lastNotification, thresholds);

    return !lastPassedThreshold.equals(lastPassedThresholdNotifiedAbout);
}

Or, more concisely:

static boolean shouldSendNotification(Instant jobStart, Instant deviceOffline, Instant lastNotification, List<Duration> thresholds) {
    Optional<Duration> lastPassedThreshold = calculateLastPassedThreshold(deviceOffline, jobStart, thresholds);

    return lastPassedThreshold.isPresent() && (lastNotification.isBefore(deviceOffline) || !lastPassedThreshold.equals(calculateLastPassedThreshold(deviceOffline, lastNotification, thresholds)));
}

And when you know the device has no previously sent notifications, you can just use:

static boolean shouldSendNotification(Instant jobStart, Instant deviceOffline, List<Duration> thresholds) {
    return calculateLastPassedThreshold(deviceOffline, jobStart, thresholds).isPresent();
}

Now the only thing left is calling these functions for every offline device, every time our job runs:

public void run() {
    Instant jobStart = Instant.now();

    offlineDevices.entrySet().stream()
            .filter(offlineDevice -> pushNotificationService
                    .getLastOfflineNotificationInstant(offlineDevice.getKey())
                    .map(instant -> shouldSendNotification(jobStart, offlineDevice.getValue(), instant, thresholds))
                    .orElseGet(() -> shouldSendNotification(jobStart, offlineDevice.getValue(), thresholds))
            )
            .forEach(offlineDevice -> pushNotificationService.sendOfflineNotification(offlineDevice.getKey()));
}

We're bordering on iamverysmart-territory here. Not everyone likes this style of coding and it is arguably harder to parse than a more traditional style. So the least we can do is document our units.

Documenting Our Solution

Aside from playing nice with others, describing our code can also help us catch that last bug, or reveal an edge case we hadn't thought of yet.

Documentation in our case most obviously comes in the form of Javadoc, for example:

/**
 * Checks whether a notification should be sent by determining which threshold has been passed last for the
 * calculated amount of time passed between the device going offline and the job running.
 *
 * @param jobStart         The instant the job calling this function was started.
 * @param deviceOffline    The instant the device went offline.
 * @param lastNotification The instant the last notification was sent.
 * @param thresholds       The list of notification thresholds.
 * @return True if the notification should be sent, false if not.
 */
static boolean shouldSendNotification(Instant jobStart, Instant deviceOffline, Instant lastNotification, List<Duration> thresholds) {
    // ...
}

Unit tests can also be regarded as documentation since they can define the intended use of the program under test very precisely. In the case of pure functions, where you often need to test a list of varying input, parameterized tests can come in handy.

While writing tests and documentation, we shift our thinking from the problem to the solution, and its users. By thinking from this other perspective, subtle problems in our code are highlighted and can be fixed on the spot.

Afterthoughts

It can be hard to convince others or even yourself of the value of this approach. Churning out functionality has more obvious business value than meticulously crafting software.

When approaching existing code this way, you can end up wasting hours while adding no measurable (short-term) value. Try explaining that during the daily stand-up.

On the other hand, preventing a bug is cheaper than fixing it, and the more clean your code is, the less time it takes to add a feature or find an error, especially if the code hasn't been touched in a while.

Somewhere along the line, the balance is tipped. On one side lies the holy grail, the unachievable perfect piece of software. On the other side, the unmaintainable spaghetti monster slithers, shitting on deadlines, ballooning budgets and destroying teams.

Listen to your instincts, communicate your intentions, listen to your peers and learn from one another: making great software is an ongoing (team) effort.

Source code

All code referenced in this post is open source and can be found in its GitHub repository. Feel free to check it out, tinker with it and come up with better solutions.

Feedback

I'd love to hear your feedback in the form of pull requests, issues, or comments! There are some nice discussions going on over here:

Or you can post your comments below.