DEV Community: defmyfunc

Integration, integration, integration. An introduction to walking skeletons in the enterprise

defmyfunc — Fri, 18 Oct 2019 00:00:00 +0000

If I was to pick out what I think the single biggest difference between “Enterprise” and “Non Enterprise” software development, it would probably be the amount of integration that is necessary to achieve success. Integration comes in many forms, not just other systems, but integration across people and process as well.

Growing Object Oriented Software Guided by Tests (GOOS) is one of my ‘cornerstone’ books as to how I approach software development, as such, I wanted to write a little about one of the most under used, and when it is used somewhat misused techniques, the “walking skeleton”.

What is a “Walking Skeleton”?

GOOS defines it as:

A “walking skeleton” is an implementation of the thinnest possible slice of real functionality that we can automatically build, deploy, and test end-to-end - Cockburn, Alistair. Crystal Clear: A Human-Powered Methodology for Small Teams. October 29, 2004.

Sometimes its easier to define something by comparing it to what it isn’t:

It isn’t a prototype

We throw prototypes away. A ‘walking skeleton’ is production code. Its important that you have validated a user need before embarking on a walking skeleton.

It isn’t a minimum viable product

Functional, isn’t the same as feature. A ‘walking skeleton’ should work, but it might do little of value from the users point of view.

So what is it?

There are probably as many definitions of what is technically a walking skeleton, there may even be a ‘minimum viable skeleton’ out there.

My view of a ‘walking skeleton’ is probably a bit closer to Dan North’s “Dancing Skeleton” and I tend to approach it in a similar vein to that articulated here: https://gojko.net/2014/06/09/forget-the-walking-skeleton-put-it-on-crutches/

When we approach developing the ‘walking skeleton’ within enterprise software delivery its important to take an extra factor, integration.

I define a ‘walking skeleton’ as:

Something which proves all the known integrations, across people, process and tech, for a given scope.

It does not have all the functionality required to deliver value to a user, but should eliminate the risk that a integration or dependency problem will delay delivery of that value or cause it to fail completely.

Walking skeletons are essential in the ‘de-risking software delivery’ toolkit. Often software delivery starts by building out the easy things … “get the quick wins!”… but without integration into the wider ecosystem in which that software lives, the users will never see the value of that whizzy new system you are building them!

Doing the ‘new feature’ is often the least risky part … a quick technical spike and you can understand whether the feature or idea you want is worth progressing. Failing to integrate can cause untold pain on the users of the system and have disastrous consequences for a business and its customers. Walking skeletons are a technique that help us front load the risk of software delivery.

The aim being we reduce risk over time rather than leave it all to the end when a shiny new product has to learn how to play nicely with the rest of the world.

Defining a walking skeleton

Architecture diagrams hide a multitude of sins in a rounded rectangle.

The simplistic view of the walking skeleton bounds it within the context of the software you intend to deliver, and this should often, rightly, be your first pass:

I tend to think of these as a kind of fat, ‘tracer bullet’, recently discussed here: https://twitter.com/martinfowler/status/1184234201614372865 and https://blog.thepete.net/blog/2019/10/04/hello-production/

You should make sure it includes all the good things your hello world application needs, unit tests, functional tests, etc, etc.

Making it dance

I think its super important to ‘make it dance’ and be able to interact with it via an interface. This makes it more real from the point of view of your stakeholders. So lets add that:

You should also make sure you update your pipelines … do you now need contract tests for example? Maybe some e2e type tests?

Instrumenting

Being able to learn from your users and system is important, next we layer on the instrumentation so that we can learn from their interactions with our system:

This means adding integrations with the systems that we use for monitoring, observability, audit and reporting. As well as any others you can think of. Yes. Instrument that ‘Hello world’ app so you can observe it, change it, understand your live service needs, confirm audit, and whatever else your business requires.

Integrations

Being able to prove the technical integrations between systems is an important step. If you can’t integrate with the existing SAP monolith … then well may as well go home. But its not just a chance to “enjoy” integrating with a system … this is your chance to build those relationships with those teams. Because guess what, those teams are central to your success… without them you are nothing … and you can piss and moan all you want about how “bad” they are … or you can knuckle down and collaborate… hell, get one of them to join your team. You might learn something :)

People and process

Now we are getting somewhere … but now comes the tricky part … find a way to role play … find a way that means you can role play your use cases in the production environment. To do this you are going to have to learn how to “Test in production”. I’m sorry … but you kind of have to have this fight with your organisation now … I know you’ve been putting it off … but … ¯\(ツ)/¯

Being able to role-play the use-cases through the system as you develop them is an important way of learning about what you have to do and integrating with existing people and processes across the organisation. This is the difference between a well run transition to a new process and a nightmare transition that everyone hates and means no investment for 5 years until the organisation has forgotten the pain of last time. So figure it out… a good summary of the techniques you may find valuable is here: https://medium.com/@copyconstruct/testing-in-production-the-safe-way-18ca102d0ef1

Phew

I appreciate this may seem like a lot of work, and to some extent it is … but depending on your organisation you may already have a lot of this in place. Obviously, you may not … especially if you are just starting or mid way through your ‘transformation’. But there are some great SaaS services that you can get for low cost as you learn and then evaluate as you grow. And remember … you don’t have to do all of it … you want just enough for ‘hello world’ in each of them.

But the benefits are worth it. If you do this you will have all the tools at your disposal to become a genuine product team. If you repeat this pattern then as an organisation you get very good at doing it, and whilst the first time might be hard, by the time you get to your 5th it’ll be a walk in the park. And if you are feeling adventurous … you might find you don’t need those other environments at all…

The walking skeleton is an important tool in the developers toolkit, for when there is a proven need to deliver a product in a problem space. And whilst a thinner ‘tracer bullet’ style walking skeleton is suitable for some environments, within a heavy integration environment, like a lot of enterprises are, you may find more value in making something fatter. Its also an excellent way to de-risk the whole project and set yourself up for future success by being to adapt and learn from your users.

Thanks for reading! Thoughts as ever, appreciated.

An analogy to describe software development

defmyfunc — Fri, 09 Aug 2019 13:24:00 +0000

In my twitter feed recently, I have noticed a spate of analogies, metaphors and similes to describe software development (eg. https://twitter.com/jessitron/status/1165787547764551680) and so I wanted to share the analogy I have been using for the past few years. Most of the analogies I have seen have focused on allowing for the distinct difference between building a "traditional factory made manufactured product" and the concept of "constant product development" within software delivery, where you are constantly discovering and adapting "the product" to user need.

The analogy I use the most to compare to software development is that of "running a restaurant".

Why?

According to this 5 second google search for what makes a good analogy:

"A good analogy is a compromise between two conflicting goals: familiarity and representativeness" - https://www.scotthyoung.com/blog/2014/10/21/good-analogies/

I would also add:

"A good analogy gets out the way quickly as the person uncovers the truth." - @defmyfunc. A person I respect once told me, "sometimes we have to stop using analogies and just explain software development."

Which is why I think restaurants make a great analogy.

Familiarity

First, what I hope is the easy one, familiarity, A large proportion of people:

understand the work that has to go into food preparation
will have had food prepared for them by someone else at some point
will have paid for someone to prepare food for them
have seen some form of restaurant cooking show (even if just to turn it off before Gordon Ramsey swears again)

As a bonus in the UK, we have these lovely stickers:

in every place that prepares food, which means that pretty much everyone understands there is legislative and compliance regime to food standards (even if they don't know the detail).

Representativeness

Now the one I have to prove with stories and let you decide. I often start with the initial analogy which I use to explain the key framework items by which everything else hangs:

General analogy

Think of your software 'product' as a restaurant, and when people use your product, thats them having a tasty and delicious meal at your restaurant. And we charge them and they pay us, and if we want them to come back, we need to find ways to show our value to them and make them loyal customers. But behind the scenes, so that our customers can keep eating tasty and delicious meals and keep paying for them, we need to do a whole host of things every day, like clean the restaurant and its equipment, clean the tables, clean the plates and cutlery and the pots and pans, ask 'how was your meal?'. And every so often we need to do things like, health and safety accreditation, or create new and exciting menus for our customers.

Technical debt

We have to constantly work on technical debt, just like a restaurant has to clean the plates and tables to continue to provide a service. If we do all the cleaning at the end, rather than as we do service, then its going to take us longer to get ready for the next service. If we don't clean at all then its going to stop service at some point as we will run out of clean plates.

Operability

When a customer spilled a drink it used to cause all of service to stop as we tried to clean it up, now, whilst it disrupts what we are doing we are getting pretty good at cleaning up and learning how to prevent spills when we can.

New features and release cadence

Adding a feature is like changing a menu. We can either incrementally and constantly change our menu dish by dish, or every 6 months completely re do it from scratch.

Sticking to a plan vs having a strategy

At the moment we detail all our dishes and laminate our menus, this makes them expensive to change, which means our customers only come back to us every 6 months and sometimes if we can't buy the ingredients we have to take things off the menu. This is hard for our staff to keep track off and ultimately disappointing for the customer. If we need to laminate the menu now, what if I just put "Fish of the day" on the menu, rather than "Cod with New Potatoes" that way I can adapt to the ingredients available. Of course ... it would be nice to not have to laminate the menu ... but one step at a time... :)

Build/buy decisions

At the moment we are a small, bespoke, fine dining, local, restaurant, we can decide our menu just the week before, and use the fish that is fresh that day. However, this is expensive to run and our prices our high but our customer's love us and keep coming back... we could become more like McDonalds if you like... we could get really good at churning out the same kind of thing over and over... we'd need a 'set menu' that didn't have much variation, but we could buy in bulk and save loads of money as we could probably get lower skilled people to run the restaurant itself. People love McDonalds, but they love it differently to us. The trade off is, if you ask us to significantly change our menu, it might cost a lot of money as the specialist equipment we have bought in might not allow us to cook the new menu. Sure, we could crank out a new burger, but if you want us to do fine dining, we don't stand a chance. Do you want us to be Harvester maybe? Some variation in menu, and maybe once every 3-6 months we get to re-think if the menu is right for our client base?

Legislative requirements

Understanding data security is like understanding food hygiene requirements. Would you eat at a place that had a hygiene rating of 1? You might, if it was cheap ... :S ... Sure. Anyway, theres some stuff we have to do to meet legal compliance, not negotiable, but how we do that, if we centralise that behaviour and don't train all our staff to understand the impact, we might get a one rating and its going to significantly impact the trust placed in the restaurant by the customer.

And many many many more...

One of my favourite past times is stretching an analogy, so please feel free to stetch this one away and let me know how you use it :)

How to move to #noestimates

defmyfunc — Mon, 01 Jul 2019 20:40:00 +0000

So you want to do #noestimates but some people in your organisation can’t live without their estimates… how do you convince them otherwise?

But first… before we continue…

Some important things that #noestimates doesn’t solve

The time between the time your stories are created and when they are implemented is important. The greater the time between those 2 events the worse your estimates would have been, but also the worse any forecasts are going to be. If stories are generated more than a month in advance of the work… its going to be tough.
Understand what done means for you… Ideally done means “In production being used by our users” but this may not necessarily be the case for you, your team and most importantly your process control board (Scrum/Kanban/whatever).
Often, during estimation sessions the real value wasn’t in the act of estimating… its in the act of the whole team being in a room discussing the scope of the work. #noestimates does not mean that other important activities around scoping and alignment do not happen. It just means that we are replacing estimation with forecasting. You will as a team still need to make time for these other activities.

OK… now the answer…

Timebox all the things!

I love a good timebox. Now, a lot of agile development teams will be used to timeboxes when it comes to things like spikes, where the main objective is learning. Well guess what? Lucky you! It turns out the main outcome in product team is the learning :)

Build -> Measure -> Learn

So timebox!!

As a story is solidifying, I ask the team the question… Could you get this to done in 5 days?

I re-iterate that done means:

In the hands of our users
This is elapsed not working time (please consider meetings, etc)

If the answer is ‘Yes’… then Yay! Well done! Whether this story will take 30 minutes of 4.5 days DO NOT THINK ABOUT ADDING ANYTHING MORE TO IT. ITS PERFECT AS IT IS.

If the answer is ‘No’ we break the story down.

Now sometimes… a story is resistant to being broken down to less than 5 days elapsed. Lets say it relies on an external team that can’t be relied upon. If we genuinely feel this is the case, that it can’t be broken down, then we leave it alone. But I make sure the team assess categorises the problem first.

Breaking things down further

Often when problems are resistant to being broken down further it is because of one of the following things:

It is a problem that needs approaching in a different way
It is a problem that has dependencies that can’t be relied upon
It genuinely is a problem that can’t be broken down

It is a problem that needs approaching in a different way

Take a look at my post on Releasing value in software development for different approaches to slicing work that may allow you to think about the problem differently.

It is a problem that has dependencies that can’t be relied upon

Dependencies that heavily rely on asynchronous communication, especially across teams and organisational boundaries that cannot be relied upon, are tough. Its probably a little bit of anti pattern but I often split problems into tickets along the following lines:

Outcomes
Orchestration

Where an orchestration ticket is something along the lines of “Organise a meeting with InfoSec”, “Arrange conversation with key stakeholders”. These tickets are allowed to take longer than 5 days. Orchestration tickets often signify a key dependency and a high amount of them could highlight a tricky time ahead.

What is important is that there is an outcome ticket that follows immediately from an orchestration ticket. Often there will be one before it as well if there is some collateral needed for a meeting.

To aid in understanding these across the team, I often create “Logical Scope” models for the epic/group of tickets/ticket/whatever.

Logical scope model

A logical scope model is a lightweight mind-map of the dependencies and outputs required to consider the ticket complete.

A nice one often looks like this:

A simple logical model with external dependencies

A not nice one with lots of gnarly dependencies and/or outputs may look like this:

A complex logical model with external dependencies

The point of this is to help categorise problems. If lots of dependencies need to be aligned, then you had best get working on them! If lots of outputs need changing for the ticket then this could be a smell the the problem needs more thinking. You can get a good understanding of the complexity of a grouping of tickets/epic/whatever by merging the logical scope models of the individual tickets. This can really help with prioritising of tickets as it allows you to work through those with challenging dependencies first.

It genuinely is a problem that can’t be broken down

Sad. Face. Where this is the case I often think about what I could learn that would allow me to break this problem down more and spit the ticket then into at least 2. One about the learning, the other about what happens after the learning and then go again.

So I kind of lied

This won’t help you get to #noestimates automatically, Sorry. What it will help with is that you will get better at predicting and be honest about what is a wild prediction vs what is a decent prediction.

At less than 5 days, especially as elapsed, to run the “build measure learn” loop we actually get pretty good at predicting. The team becomes much more realistic about what it can and can’t do and it also gets a lot better at breaking problems down.

As a bonus the team also gets better at experimenting to learn the right answer rather than jumping to a big design and development piece.

Now, I appreciate, that some things will go over 5 days. Thats natural. But somethings will be a lot lot lot less than 5 days (hopefully!)

Combine this technique with a switch to forecasting and predictability will dramatically increase.

Tech lead Bruce-y Bonus

As a Tech Lead the nice thing about this is that you get a clear framework of autonomy and intervention with your team. On day 3, I can have a chat about where they are up to, what tasks they have left to do, how confident they are about getting through it and what their feel for the size of this is now they have learned something. Day 4 we can have a proper chat about what value can be delivered out of the ticket and what value can’t. Day 5 we split the ticket, ship what can be shipped and revert what cannot be. In the worst case, we have achieved nothing and only wasted 5 days. In all other cases we have probably learnt something very important which itself can be used to inform the backlog. This doesn’t replace our normal conversation of course and isn’t a rigid framework :)

It takes time

I find it takes somewhere between 1 and 3 months for this to improve things for a team, especially when used in conjunction with forecasting. It often takes 3-6 months for stakeholders to stop asking you to estimate things and start letting you forecast things.

The views in this article are my own and are not necessarily endorsed by my employer.

Being a little more QA

defmyfunc — Mon, 01 Jul 2019 20:40:00 +0000

As a consultant, after the stakeholder shenanigans have been completed, often the first thing I do is go talk to the QA community. The health of the QA community within an organisation is often a key indicator as to whether the organisation is going to make it through whatever we are being asked to do. QAs regardless of whether they are constrained by the system in which they operate, have to understand the dependencies and processes that gets working software to the user, even in the most siloed of organisations.

It's my opinion that the future of software development, lies in everyone becoming "more QA", in the way they think as well as the activites they perform on a day to day basis. As machine generated code moves up "the stack", more and more of our "development" responsibility will lie in defining, observing and validating the qualities of the system and its behavior rather than typing the letters that make it work. In effect, the software developer of the not too distant future... is a QA.

(overly) Simplified Layers of Concern for a "Developer"

Whilst this is overly simplified model (perhaps to the point of being useless... I actually don't think it matters if the model is correct or not necessarily... just that you could create a model for your world based on this generalisation to understand what is happening in your world... phew :S), generally, as a developer for a service provided over the internet, on something that is user facing you have the following "stacks" to deal with when you are attempting to develop value that can be released to your users:

And here is a copy of that information in tabular form:

Value (what we are trying to achieve)
Humans (the humans who agree on what needs to be done)
Application (the software)	Infrastructure (what the software runs on)
Business logic (the new thing that is being created)	Qualities of the system (the qualities we want to see in the system)
Orchestration (the things we need to glue together to release the new thing)	Orchestration (the things that glue together to make that quality)
Configuration (the values we need to change to make the software build and run)	Configuration (how those things are set up to make that quality)
Compilation (the steps performed to change human readable code into machine readable code)	Shared software (what is running on those things)
Compiled code (the code the machine run)s	Firmware (the software baked into the hardware)
Runtime execution (the act of running the code on a machine)	Hardware (what runs those things)

As time has progressed the ability of machines to automate/generate/provision/whatever, those tasks involved in those processes has increased dramatically. At the moment, as humans, we probably meet machine half way ... whereas in the past we have been a lot closer to the bare metal. As a % of 'total humans involved in the development effort' there are less and less humans working further down the stack, and more and more humans able to focus higher up the stack. The theory being ... the further we are abstracted from the underlying mechanics, the better able we are to focus on generating value for our users ... as opposed to working on "enablers" that mean we can then focus on creating value for our users.

Its probably worth mentioning Wardley Maps here ... because hey, thats what all the cool kids are doing to visualise this progress.

We have to get better as developers at this key skill of making the systems and services we develop more observable, especially in cases where we have little say over what the actual code is that produces the effect we are looking for. In the short term, the poor long suffering QA will thank you as you make the qualities and the behaviours of the system (intended or not!) observable. In the long term, you will thank yourself by helping yourself develop the skills required to discover the unknowns of your service and its behaviours and its impact on your users.

This is only going to get more apparent as key technical enablers allow Machine Learning generated systems to become more and more prevalent. As I wrote here, if we can't make these systems and services observable, we can't make these systems and services accountable which is going to be a key requirement of any decision an algorithm makes.

Its why I love tools like Honeycomb.io and the changes it will bring as we really start to dig into what this means for our profession and the future of observability tooling.

I wrote this because of the inclusion of Wardley Maps and Honeycomb.io in the ThoughtWorks Technology Radar and wanted to start articulating why I find them valuable.

"The uncomfortable world of constant discovery" or "the difference between Delivery and Product teams"

defmyfunc — Wed, 12 Jun 2019 13:43:38 +0000

As “agile” and “product” development permeates more and more of mainstrain software development, I wanted to write about the prevalance of two team archetypes that appear (when things are functioning well… dysfunction is a whole other topic!). Outwardly, they appear to be made of similar roles, can actually end up with very different outlooks to delivering that software and are useful in different situations. This can make comparison hard and explaining the difference between the two difficult and less easy to understand.

In my experience during a ‘transformation’ (digital/agile/product/etc) there are many stages organisations, teams and individuals go through on their journey. On that journey, to what I would describe as a ‘product’ teams, a lot of teams and organisations get stuck in ‘delivery’ teams.

Firstly, lets take a model for the activities involved in defining and delivering software. At ThoughtWorks we often use the “double diamond”

An image of the “double diamond” approach.

Delivery Teams

The first step on your journey is probably to move a certain population of your users from role based teams to a form of cross functional team. What I would describe as a Delivery team. The first roles on this journey are often Delivery, Dev and Test followed by BAs, Product then UX. Which means you have a team who can effectively execute a solution.

Delivery teams can effectively work in the ‘execute solution’ phase

If a team stays in this phase or this is considered the “end” of the transformation things can get very difficult. The Delivery team don’t really understand why they are building what they are building, even if they become very good at building it. Pressure to blur the boundary back towards the right increases so that teams don’t ‘build the wrong thing’ again so processes are setup and theatre between these two sides increases. The “define strategy” phase can happen months if not years before the “execute solution” phase meaning more and more boundary crossing and more and more pain when the solution is actually explored.

How pain effects the Delivery team when the two diamonds are seperated

Good Delivery teams also understand the systems they build and how they fit into processes. They understand what the system is trying to achieve for their user. They can run, maintain and expand those systems.

Delivery teams understand how the user is affected by their solutions

Product Teams

Product teams can take responsibility for the whole “double diamond”

Product teams cover all the activities in the “double diamond”

This can be quite an uncomfortable step for an organisation to take on its transformation journey. This is because all of a sudden we are facing into the chaos of the unknown and forcing the organisation to deal with the fact that a lot of what it did before was theatre to give the illusion of certainty. However, this step is the most important and the most valuable to organisations.

Often we describe this as “becoming user focussed”.

Good Product teams understand what their user wants to achieve and create a holistic process sympathetic to them and their needs as well as those of the organisation.

Product teams focus on what their user wants to achieve

When I work with clients I often have to split these two diamonds up into “Discovery” and “Inception” phases. This is to allow for the necessary cross pollination of skills and domain expertise to happen. However, within a genuine product team, these diamonds should be happening constantly. At a micro level within tickets, at the macro level on the roadmap and in the middle with delivery of a feature.

If we are using something like a Lean Value Tree to understand our work, then its important that a Product team can get through a “bet” in around a month.

From definition to execution in less than a month

This means significant differences to how most organisations understand and break down problems that take multiple years to resolve which can be very hard to handle at an organisational level if it is not setup to cope with this change.

Summary

Transformation is a constant journey, and so its useful to be able to understand where you are and where you want to be. If you find your self ‘stuck’ in a Delivery team and unable to change things its important to understand whether this is intentional or by accident. You can then start pushing in the direction you wish to go.

In reality Delivery teams should be highly transitional (but often aren’t) and should either be working towards becoming a Product team (because the situation is such that this is valuable) or if there work has genuinly already deeply understood be working to understand how they can transfer their work to a commodity provider.

Appendix

Signs you are in a Delivery team

Often created by the first few steps on the agile journey, this team will have close collaboration between Delivery, Dev and Test. Dev’s will understand the importance of quality and will take responsiblitly for automated testing. Test will take responsibility for exploratory testing and will be able to generate risk based approaches to testing any solution or service.

Teams are responsible for a system/service or set of system and services. Often a complete user journey is spread across multiple teams. Operational roles have been taken on by the team, they will ideally look after their service 24/7. The team can help people understand the problem of technical debt. They may have 10 or 20% time in which to explore new ideas which are shown to people higher up in the organisation. They may have a good relationship with UX roles especially the more design orientated roles. BAs are often responsible for helping the team understand requirements and by the time a developer or tester first looks at a ticket a lot of work will have already gone into it.

User research often happens months in advance. The dev and test roles rarely if ever go to events that involve their user. The roadmap is often defined past a year in the future. The team have very little say over the large piece of work they are working on, but as that works break down they may have more and more say over how it is achieved.

The board used to visualise work is often split into multiple swimlanes so that high priority items can flow quicker. The columns are often based on roles or environments the code has to pass through. It focuses on ‘development’ and ‘test’ and so does not contain the entire value chain, often missing out the ‘ideation’ phases of work and the ‘observation’ releases meaning that some work is happening ‘off board’ and can be hard for the team to understand. A ticket is often “done” but not “done done”. Tickets often represent a slice of a set of tasks that a single role can achieve rather than a piece of value delivery to a user. Retrospectives often focus on the impact of other teams on this teams work which means sometimes retros can fall flat as the ability to change is limited.

NFRs/CFRs are often considered but then treated somewhat seperately by the team. Sometimes this delineated by environments (ie performance testing) and other times it is dileneated by dealing with another team (ie security or documentation tickets). Often the team has to produce documentation after the fact for some gatekeeper roles including (but not limited to) Architecture and Security. They may have to coordinate releases with other teams or be part of a schedule (like a release train).

People are often paid different amounts depending on the role within the team and their individual performance reviews are not intentionally linked to the teams goals. Individual contributers often generate prolific amounts of output, creating knowledge silos and reliance on specific team members for specific tasks.

Signs you are in a Product team

They have complete control of their own backlog. The backlog is no larger than a months work and it is relentlessy pruned and prioritised. Because of this retrospectives focus on what this team can change and actions are prioritised against all other work. The roadmap is only relatively certain for the next 3 months. The roadmap consists of a series of ‘bets’ rather than requirements and it is understood that each ‘bet’ can be scrapped as its picked up by the team.

Teams are responsible for the service design, delivery and operation of an end to end process or user journey with no handovers to other teams in that process. A deep understanding or production is encouraged across the team, and the events that happen in production regularly help form the backlog.

The team is free to use solutions built by them, other teams within the organisation or external providers.

At particularly gnarly points in time, especially were something is not very well understood the team often ‘mobs’, this can be on any type of task not just “typey-typey”. Alignment of vision is valued in the long and short term, the team actively work to compromise the tradeoffs between upfront design and delivery. Upfront design is mostly done ‘Just In Time’ and is immediately followed by attempted delivery of that design. This alignment is valued by the team as a way to achieve autonomy. Any technical debt can be assigned value in the same way as a feature and so can be prioritised accordingly.

The work is visualised on a board and the columns represent value add activities. Where they don’t the team are able to change their process as they need. There is only 1 swimlane. Everyone on the team understands how to get a ticket from begining to end and regularly takes responsibility for this. The board includes “ideation” and “release” phases. Releases are actively monitored and observed to understand the effect they are having. Releases happen independently of any other team.

Tickets often represent a piece of value to a user. Any key gatekeeper teams (like architecture and security) are actively involved during the key discovery and solution phases the team goes through. This means that NFRs/CFRs are integral parts of the individual tickets. It is unlikely that a ticket could be solved by a single person. The person responsible for getting a ticket through the board will have to bring in a variety of roles at different points to make it happen successfully.

User research happens consistently and regularly and the team are actively involved. The research that is happening is relevant to either something that has just been released or something the team wants to do next.

People are often paid similar amounts regardless of role within the team and their individual performance reviews are intentionally linked to the teams goals. Whilst individuals bring key skills to the team that no single other person brings, the culmination of skills from the rest of the team can cover that individuals abscence. Where knowledge silos occur, the team actively work to break them down.

Links used in article and as inspiration

The Double Diamond: Strategy + Execution of the Right Solution

https://medium.com/media/3ca50a105cac63b20d5b0952d7d84886/href

Organisation Antipattern: Dual Value Streams

Serverless Deployment Patterns and Techniques

defmyfunc — Mon, 20 May 2019 13:04:13 +0000

As serverless has evolved and grown there have been a plethora of architectural patterns growing around to support the problems we are all trying to solve. Key to a strong architectural pattern is an understanding of how those pieces are broken up and deployed to live. Without that, the pattern is useless. I’ll explain the patterns I’ve tried before and which one is my current default approach.

Its very AWS specific, but I think the core tenets can be applied to any serverless deployment. Sorry about that if its a problem.

As ever, comments, thoughts and other approaches are appreciated.

Terminology

Pipeline - Something that manages the movement of code in a repository to being run by your users, often has phases such as:

Test
Build
Deploy to Int
Deploy to Live

1. The serverless monolith

Where you have one repository and one build pipeline for it all! Infrastructure and application. I don’t know if I need to talk about the difficulties in this one. If you are starting from scratch and want to learn fast… the right way to go :) But at some point you will probably want to move on from this.

Example pipeline showing deployment of all infrastructure and application in one pipeline

2. The separation of infrastructure and application

Where the infrastucture (using terraform or cloudformation) has one pipeline and the application ( most likely a bunch of lambda’s) that use that infrastructure have another.

You might still have one repo at this point and you might be doing something funky in your build process to make sure changes only trigger the appropriate pipeline or you have 2 repos, one containing the infrastructure and one containing the application.

This is probably closest to traditional (non-serverless) architectures. Where the infrastructure deployment is responsible for things like API Gateway, Database creation, Queue creation, the lambda ‘placeholder’ but not the lambda itself.

This means you can start to independently deploy your application from your infrastructure. Often there is “connascense of timing” issues, where things need to be pushed through the 2 pipelines in specific orders, or perhaps certain steps have to be re-run once another one has finished. The lambda placeholder in the infrastructure is the biggest source of pain, as the lambda can often end up in a weird state, unless you do some engineering to make the infrastructure pipeline deploy the latest version of the lambda.

Whilst the application can be released independently as long as you aren’t changing the database tables/queues/etc it relies on or what not it can still be a pretty painful place to live.

Example pipelines showing seperation of infrastructure and application deployments

3. Every lambda has a pipeline

Where each lambda has its own pipeline and deploys the infrastructure alongside. Often this co-incides with the move to a framework like serverless to manage the infrastructure the lambdas are dependent on. You are probably still using terraform or such to manage some other bits and bobs of your infrastructure but this can be minimised by using serverless.

This gets rid of a lot of your “connascense of timing” issues and can really help speed up development. Isolation around lambdas mean that developers can feel a lot more free. And whilst development can speed up, often releasing can be slower as the application gets more complex. This is because the isolation of individual lambdas hides the coupling across the application. It can get difficult to manage this across your estate in your multitude of buckets, queues and lambdas.

This often coincides with an increase in the number of repositories being produced and used across the estate (by default, one for each lambda), again, unless engineering effort is put in to challenge this. This can have a massive impact on the “locality of connascence” which can be detrimental to the health of a system unless properly managed.

Example pipelines showing individual pipelines per lambda and some related infrastructure

4. Each “Trigger” infrastructure and its dependant lambdas have a pipeline

A common pattern when moving to serverless that we have seen is to have a piece of ‘trigger’ infrastructure, ie: a queue, a db, linked to one or many lambdas, and I think (although I have no basis for this think) that this is some of the inspiration behind the ‘serverless framework’ and how it allows you to define some infrastructure pieces that relate to the lambda.

This is important because often the ‘trigger’ infrastrucure and the dependentant lambda have impicit or explicit contract. Leaving aside the AWS part of the event payload, the dependent lambda(s) often understand specific event payloads from that particular piece of infrastructure, as such they often end up very tightly coupled and so it suits to be considered a single deployment.

Whilst we appreciate that this can’t be defined as a hard and fast rule… there may be some points where we need to make a decision about whether something is “more coupled” to its trigger or the thing it communicates with…

What we are finding is that more often than not, this approach encompasses a ‘domain’. Sometimes those domains are relatively small and sometimes they are relatively large but by taking this approach as the default and then considering our options we are finding it a sensible boundary of deployment.

This can be done in many ways including terraform, cloudformation or serverless, the important thing to try and get right is the boundaries and groupings of the lambdas and their associated infrastructure.

Example pipelines per “trigger” infrastructure and its dependant lambdas

Summary

As serverless solutions get more and more common there will be many ‘best practices’ to architecting and deploying solutions on serverless that are created and some will last and some won’t. Its too early at the moment to say and undoubtdly in 6 months the landscape in this are will have changed, however, at the moment our team finds grouping a piece of ‘trigger’ infrastructure and its associated lambdas a useful way of keeping architectural boundaries of our systems clear, whilst making live releases as painless as possible.

Comparison image of pipeline and deployment types

How I use CRC (Class Responsibility Collaboration) Cards every day (Part 2)

defmyfunc — Wed, 28 Mar 2018 16:04:26 +0000

In my last article (How I use CRC (Class Responsibility Collaboration) Cards every day (Part 1)) I explained how to introduce the use of CRC cards into your processes by the back door. In this article I am going to explain how to start using the ‘CRC Family’ of techniques to do ‘just enough’ upfront modelling and designing to give your software development process predictability and sustainability.

I’ll hopefully also explain why I think that whilst the ‘CRC Family’ of techniques is no silver bullet, it may be one of the single biggest improvements you can make to your existing process. Done properly, it dramatically improves the alignment and understanding of your software throughout your cross functional team and by extenstion, across your organisation through all roles, leading to more predictable, sustainable software delivery.

I make a lot of assumptions in this article about how people work. My experience is heavily weighted towards autonomous, empowered, engineering led teams within varying sized organisations who have either made or are making the transition to “Tech@Core” organisations. YMMV.

The CRC Family

In this part I will explain two directly related techniques that I know as ‘Naked CRC’ and ‘Physical CRC’. Generally when I say to people: ‘CRC it for me please’, I mean, ‘Do a variation on one of these 3 techniques, please’, ie. I use them interchangeably and use whatever technique seems appropriate or necessary for the occasion.

What is ‘Naked CRC’?

‘Naked CRC’ is where you use the CRC cards, but don’t fill them in, you leave them empty. You’ve probably done something similar to this in your every day life even if you have never done it in software: ‘If the salt shaker is the shop on the corner and the pepper is you then you turn left’. This technique is the ultimate in lightweight modelling. You use face to face communication to describe the collaborators and the responsibilities rather than written words. I am a big believer in ‘Code that fits in your head’ and Naked CRC really tests that to its limits.

What is ‘Physical CRC’?

‘Physical CRC’ is where you use the cards you have made and hand them out amongst your team. You then as a team stand up and play through the scenarios you have, where each person pretends to be one of the objects and assumes its role through the scenario. This technique is great for team alignment and making sure everyone has the same understanding.

Why the ‘CRC Family’?

Purely anecdotely, I have found that these 3 techniques more than anything else have helped me deliver software predictably, that is simple and understood by all. These 3 techniques, used appropriately during the development lifecycle force you to work within the constraints of your team and your business. They act as a guiding force (almost a fitness function for work… hello evolutionary architectures!) for the size of problems your team can tackle, how they break those problems down, how they communicate to each other and the outside world and how they go about their work. If work cannot be explained and understood by the team using these techniques then its probably not work the team should be attempting to do given their current understanding/state, ergo, do more spikes, break the work down a bit more, find someone to help, whatever, whatever, whatever until you can use these techniques to explain your problem.

Leveraging CRC Techniques to do ‘just enough’ upfront design/modelling

So now you are using CRC cards at the end of your process(es) for retrospective type activities to better understand your current state and shape your future work. I’m now going to explain when I typically use them up front to guide the design of the software.

All these scenario’s are what I consider ‘ideal’ scenarios. ie I work in teams and I don’t get to decide all the rules (damn teams! you know you want to be ruled like a king) but when a problems occurs in the teams processes and I believe one of the CRCs can solve it, I suggest these type of actions at retrospectives as possible solutions.

Scenario 1: Ticket Kick Offs

I have found many benefits to running that Naked CRC is done during the kick off by one of the developers on the ticket. This should be done in such a form that every one at the kick off can understand it and the output. (Where ticket kick off’s look something like a “3 amigos”).

Example , here are 3 blank cards:

3 blank CRC cards

I might place them down in a specific order and as I place the first one say “This is the controller. It is responsible for orchestrating the View and the Model. It is also responsible for firing stats.” As I place the second card I might say “This is the view. It is responsible for managing user interaction and communicating those events to the controller. As the user clicks the save button, that will raise an event which the controller will understand and pick out the new bit of data we are interested in sending in the stats we are already currently sending.” As I place the third card, I might say “This is the model, there is no new data required in the model, however, as the property you are asking to be sent is an amalgamation of two fields, we are going to do that in here.” All 3 down, I might say “This is the change we are intending to make to support this ticket, lets play the scenarios through and the AC’s and see what the scope of the change is.”

As you then create the ACs, you judge the solution against them and change the solution accordingly. You may of noticed that I didn’t mention any tests, so as a minimum, in this example, I’d be talking about the test code that needs to change to support it. But obviously, IRL, you go with the flow and change accordingly to support what you need to support.

Outcome , during the process hopefully you will all be getting a shared understanding of the work required to complete this ticket. The first thing that should become obvious very quickly is ‘is this ticket too big?’. If everyone in the ticket kickoff can’t keep track of whats going on, then you probably have a ticket that’s too big. Excepting that circumstance, once everyone is agreed on what the ticket involves and changes made to the suggested implementation via the CRC process, we all go our seperate ways, everyone has a much more concrete idea over what is going to be done and what impact this ticket will have on their role.

Positive side effects , I often ask the developers in the 3 amigos to make the outcome concrete and visualise it as they work on the ticket, ie. switch to CRC, so they signal their intent to the rest of the team on the change they are going to make. As they do the work and as things don’t quite pan out how everyone expects I would expect this to be reflected back to those involved in the kick off, helping them change their work as necessary. Significant differences should be used as a smell to ‘re-amigo’ if necessary.

Why Naked CRC over CRC? It’s more lightweight, it proves understanding and you have to do just enough ‘pre work’ going into the ticket to make it work and also . It’s less imposing than CRC to people who are ‘less technical’ (whatever that means), it forces collaboration and aligned understanding across disciplines as people are more likely to challange, it encourages involvement and participation because if you switch off during the ‘3 amigos’ . And finally, in most scenarios (though not all!) it acts as a natural limiter on how ‘big’ and ‘complex’ a ticket can be because it forces ticket sizes to fit the size of the teams head. Finally, I am a believer of the ‘to prove you understand a topic, teach it’ and Naked CRC above all other techniques requires a level of understanding of the problem, as well as being an excellent aid for teaching.

Scenario 2: Epic Level Kick Offs

Firstly, I generally use CRC to help design the solution. I generate new cards for all the initial affected parts of the system(s), I then work with the team to add the new Collaborators and Responsibilities to these cards as required. Once we have a candidate, I then use Physical CRC to play through all the current known scenarios.

Give every team member a card and get them to act out their part in the scenario. This might involve saying things like: “I am the View, when a user interacts with the button I fire the event with this data on it”, “I am the Controller, I am listening for the ‘Button Clicked’ event, when it is fired, I parse the data from the event looking for the new data item and calling the stat library with the required data’.

Outcome , everyone on the team should have an understanding of the desired solution because they have been a part of making it as well as having played through as a team, all the known scenarios.

Why Physical CRC over CRC? Physical CRC really highlights any ‘connascance of timing’ issues the solution has. Its often easier to spot other things (like coupling, too much responsibility, etc) by how much one person talks during the scenario play throughs than it is in on paper. People ask questions when they are forced to act something out that they don’t when they are reading.

Scenario 3: Debugging

Much like a rubber duck any form of CRC done during a debugging exercise is worth its weight in gold. Even if the ‘solution’ is easy, its often worth playing through a CRC to gain the right level of abstraction helping you spot things you haven’t thought of.

Summary

Hopefully I have convinced you over these 2 articles that CRC is worth a shot :) I made some bold statements and I hope you can see the value. Unfortunately all my ‘evidence’ is anecdotal and based on experience, not facts.

Regardless of whether you belive me or not enough to try I think CRC is one of the great team alignment tools in the ‘Tech Leads’ toolbox. And through alignment come a lot of other good things. CRC massively improves code ‘understandability’. Not only does it help people come to terms with the current code, it also helps the team align over what good looks like for them. It helps that team figure out ‘what fits in their head’. One teams good is another teams bad after all. But not only that, CRC more than any other tool shines an uncompromising light on what you don’t know. It helps the team figure out what they do and don’t understand, what things they need to investigate and spike, where the dragons are.

It also does the same for complexity. Allowing the team to understand the complex parts of their system(s) as well as allowing the team to express what is and isn’t complex for them, abstracting things at the right level, for them.

In a future article I’ll talk about how these techniques (and others) relate to the cynefin framework to help teams make decisions.

How I use CRC (Class Responsibility Collaboration) Cards every day (Part 1)

defmyfunc — Fri, 23 Mar 2018 19:20:58 +0000

On some discussion with colleagues I realised that a technique I took for granted is often not used by others so I wanted to do a write up of how I use them to further the general ‘make my team good at the software’ remit.

You can see Part 2 of my thoughts on CRC cards here: https://medium.com/@defmyfunc/how-i-use-crc-class-responsibility-collaboration-cards-every-day-part-2-8f9a6f79f7c2

I first read about these in the White XP book but I didn’t really use them until I was introduced to them by Kevin Rutherford when he worked with us at the BBC as part of the Software Craftsmanship program we had there. So massive props to him (https://silkandspinach.net/).

The views in this article are my own and are not necessarily endorsed by my employer.

What are CRC Cards?

Class Responsibility Collaborator (CRC) Models: An Agile Introduction

The only issue I take with that description is the name. This may sound like semantics but I have tended to find the technique more successful if you think in terms of objects (ie pretend the system is actually running) instead of classes. But given my internal biases I think thats true of pretty much everything in software development.

When do I use them?

The answer to this is … all the fricken time. As a ‘Tech Lead’ they are invaluable to me as they allow me to abstract myself from the code but still offer value to my team by exploring problems with them.

Often though, if you have never used them before its hard to introduce something at the start of your process. When I first started using them with my team, I started by introducing their usage at the end of a task/story/epic/whatever, until the team becomes comfortable with them and then you can start using them at the begining of your processes as a design tool, rather than at the end of your processes as a reflection tool. Anyway… onwards!

Scenario 1

You/Your team have just finished a task. Woo hoo. Good job! Now is the time to reflect. Write up a CRC card(s) for the task you have just completed.

Now compare the before and after state, did you add a new collaborator? Did you add a new responsibility? Was it just a change to an existing responsibility? If we added something new was it genuinily the right place to add it?

As you practice asking your self and the team these questions, you’ll start to build the mental model of the part of the system you are working in.

Scenario 2

You/Your team have just finished a story. Woo hoo! Good job! Now is the time to reflect. Write up a CRC card(s) for the story you have just completed.

On the assumption that a task is smaller than a story… then this may involve writing more cards. You should have a slighly bigger picture to work with and a genuine view on the end to end value for a particular thing. This is a great time to reflect :)

Look over the cards, ask yourself/your team the same questions as before… anything new? what was changed? was there a better place to put it? You can probably now also start asking questions like… did we introduce any new coupling? (or connascence http://connascence.io/) was this the right thing to do? Did we intend to do this or is it an accident? Could we have done it better? How might we refactor this to improve it? Did this story have a wide impact or narrow impact? Is that what we thought would happen?… etc… etc…

Scenario 3

You/Your team have just finished an epic. Woo hoo! Good job! You lot are awesome. Now is the time to reflect! Write up a CRC card(s) for the epic you have just completed.

On the assumption that a story is smaller than an epic… then this may involve writing more cards. This is probably the biggest picture you have had to work with.

This is a great point in time to run a ‘technical retrospective’ with your team. Have cards written up for all the objects modified/created/deleted by your epic and sit down as a team and discuss. Did you build what you thought you would at the begining of the epic? If not why not? Are there any changes in there that where completely unexpected? What would you do to refactor it?

For instance, on one of my teams I did the following in advance based on a piece of work we had just completed:

And then asked the team to collaboratively refactor based on our teams technical strategy. They did this:

They consolidated genuine duplicated logic, making one of our controllers more like other controllers in our application and reducing our dependency on some objects.

This not only acted as a knowledge share with those members of the team who hadn’t worked on the piece, but also meant the whole team could be actively involved in considering its future direction and everyone had their opinions taken into account.

In my next piece on CRC cards Ill write about how I use them upfront to aid design decisions.

Top Tips

Write a card for before and after the change and then compare.
Write the cards based on the genuine code. Do not base it on some drawing or some memory you have of what it does or whatever. Open the code, read the code line by line, write it out as you do it. Make sure EVERYTHING is in there.
In my view writing a good CRC card means introducing a level of abstraction from the code. Whilst the list of collaborators might be a 121 mapping to the objects, the responsibilities should be reflective of your domain NOT necessarily your code. What do I mean by that? Well lets say you have a ‘EmailAddressValidator’ object, its responsibility might be ‘validate email addresses’ however, it might take 3 functions to do that. Responsibility > Implementation.
Throw them away. Often. If you can’t recreate these cards quickly then its often a code smell that your code is too complex — see ‘Software that fits in your head’
If you can’t fit all the responsibilities and the collaborators on one side of the card then its often a code smell that your code is too complex — see above.
If you are doing any kind of refactoring sessions, put the responsibilites and collaborators on seperate slithers of paper that you can stick with blue tack to the cards. Encourage people to move the bits of paper around.
Rememeber to include ALL THE THINGS. Common things people don’t include but you should include include:

Configuration changes
Changes to test code
DB changes
Infrastructure