DEV Community: Alexandre Quercia

Learning Test - How to test adapters with behaviour?

Alexandre Quercia — Tue, 02 Jan 2024 14:21:35 +0000

"Learning Tests" is a kind of "Characterization Test" for third-party API.

My learning journey

I had a dilemma:

I need to be confident about the logic of a database “Adapter”, with test, but this kind of test are slow. There are not “plumbing test” as they test a behaviour. And this behaviour is a behaviour of a third-party API.

And I had many questions

How to name the test suite that holds “Adapter” behaviour?
How to keep “Unit Test” ran very fast?
How to keep “Integration Test” focus on plumbing, without behaviour?
How to keep a sense between test suites?

I read some

I read again the part talked about
- “Learning Test” on chapter 8. Boundaries on “Clean Code” book wrote by Robert C. Martin
- “Integration Test” on chapter 8. Testing Strategies on “The Clean Code” book wrote by Robert C. Martin
- “Humble Objects” on chapter 4. Test Design on “Clean Craftsmanship” book wrote by Robert C. Martin
https://www.google.com/search?q=Jim+Newkirk+learning+test
https://martinfowler.com/articles/practical-test-pyramid.html
https://martinfowler.com/bliki/IntegrationTest.html
https://martinfowler.com/bliki/ContractTest.html
https://testing.googleblog.com/2010/12/test-sizes.html?m=1
https://martinfowler.com/bliki/SelfInitializingFake.html

The result of my learning

The technic I built need to be experimented on real life code base. It is not something that can be done on a "Kata".

What is the technic?

We want that a detected behaviour change on the third party API force us to propagate this changes to all dependent code. In other words, the "Unit Test" suite must fail.

Steps:

Make "Learning Tests" to characterize the behaviour of a third party API, as we expect to use it in our application.
Make "Fake" to implements behaviour of "Learning Tests" This "Fake" is a kind of cache for behaviour.
On the "Adapter" "Unit Test" inject the "Fake"
Execute "Learning Tests" periodically to check the "Fake" validity.

How to make a "Learning Test"?

We’re essentially doing controlled experiments that check our understanding of that API.

It is common to experiment to learn about a library by:

reading the documentation examples
checkout the code and execute in the console using debug tools.

This kind of things should be done on controlled environment. And each learning are saved inside a test case that characterize one behaviour.

When to use this technic?

The "Adapter" have a bit of behaviour We want that our "Adapter" to be an "Humble Object", it is too simple to bother testing. For example, when the "Adapter" need to transform third party API response.
When the third party API is stateful, two different responses for the same request.

In practice

One kind of "Fake" for a database like service have many versions.

Why?

Ours applications should be resilient to failure.
The "Adapter" layer is subject to failure.

The database may be unavailable => OutageFakeDatabase
Access may be denied => AccessDeniedFakeDatabase

Glossary

"Learning Tests" - look at the @taylorsilva's post. He did a nice in practice example.
"Characterization Test"
“Adapter”
“plumbing test”
“Unit Test”
“Integration Test”
“Humble Objects”
"Fake"

Dérisquer, ce n'est pas seulement planifier votre futur

Alexandre Quercia — Mon, 05 Jun 2023 16:29:24 +0000

Vous dérisquez en planifiant votre futur, mais vous oubliez que le futur est imprévisible.

Les prévisions météo sont-elles exactes ?
Regardez vous-même sur le site de météo France. Plus vous regardez vers le futur, moins il y a de précision.

Imaginez...

Chaque jour, vous résolvez un problème.
Vous investissez petit et à très court terme.
Vous répondez rapidement au besoin de vos clients.

Vous vous demandez :

Comment puis-je atteindre ce niveau de performance ?
Comment puis-je améliorer mes chances de succès ?

L’analogie qui suit n’est pas entièrement exacte, mais s’en rapproche suffisamment.

Une fine tranche de jambon, pour dérisquer sans gaspiller

Comment Bill le boucher a répondu au besoin de Martine, la mère au foyer ?

En ce jour ensoleillé de mai, Martine se rend à la boucherie de son quartier.

Bonjour monsieur, je souhaiterais vous acheter un jambon.

Bill lui fait une proposition.

Bonjour madame, est-ce qu'une fine tranche de jambon ibérique, vous irez ?

Martine fait confiance à Bill et accepte sa proposition.

Trois jours plus tard, Martine retourne voir Bill. Elle lui donne son avis sur le jambon.

Le jambon que je vous ai acheté il y a trois jours est trop épicé. Avez-vous un autre à me proposer ?

Bill lui répond avec amabilité.

Chère madame, je peux vous proposer un serrano, c'est le jambon le plus doux que j'ai actuellement.

Martine est heureuse d'avoir le choix.

Merci monsieur, je vous en prends une tranche bien fine.

Bill accueil chaleureusement le choix de Martine

Je vous prépare votre tranche, bien fine.

Bill, à l'aide de son opinel étincelant, découpe verticalement une fine tranche de serrano.
Puis lave son couteau et son plan de travail, prêt pour la prochaine utilisation.

Voici votre tranche de serrano, bien fine.

Bill aurait pu proposer à Martine, du cheval ou un pâté en croute. Alors qu'elle souhaitait un jambon qu'elle aime.

Le métier de Développeur est similaire à celui de Bill qui découpe le bloc de jambon, à l'aide de son couteau opinel tout propre, il réalise de fines tranches pour Martine.

Mais alors, comment dérisquer efficacement ?

Découper finement en tranche utilisable par celui ou celle qui en aura l'usage (on l'appellera, l'utilisateur)
Ordonner les tranches
Livrer la 1ʳᵉ tranche
Vérifier si ça répond au besoin/problème de l'utilisateur
Continuer à la 1ʳᵉ étape.

Frédéric Leguédois nous l'explique en répondant à une question posée lors du meetup de Software Craft Romandie.

Comment est-ce qu'on fait pour gérer les aspects réglementaires ?

Regarder l'extrait vidéo

Allen Holub nous l'explique en 17 secondes

Commencez par parler à vos clients et découvrez ce dont ils ont besoin.

Coupez-en un petit morceau, construisez-le et livrez-le.

Obtenez des feedbacks de vos clients.

Regarder l'extrait vidéo

Bonus, il faut aussi comprendre son besoin

Pour y arriver sans tâtonner ou « prendre le périphérique parisien », le Développeur a besoin de comprendre le besoin de l'utilisateur.

Pour recevoir, il faut donner. Si vous voulez que quelqu'un réponde à votre besoin, vous devez commencer par répondre à son besoin.

Le besoin d'un Développeur est de comprendre votre besoin.

Ce qui vous permettra de trouver ensemble une solution adaptée à votre besoin.

Bonus disruptif, faire confiance

Faites confiance aux personnes pour faire leur travail.

Je vous invite à voir mes écrits, non pas comme une opinion statique, mais comme un flux de pensées attrapées en plein milieu de leur mise à jour.

Stop waiting for backend

Alexandre Quercia — Fri, 30 Dec 2022 16:59:12 +0000

How to remove dependencies between tasks?

Two factories produce two times more products than only one in a period. In software development, this does not apply, and can be worse. How, two programmers in a team, can get closer or even outperform this level of output?

Chapter 1 An example
Chapter 2 Decide a boundary between two tasks to enable parallel work.
Chapter 3 While increase the communication among team members.

This post will not say that your team should focus only on output, as the outcome is even more important.

This post will provide you with guidelines to maximize your teamwork. It will explain the most important concepts to avoid dependencies instead of managing them.

Lets starts in practice

Prerequisite

Given a component with a dependency like the GUI or the database.

In order to test a component with only a boundary definition, we need the ability to mock a boundary and its behavior.

Let's assume the prerequisites are met.

How does all this play out in practice?

Given "A" and "B" seems to be dependent tasks.

Firstly, we ask ourselves the following question:

Is it possible to define a boundary between A and B?

The answer must have 5 minutes timeout.

If it is possible to define a boundary between "A" and "B"

Then, we add a task "C" to implement this boundary.

If no boundary can be drawn yet

Then, merge both tasks in one and starts implementing in pair or "mob programming".

What are the concepts behind this?

Architectural concepts

To enable parallel work on tasks without interfering, use the simple design principle:

High-level policies are ignorant of low-level details.

— Quote from the book "Clean Craftsmanship: Disciplines, Standards, and Ethics (Robert C. Martin Series) (English Edition)" wrote by Robert C. Martin

In practice, this principle is applied by drawing a boundary between components. Therefore, a task needs to focus on a single component.

Boundary drawing

Between high and low level components, a boundary is drawn. It's like an electrical outlet with a male plug on one side and a female plug on the other.

The male socket plugs into the female socket.
The male plug completely ignores what is plugged in.

So with business rules, database and GUI:

The database plugs into the business rules.
The GUI plugs into the business rules.
Business rules completely ignore the database and the GUI.

Look at this diagram, both arrows cross the boundary and point to business rules.

Tips, when drawing a diagram, it is recommended to place high-level rules above low-level details. Therefore, all arrows will point upwards. Then, It is easy to detect a wrong dependency direction.

Let's avoid rebuild the wheel, as Robert C. Martin done a perfect description.

Software architecture is the art of drawing lines that I call boundaries. Those boundaries separate software elements from one another, and restrict those on one side from knowing about those on the other. Some of those lines are drawn very early in a project’s life—even before any code is written. Others are drawn much later. Those that are drawn early are drawn for the purposes of deferring decisions for as long as possible, and of keeping those decisions from polluting the core business logic.

Recall that the goal of an architect is to minimize the human resources required to build and maintain the required system. What it is that saps this kind of people-power? Coupling—and especially coupling to premature decisions.

Which kinds of decisions are premature? Decisions that have nothing to do with the business requirements—the use cases—of the system. These include decisions about frameworks, databases, web servers, utility libraries, dependency injection, and the like. A good system architecture is one in which decisions like these are rendered ancillary and deferrable. A good system architecture does not depend on those decisions. A good system architecture allows those decisions to be made at the latest possible moment, without significant impact.

Boundary crossing

At runtime, a boundary crossing is nothing more than a function on one side of the boundary calling a function on the other side and passing along some data. The trick to creating an appropriate boundary crossing is to manage the source code dependencies.

When a low-level client invokes a higher-level service.

The simplest possible boundary crossing is a function call from a low-level client to a higher-level service.

In figure below, the flow of control crosses the boundary from left to right. The Client calls function f() on the Service. It passes along an instance of Data. The <DS> marker simply indicates a data structure. The Data may be passed as a function argument or by some other more elaborate means. Note that the definition of the Data is on the called side of the boundary.

— Quote and image from the book "Clean Architecture: A Craftsman's Guide to Software Structure and Design (Robert C. Martin Series) (English Edition)" wrote by Robert C. Martin

How to test?

Let's take the example of adding proof to ensure the GUI Controller work as it should.

Testing the controller is trivial, our tests simply invoke events through the event handler interface and then make sure that the controller builds the right request model.

— Quote from the book "Clean Craftsmanship: Disciplines, Standards, and Ethics (Robert C. Martin Series) (English Edition)" wrote by Robert C. Martin

It needs to mock the higher-level service in order to spy and assert on the request model passed. And also stub the higher-level service to tests the usage of outbound data.

When a high-level client invokes a lower-level service

When a high-level client needs to invoke a lower-level service, dynamic polymorphism is used to invert the dependency against the flow of control.

In figure below, the flow of control crosses the boundary from left to right as before. The high-level Client calls the f() function of the lower-level ServiceImpl through the Service interface. Note, however, that all dependencies cross the boundary from right to left toward the higher-level component. Note, also, that the definition of the data structure is on the calling side of the boundary.

— Quote and image from the book "Clean Architecture: A Craftsman's Guide to Software Structure and Design (Robert C. Martin Series) (English Edition)" wrote by Robert C. Martin

How to test?

Let's take the example of adding proof to ensure the database GatewayImpl work as it should.

Testing the database is trivial. You create a suitably simple test database, and then you call each query function of the GatewayImpl from your tests and ensure that it has the desired effect on that test database. Make sure that each query function returns a properly loaded set of business objects. Make sure that each update, add, and delete changes the database appropriately.

— Quote from the book "Clean Craftsmanship: Disciplines, Standards, and Ethics (Robert C. Martin Series) (English Edition)" wrote by Robert C. Martin

The result of the simple design principle

With boundaries between components, we can remove dependencies not bound to a boundary. We can build something like an onion with some levels. But be careful, when there are too many levels, one modification made on the highest level will may require the modification of all dependents.

Tips, try to avoid using more than two levels.

How boundaries emerge?

There are at least 2 ways to get emerging a boundary.

During the implementation
With a dedicated task to draw the boundary

The best way is to start coding and let emerging boundary. Time go on, programmers gather experience and are able to design boundaries before coding using heuristics.

Keep in mind that boundaries must be able to move as easy as changing shirts. This allows for mistakes, and makes them inexpensive. How to allow for mistakes?

How can you prevent harm to the structure of your code if you don’t have the tests that allow you to clean it? And how can you guarantee that your test suite is complete if you don’t follow the three laws of test-driven development (TDD)?

— Quote from the book "Clean Craftsmanship: Disciplines, Standards, and Ethics (Robert C. Martin Series) (English Edition)" wrote by Robert C. Martin

Lets explain the first way.

Boundaries emerge during implementation

The implementation starts with nothing, but it quickly achieves a simple behavior. Each time a new test case passes, the complexity of the production code increase unless the structure is improved. During theses improves, the boundaries emerge.

The first step is to draw the boundary line.

With a function call
With an HTTP request

The second step is to describe the inbound and outbound data that will cross the boundary.

Add an argument to the function call.

Name it well
Add a data type to it

When there are more than 2 arguments, it is time to create a data structure.

I wish you happy drawing, this principle is the first step to increase your productivity. The second step will be to increase the communication.

Communication between team members

How can you make progress on a task if you do not have feedback on about choices made? And how to have a feedback loop without increasing your communication?

Your brain have high difficulties to focus on a single task. As the task switching effect is too easy to practice as your unconscious do it himself.

As an example, while you are working and focus on a task, but someone asks you a question related to another topic. You ask him that you will answer later. When you come back to your task, your unconscious thinks about that question. Therefore, you just lost 20% of your efficiency.

Working together on the same topic, aka swarming, enables better communication and reduces task switching effects.

So to increase communication with your teammate, you need to work on the same topic as him. Therefore, questions will not be on another topic. And you will not be afraid to disturb him.

In practice, all programmers are in constant communication — Open Space or video — in order to answer any questions as quickly as possible or even help to ensure that boundaries are well understood.

This high communication level is the key of adaptation. Therefore, boundaries can evolve over the learns carry out during the implementation.

Summarize in image

And now the conclusion

Even in a monolithic, statically linked executable, this kind of disciplined partitioning can greatly aid the job of developing, testing, and deploying the project. Teams can work independently of each other on their own components without treading on each other’s toes. High-level components remain independent of lower-level details.

— Quote from the book "Clean Architecture: A Craftsman's Guide to Software Structure and Design (Robert C. Martin Series) (English Edition)" wrote by Robert C. Martin

☯ The yin

Pairs to implements the highest-level component
- Draw boundary line
- Define how to cross the boundary
Dispatching the work force: enables parallel work
- One guy work on the Database
- One guy work on the GUI

☯ The yang

Pairs to draw boundaries
Dispatching the work force: enables parallel work
- One guy work on the highest-level
- One guy on the GUI
- Once one guy is finish a component, then switch on the Database

☯ Mixing the yin and yang in order to reach your efficiency with your formula.

Tips

At the beginning it will feel slow, be disciplined and practice.
Never, never force your teammates applying a practice.
Create your own formula, depending on your team and your context.
Invite all team members to practice each step.
Practice as a team, practice, and practice as a team.

Appendices

Usage of term "boundary"

The term "boundary" has been used instead of "interface" in order to mark greater separation and isolation between components.

Usage of term "task"

The term "task" represent what we call a technical task, which is a subtask of a Sprint Backlog item, which is a Product increment.

A technical task has no value outside the context in which it takes place. As soon as it no longer contributes to answer the need, it no longer has a reason to exist. It is only a means, and not an end in itself.

Don't hit the mistake to forget the end goal and getting bogged down in code.

Outcome over output

This post does not say that a product team should focus only on output, as the outcome is more. On this topic, I suggest reading about agile context.

Resources

Architecture

Read the book "Clean Architecture: A Craftsman's Guide to Software Structure and Design (Robert C. Martin Series) (English Edition)" wrote by Robert C. Martin
Read blog post: The Clean Architecture
Watch video: The Principles of Clean Architecture by Uncle Bob Martin

Craftsmanship

Read the Manifesto for Software Craftsmanship
Read the book "Clean Craftsmanship: Disciplines, Standards, and Ethics (Robert C. Martin Series) (English Edition)" wrote by Robert C. Martin
Watch the video: Uncle Bob Martin - The Craftsman's Oath at SC London Conference 2018
Watch the video: The Three Laws of TDD

Task switching

Read the blog post: Swarming: How To Instantly Boost Your Scrum Team’s Productivity
Reading the about the pattern: Swarming: One-Piece Continuous Flow
Watch the video: Henrik Kniberg : Multiple WIP vs One Piece Flow Example

Agile context

Diagrams

Acknowledgments

Without them this article would not exist, because they were part of the adventure that allowed the birth of this article:
For review:
- Aurel Abidos
For resource sharing on Swarming:
- Laurent Prost

I invite you to see my posts, not as my static opinions, but rather a stream of thoughts, caught in the middle of their update.