DEV Community: Rogério Chaves

Building a fully testable Lovable clone agent in Python with Scenario

Rogério Chaves — Tue, 15 Apr 2025 14:48:44 +0000

AI agents are super powerful and impressive. You might have heard for example of Lovable, the agent that builds website for you with a single prompt.

Of course it takes months and years of work to make it as good as they did, but the fundamentals are actually super easy to replicate: take a good LLM coding model, like Claude or Gemini 2.5, give it tools to read and write files, and ask it to carry over a task.

The devil of course is on the details. As user experience will show you, you probably should ask the LLM to think the whole stylesheet on the first interaction, otherwise the website will look "bland" and "basic", this is what lovable does for example. You will also want it to be self-aware of the tools and reduce refusal of using tools, which happens on Gemini sometimes, and you probably have an opinion on what libraries it should use, placeholder images, coding standards, etc.

This is where the challenge begins, and most of the time this learnings are accumulated over the prompt, but the prompt is not the initial specification itself, it's just what made it work, nor it's testable. As your agent workflow, tools and prompt complexity grows, the higher the chances of a small change breaking something that was already working. And retesting everything again by hand is a pain in the ass.

This is why the goal of this post is not only to build a lovable clone, but to make it fully testable!

Let's go, to build this, we are going to use Pydantic AI:

uv init lovable-clone
cd lovable-clone
uv add pydantic-ai

Then we are going to start creating our lovable clone agent, create a new file lovable_agent.py:

from pydantic_ai import Agent

class LovableAgent:
    def __init__(self):
        agent = Agent(
            "anthropic:claude-3-5-sonnet-latest",
            system_prompt=f"""
        You are a coding assistant specialized in building whole new websites from scratch.

        You will be given a basic React, TypeScript, Vite, Tailwind and Radix UI template and will work on top of that. Use the components from the "@/components/ui" folder.

        On the first user request for building the application, start by the src/index.css and tailwind.config.ts files to define the colors and general application style.
        Then, start building the website, you can call tools in sequence as much as you want.

        You will be given tools to read file, create file and update file to carry on your work.

        You CAN access local files by using the tools provided, but you CANNOT run the application, do NOT suggest the user that.

        <execution_flow>
        1. Call the read_file tool to understand the current files before updating them or creating new ones
        2. Start building the website, you can call tools in sequence as much as you want
        3. Ask the user for next steps
        </execution_flow>

        <files>
        After the user request, you will be given the second part of this system prompt, containing the file present on the project using the <files/> tag.
        </files>
        """,
            model_settings={
                "parallel_tool_calls": False,
                "temperature": 0.0,
                "max_tokens": 8192,
            },
        )

The prompt I'm putting here was already somewhat battle tested by myself, and can take you a long way. It also works better with Claude 3.5 then with 3.7 at the moment, but you can try changing it and see how it goes.

You will want to give it some tools too, just the 3 most basic ones: read_file, update_file and create_file. Those 3 are already enough to have a very powerful website builder agent at hand

import os
import shutil
import tempfile

template_path = os.path.join(os.path.dirname(__file__), "template")

class LovableAgent:
    def __init__(self):
        agent = ...

        @agent.tool_plain(
            docstring_format="google", require_parameter_descriptions=True
        )
        def read_file(path: str) -> str:
            """Reads the content of a file.

            Args:
                path (str): The path to the file to read. Required.

            Returns:
                str: The content of the file.
            """
            try:
                with open(os.path.join(self.template_path, path), "r") as f:
                    return f.read()
            except FileNotFoundError:
                return f"Error: File {path} not found (double check the file path)"

        @agent.tool_plain(
            docstring_format="google", require_parameter_descriptions=True
        )
        def update_file(path: str, content: str):
            """Updates the content of a file.

            Args:
                path (str): The path to the file to update. Required.
                content (str): The full file content to write. Required.
            """
            try:
                with open(os.path.join(self.template_path, path), "w") as f:
                    f.write(content)
            except FileNotFoundError:
                return f"Error: File {path} not found (double check the file path)"

            return "ok"

        @agent.tool_plain(
            docstring_format="google", require_parameter_descriptions=True
        )
        def create_file(path: str, content: str):
            """Creates a new file with the given content.

            Args:
                path (str): The path to the file to create. Required.
                content (str): The full file content to write. Required.
            """
            os.makedirs(
                os.path.dirname(os.path.join(self.template_path, path)), exist_ok=True
            )
            with open(os.path.join(self.template_path, path), "w") as f:
                f.write(content)

            return "ok"

        self.agent = agent
        self.history: list[ModelMessage] = []

Notice we also define a template_path at the top. The way it works is, just like Lovable, we don't build a whole website really from scratch, we start with a react, shadcn and vitejs template.

Let's then create this template so it can be used by the agent, follow the shadcn tutorial to start one:

https://ui.shadcn.com/docs/installation/vite

When asked, name the folder template, just next to our python agent file.

You can test your template individually by running:

cd template
npm install
npm run dev

If it's all working, it will work too for our agent, we are good to continue!

Now, add those methods to the agent:

from pydantic_ai.messages import ModelMessage, UserPromptPart
from pydantic_graph import End
from pydantic_ai.models.openai import OpenAIModel
from openai.types.chat import ChatCompletionMessageParam

class LovableAgent:
    # ...

    async def process_user_message(
        self, message: str, template_path: str, debug: bool = False
    ) -> tuple[str, list[ChatCompletionMessageParam]]:
        self.template_path = template_path

        tree = generate_directory_tree(template_path)

        user_prompt = f"""{message}

<files>
{tree}
</files>
"""

        async with self.agent.iter(
            user_prompt, message_history=self.history
        ) as agent_run:
            next_node = agent_run.next_node  # start with the first node
            nodes = [next_node]
            while not isinstance(next_node, End):
                next_node = await agent_run.next(next_node)
                nodes.append(next_node)

            if not agent_run.result:
                raise Exception("No result from agent")

            new_messages = agent_run.result.new_messages()
            for message_ in new_messages:
                for part in message_.parts:
                    if isinstance(part, UserPromptPart) and part.content == user_prompt:
                        part.content = message

            self.history += new_messages

            new_messages_openai_format = await self.convert_to_openai_format(
                new_messages
            )

            return agent_run.result.data, new_messages_openai_format

    async def convert_to_openai_format(
        self, messages: list[ModelMessage]
    ) -> list[ChatCompletionMessageParam]:
        openai_model = OpenAIModel("any")
        new_messages_openai_format: list[ChatCompletionMessageParam] = []
        for message in messages:
            async for openai_message in openai_model._map_message(message):
                new_messages_openai_format.append(openai_message)

        return new_messages_openai_format

    @classmethod
    def clone_template(cls):
        temp_path = os.path.join(tempfile.mkdtemp(), "lovable_clone")
        shutil.copytree(template_path, temp_path)
        shutil.rmtree(os.path.join(temp_path, "node_modules"), ignore_errors=True)
        return temp_path

Okay let's unpack what is going on here. The process_user_message is really where the execution happens. It's basically just running the user message, together with the current file list, through our pydantic-ai agent. The rest of the code is all about parsing processing those messages.

We then convert the messages to openai format. This is very userful as, being the standard formatting for basically all LLMs, we can use it back when running tests, so the testing agent understands.

Finally, missing piece, add this files tree helper to the file:

from typing import List, Set, Optional


def generate_directory_tree(path: str, ignore_dirs: Optional[Set[str]] = None) -> str:
    """
    Generate a visual representation of the directory structure.

    Args:
        path: The path to the directory to visualize
        ignore_dirs: Set of directory names to ignore (defaults to {"node_modules"})

    Returns:
        A formatted string representing the directory tree
    """
    if ignore_dirs is None:
        ignore_dirs = {"node_modules", ".git", ".venv"}

    # Normalize the path
    root_path = os.path.abspath(os.path.expanduser(path))
    root_name = os.path.basename(root_path) or root_path

    # Start the tree with the root
    tree_str = ".\n"

    # Get all directories and files
    items = _get_directory_contents(root_path, ignore_dirs)

    # Generate the tree representation
    tree_str += _format_tree(items, "", root_path)

    return tree_str


def _get_directory_contents(path: str, ignore_dirs: Set[str]) -> List[str]:
    """
    Get all files and directories in a path, sorted with directories first

    Args:
        path: The directory path to scan
        ignore_dirs: Set of directory names to ignore

    Returns:
        List of paths relative to the provided path
    """
    items = []

    try:
        # List all files and directories
        all_items = os.listdir(path)

        # Sort items (directories first)
        dirs = []
        files = []

        for item in sorted(all_items):
            full_path = os.path.join(path, item)
            if os.path.isdir(full_path):
                if item not in ignore_dirs:
                    dirs.append(item)
            else:
                files.append(item)

        items = dirs + files
    except (PermissionError, FileNotFoundError):
        pass

    return items


def _format_tree(items: List[str], prefix: str, path: str, ignore_dirs: Optional[Set[str]] = None) -> str:
    """
    Format the directory contents as a tree structure with proper indentation

    Args:
        items: List of filenames or directory names
        prefix: Current line prefix for indentation
        path: Current directory path
        ignore_dirs: Set of directory names to ignore

    Returns:
        Formatted tree string for the current level
    """
    if ignore_dirs is None:
        ignore_dirs = {"node_modules"}

    tree_str = ""
    count = len(items)

    for i, item in enumerate(items):
        # Determine if this is the last item at this level
        is_last = i == count - 1

        # Choose the appropriate connector symbols
        conn = "└── " if is_last else "├── "
        next_prefix = "    " if is_last else "│   "

        # Add the current item to the tree
        tree_str += f"{prefix}{conn}{item}\n"

        # Recursively process subdirectories
        item_path = os.path.join(path, item)
        if os.path.isdir(item_path) and item not in ignore_dirs:
            sub_items = _get_directory_contents(item_path, ignore_dirs)
            if sub_items:
                tree_str += _format_tree(sub_items, prefix + next_prefix, item_path, ignore_dirs)

    return tree_str

That's it! The Lovable Clone agent should fully run now, if you create an instance for it and send it a request.

However, what we are going to do is not test it directly, but test it via our tests, using Scenario:

uv add pytest langwatch-scenario

Writing a test for it is the easiest thing, create a new file tests/test_lovable_agent.py, then paste this:

import pytest

from lovable_agent import LovableAgent
from scenario import Scenario, TestingAgent

Scenario.configure(testing_agent=TestingAgent(model="anthropic/claude-3-5-sonnet-latest"))


@pytest.mark.agent_test
@pytest.mark.asyncio
async def test_lovable_clone():
    template_path = LovableAgent.clone_template()
    print(f"\n-> Lovable clone template path: {template_path}\n")

    async def lovable_agent(message: str, context):
        lovable_agent = LovableAgent()

        _, messages = await lovable_agent.process_user_message(message, template_path)

        return {"messages": messages}

    scenario = Scenario(
        "user wants to create a new landing page for their dog walking startup",
        agent=lovable_agent,
        strategy="send the first message to generate the landing page, then a single follow up request to extend it, then give your final verdict",
        success_criteria=[
            "agent reads the files before go and making changes",
            "agent modified the index.css file",
            "agent modified the Index.tsx file",
            "agent created a comprehensive landing page",
            "agent extended the landing page with a new section",
        ],
        failure_criteria=[
            "agent says it can't read the file",
            "agent produces incomplete code or is too lazy to finish",
        ],
        max_turns=5,
    )

    result = await scenario.run()

    print(f"\n-> Done, check the results at: {template_path}\n")

    assert result.success

That's it, now run with:

uv run pytest tests/test_lovable_agent.py

You will see the agent talking to each other, asking for a dog website, asking for updates on it, it's super fun to watch. The agent will be keep chatting until it reaches all the goals in the success criteria, or fails in any of the failure criteria. But with Claude 3.5 and this prompt, it should work every time:

And then, you can go to the /tmp folder that will be displayed, and see the final results:

cd /tmp/path-given
npm install
npm run dev

I got something like this: 😄

Now you are free to play with the prompt, make changes! See if it still works. Did it do something you didn't expect? Maybe add that to the failure criteria.

The cool thing is that now you can add a next scenario, for example what if you want it to use Unsplash API for the placeholder images?

This is exactly the example I go through on this video, how to extend our lovable clone with one more test scenario test and support more functionality, check it out.

Check out the Scenario GitHub Repo for more examples, and drop a star ⭐

See you next time!

Scaling from a Billion to a Million to One

Rogério Chaves — Thu, 02 Jan 2025 10:18:42 +0000

When I started LangWatch, I had a crystal clear development vision for it in mind, I had been through it all, from starting my first business and entangling myself in code so messy I couldn’t move any longer (and therefore losing money and sleep), to working on a consultancy with perfect TDD, pairing and couldn’t-be-more-refactored codebase (sleeping, oh, so well!), to incredibly messy code again this time done by more senior devs (sleeping ok but wait, this actually moves about the same speed and makes way more money??), to then having to migrate this same codebase to an actual well-written one to cut the fires (check it out more Designed v Evolutionary Code).

All this learning allowed myself to see a path to that deliciously sweet spot: maximizing speed and flexibility, without ever shooting myself in the foot; maximizing the short term, with no cost to the medium and long term; global scale and zero overengineering at the same time; tryna find that blackswan for the startup phase as soon as possible, without having to stop and migrate any architecture for scaling later.

So armed with a lot of dev experience, probability theory, and all there is, off I went, started building as a startup: Next.JS and Vercel, yes, get it up in 5 minutes, what database? Oh, Planetscale? Sure, MySQL then. Boring tech yes, that’s how you win. What about AI, the cool new tech part which is what the startup is all about? Python of course, don’t need to think for half a blink on it, just swim in favor. Tests? Only where it makes me code faster (yes, like data transformations, you actually code faster), and monitoring for the critical, none for the rest.

The goal is, everything that is easy to change later (a lot of the frontend for example), I don’t care, needs to stay flexible, this is where we keep buying our different lottery tickets, everything that will be hard to migrate later, needs to be done right from the get go. Rock solid foundation, flexible construction.

So this boring tech obviously scales to a billion, it’s proven. Vercel won’t, but that doesn’t matter, I can just move to AWS when time is right, and I don’t need to change anything about my app, it will still be TypeScript, MySQL, Python. But there is one piece of the puzzle that is no so obvious, we are mainly a monitoring tool, an LLM monitoring one, so where do we store the traces?

I went with ElasticSearch there. I already knew elastic, got annoyed with OpenSearch after-fork quirks (so definitively going for the no-longer-open-source one), and I knew that even though “oh, startup, just use Postgres for everything, you are overengineering”, those would be one of those big shoot-in-the-foot as soon as we needed a proper timeseries db, fast analytics, millions inserts and so on. One of the reasons it attracted me too was the vector search, so hot for AI, also on Postgres with an extension (and the reason I didn’t consider ClickHouse, didn’t know it could also do that at the time). Plus I needed a queue, went with Redis, no biggie, it will hold a lot, Kafka is too much, queues are easy to replace later.

Everything worked, and I built so fast, got the MVP working in no time, we went talking to leads, design partners. First design partner was very interest in analytics, made myself pretty happy with my elastic choice, we soon had snappy, colorful graphs, all kinds of filters, cut and dice the data all you want, all was good.

Then we went to talk to more companies, and they were like — but you are SOC2/ISO27001 compliant right? Where are your servers? It’s all GDPR in Europe right? Where is this OpenAI endpoint going to? We can only use Azure OpenAI, and only if that’s in Sweden

Okay shoot, is Vercel this? Is PlanetScale that? Where are my nodes on ElasticCloud? Azure give me startup credits please!

Okay, no worries, I have been preparing for this moment, I knew I’d have to move to something like AWS/Azure/GCP sooner or later, but damn second month in, that’s pretty soon. I actually started with GCP, because they gave way more credits, but turns out, the AI helping me writing infra code (and it was no Claude 3.5 back then) was actually way better when writing terraform for AWS and I had a way easier time with libraries available and so on. Again, boring tech wins.

So okay, all good, all in AWS now, all in Frankfurt Germany now, where all cloud providers are, Azure OpenAI in Sweden, Vanta is checking our compliance, yes, all encrypted, yes rest and transit I made sure, can we do business now?

“Wait, do you want me, to send my data, to your cloud? There ain’t no way”“But we are all in europe, all secure!”“There ain’t no way”“But ISO, SOC2, in transit too!”“Nope”So this was something that I really didn’t see comming. Companies have become more and more concerned about privacy on this past decade, sure, but AI accelerated that a lot, the fear of OpenAI training your data was at all time high at end of 2023/early 2024, and the high cloud bills and back-to-onprem movements made having onprem a must-have for any solutions dealing with AI data.

I’ve started with the premise of not shooting myself in the foot for scaling for billions, but when installing it for someone else, on someone else’s AWS/Azure/GCP account (that’s what onprem actually means mostly), they don’t need to scale to billions, just millions is enough.

This means that while I made the right architectural decisions — and I still stand by them — I didn’t exactly made the right infrastructure implementation decisions. For example, we were using AWS lambdas for the LLM evaluators, because then we can scale them forever and not care, but a lot of infra setup, config env vars and so were all hardcoded and scattered, cuz who cares right, it’s our own infra, we can clean it up later. But now I need to make it portable, and things like lambdas don’t make it portable, plus they don’t need that for millions, a server will hold it.

So we had to scale back, we had to make the setup easy, easy to move around, and easy to sell, procurement-wise. Again luckily, boring tech, mysql, redis, all available in all clouds, a great win, and then there was ElasticSearch… Well, wasn’t a problem, our main enterprise lead uses AWS, so that’s nice, bit the bullet and added compatibility with OpenSearch.

Wow, so nice, now everything, EVERYTHING lives on your AWS, all self-contained and self-managed, nothing gets in, nothing leaves, except for that OpenAI Azure endpoint which you configure yourself through the UI and I don’t touch it. Nice, that’s all packed up.

At this point I also migrated it from mysql to postgres because fuck it, planetscale is long gone, was already changing things anyway, and that one was actually super easy.

Coincidentally, around this time, some fuzz popped up again on social media about great progress on Postgres vector search support, plus I had my eyes on timescaledb, a time-series extension for Postgres, and I was like “fuuuuck, should I have listened? should I have just gone with Postgres-is-all-you-need? did I overengineer? me, so careful?”.

It would be perfect, managed Postgres is available on all clouds, so easy, so nice. I got away on AWS with OpenSearch, but GCP and Azure didn’t have a managed elastic, and we already had leads from those clouds on the door asking about onprem - but if they need to go to the marketplace to purchase additional components that could mean another added procurement hurdle.

But actually, the Postgres dream is not as easy as that. Those extensions are not natively supported, this means if you want them, you need to run your own Postgres!

Ohh no no no, this would be way worse! Having to manage backups? Uptime? No sir thank you we ain’t an infra company! Sure you could use TimescaleDB from the marketplace as well, but at this point OpenSearch is winning. Azure and GCP will require it either way for both options, but hey, I think that will have to be it. I was happy with being right on my first call.

Moving on, as product and business developed, we realized more and more of the inevitable: we are a dev tool after all, so we need to get support from the devs. It will never work long-term if we go just top-down, we need to go bottom up too, we need to give back to the community, that’s the only way devs will ever consider us, adopt, keep using.

Now, we were already an open-source, but we never really advertised it as much. As we focused more on it, we realized: we are at the wrong scale, again.

A dev, playing with the open-source project, don’t need to scale to millions, they don’t need full blown elasticsearch. They need to scale to one. They need to run it on their machine, and their machine might not have 16GB of RAM laying around for us just because. Docker Compose setup? Still a bit annoying. Do I really need a Redis queue? I’m sending just 2 messages.

We have to reduce the barrier as much as possible, ideally, you should be one pip install away from greatness.

Let’s see then, we can drop Redis for a local setup, that’s more than fine, we just call the worker async directly. Postgres? We can use SQLite instead, prisma should make it easy to be compatible with both. The Next.JS app I can pre-build and shove it inside the Python package, people will still need Node installed to run the server but I think that’s a fair assumption. But what about OpenSearch? That’s the irreductible bit.

No matter how you twist and turn, how many initialization configs you disable, OpenSearch/ES is a beast, heavy, takes long to start and eats up a lot of RAM, and I really really wouldn’t try to ship this without docker. So how can we make the pip install dream work? Luckily, we found a nice little thing called quickwit. Quickwit is built for scale, but its rust-compiled binary can be downloaded for just ~80mb, it starts very fast AND it has an elasticsearch-somewhat-compatible API! Just what we need. Now it doesn’t support scripts or some complicated aggregations, but that’s fine, you don’t need it all on dev mode, and this is where we’re at right now.

We certainly came a long way, even though I think my early confidence is not without merits, portability was not something I had anywhere high on my list, but it was great to learn how heavily it counts. Thank you for reading, and now I’ll continue on my journey, of making LangWatch scale, to just one.

The most useful command to clean your server hard drive

Rogério Chaves — Sun, 09 Jan 2022 13:19:25 +0000

du -h / | sort -h

The Pragmatic Pragmatic Programmer

Rogério Chaves — Thu, 06 Jan 2022 20:57:07 +0000

This story was originally published in my blog and reached hacker news front page, click here to check the discussion

pragmatic

/præɡˈmæt.ɪk/\
adjective

solving problems in a sensible way that suits the conditions that really exist now, rather than obeying fixed theories, ideas, or rules - Cambridge Dictionary
based on practical judgments rather than principles - American Dictionary
relating to matters of fact or practical affairs often to the exclusion of intellectual or artistic matters : practical as opposed to idealistic - Merriam-Webster

Recently I started reading the book The Pragmatic Programmer, I had never read it before yet it influenced me so much, how? Simply by the title. The Pragmatic Programmer, such a cool concept. I've heard about this book left and right, it was a mandatory read once I got serious about software development, I heard people talking about "you have to be pragmatic here", or "those devs are pragmatic", this changed my mind when approaching the development of many software systems, yet for some reason I never got to actually read it, it has been sitting on my "To Read" list for a long long time until now.

So now I read it and it was... disappointing. Like very disappointing. Like "don't meet your heroes" kind of thing. The title "The Pragmatic Programmer" is so brilliant, yet they ruined it with the content.

My understanding of pragmatic is the same as the definitions given above, and it were those that I used to guide my thinking when taking pragmatic decisions. Not good practices, not dogmas, but foot on the ground, facing the harsh and cruel reality. The Pragmatic Programmer book is not about that, it's a dogmatic book.

The Pragmatic Programmer: The Good Parts

The book actually have some very good and pragmatic points, I will start with those because I think they should have been the central idea of the book:

CRITICAL THINKING

The last important point is to think critically about what you read and hear. You need to ensure that the knowledge in your portfolio is accurate and unswayed by either vendor or media hype. Beware of the zealots who insist that their dogma provides the only answer—it may or may not be applicable to you and your project.

Never underestimate the power of commercialism. Just because a web search engine lists a hit first doesn’t mean that it’s the best match; the content provider can pay to get top billing. Just because a bookstore features a book prominently doesn’t mean it’s a good book, or even popular; they may have been paid to place it there.

Critical Thinking, that's it, for me that's in the very core definition of what does it mean to be Pragmatic, don't be fooled by dogmas and hype, keep your foot on the ground.

Still for some reason they say it's the last important part (I don't know why it isn't the first topic on the book), and later they throw lots and lots of dogmas at you, without explaining the reasoning behind them, contradicting this very same paragraph.

They go further on Critical Thinking with some tips:

Ask the “Five Whys”

A favorite consulting trick: ask “why?” at least five times. Ask a question, and get an answer. Dig deeper by asking “why?” Repeat as if you were a petulant four-year old (but a polite one). You might be able to get closer to a root cause this way.

[...]

What’s the context?

Everything occurs in its own context, which is why “one size fits all” solutions often don’t. Consider an article or book touting a “best practice.” Good questions to consider are “best for who?” What are the prerequisites, what are the consequences, short and long term?

Those are perfect, that's much of what I understand by being pragmatic.

Then, to close off the good parts of the book, on Good-Enough Software:

The marketing people will have promises to keep, the eventual end users may have made plans based on a delivery schedule, and your company will certainly have cashflow constraints. It would be unprofessional to ignore these users’ requirements simply to add new features to the program, or to polish up the code just one more time. We’re not advocating panic: it is equally unprofessional to promise impossible time scales and to cut basic engineering corners to meet a deadline.

That's it, that's a Pragmatic Programmer, one that has to work within corporate constraints, one which has it's foot on the ground, that produces Good-Enough Software, because they know it's not possible to build the perfect system given real world contraints, nor to deliver on impossible deadlines.

The Dogmatic Programmer

At the same time the book argues for Good-Enough Software, it argues for the broken window theory:

Don’t leave “broken windows’’ (bad designs, wrong decisions, or poor code) unrepaired. Fix each one as soon as it is discovered.

This is already somewhat contradictory to the Good-Enough Software point made on the book just two topics later, but the worse is that it's actually based on a theory that was already debunked

A lot of times the story of crime reduction in NY with the broken-window policy is used to back-up this, but a quick read shows you that this is a mere correlation, not causation, a change of police strategy to focus on the bigger crimes rather than on petty ones might have actually been much more effective, together with abortion legalization and many other candidates, nobody knows for sure.

The broken-window theory for code feels like a nice idea, but feeling is not enough, Dogmatic Programmers are gullible and like theories, but a True Pragmatic Programer is skeptical, and values scientific rigour (when possible).

Dogmatic Programmer: gullible, does what feels right, following idealistic theories

Pragmatic Programmer: skeptical, values scientific rigour (when possible)

Now, the (when possible) part is very important, and very pragmatic too, because to be honest, in software development that's rarelly possible. The more I learned about statistics and biases in those recent years, the more I realized that having a proper experiment to discover causality is an extremely hard thing, and in software development, basically impossible.

We cannot build thousands of the same systems, with thousands of developers and add just a small variation (say, broken-window policy) to do a proper RCT and see it's effect. We don't even know if static types are better than dynamic ones, we have many studies, but all confounded or biased in a way or another.

Most of the times we have no choice, being pragmatic means you have to use what is practically available: subjective experiences, biased experiments, or even your favorite dogma when there is nothing left. Practically, you need to take a decision (or even decide to not take one), but keeping your skepticism if you want to be pragmatic, fix your foot on the ground.

In that vein, I will tell you a subjective experience I had that touches the broken-window theory, and I wrote about it on the blogpost Designed v Evolutionary Code. It's not that I'm against broken-window theory or that I'm in favor of it, it's just that it depends. I take the book's own advice of What’s the context? to give you an example of just two cases where it probably works and where it probably doesn't, following Kent Back's 3X Theory:

If you are in exploration mode, most of the code you produce has no value and you will throw away, you need to iterate faster, here broken windows are totally fine, you focus on the current biggest problem only, law of diminishing returns kicks in and fixing windows doesn't pay-off at all, intead you move quickly to the next biggest problem
If you are in extraction mode, on a huge system, even a single flaw can be cathastrophic and cost millions of dollars or lifes (you are under power law), here you don't want to keep a single window broken, you want to try your best to be sure all tiny pieces are working as they should and hope for the best

A True Pragmatic Programmer is skeptical of stories and dogmas, but the book is not skeptical at all, it tries to convice you using anecdotes, comming from the authors' experience, without any depthness, cues for the possibility of being wrong, and sometimes not even a short explanation, for example:

Duplication with Data Sources

Many data sources allow you to introspect on their data schema. This can be used to remove much of the duplication between them and your code. Rather than manually creating the code to contain this stored data, you can generate the containers directly from the schema. Many persistence frameworks will do this heavy lifting for you.

There’s another option, and one we often prefer. Rather than writing code that represents external data in a fixed structure (an instance of a struct or class, for example), just stick it into a key/value data structure (your language might call it a map, hash, dictionary, or even object).

So should I just trust your preference? It doesn't seem very pragmatic. And why do you even prefer that? There is not an explanation of the rationale whatsoever, where is the Critical Thinking? Where is the Five Whys?

Most of the times the book assume all conclusions are self evident, there are many more examples like that:

you can discipline yourself to write software that’s good enough—good enough for your users, for future maintainers, for your own peace of mind. You’ll find that you are more productive and your users are happier. And you may well find that your programs are actually better for their shorter incubation.

Why?? I mean I'm obviously all in favor of shipping ealier (generally), but why does it make things better? I feel the authors could have done a way better job explaining how production is the real test to our systems, how real life is so complex that is impossible to predict all the variables, like Nassim Taleb explains so well in Anti-fragile, so we have to be pragmatic, accept there is a lot of variables we don't know, and that it's cheaper to allow real-life to give us this feedback then upfront predicting them all, this is why Agile works (but not always! Context matters)

Your ability to learn new things is your most important strategic asset.

Why? And "strategic"? I thought this book was about being pragmatic (foot on the ground, present) not "strategic" (heads in the cloud, future)

Treat English (or whatever your native tongue may be) as just another programming language. Write natural language as you would write code: honor the DRY principle, ETC, automation, and so on.

Why? Being DRY is actually terrible advice for natural language communication, people don't pay attention, people forget, you actually need to be repetitive sometimes.

If you can’t find the answer yourself, find out who can. Don’t let it rest. Talking to other people will help build your personal network, and you may surprise yourself by finding solutions to other, unrelated problems along the way.

Tips tips tips, just tips, no pragmatism here.

Chapter 2 is titled "A Pragmatic Approach", and it starts with that, I kid you not:

A Pragmatic Approach

There are certain tips and tricks that apply at all levels of software development, processes that are virtually universal, and ideas that are almost axiomatic

So their Pragmatic approach is to bring us a chapter full of axioms, not to challenge them, but to tell us they are universal 🤯. Say what!? This is absolutely NOT pragmatic. I expect to find axioms on Clean Code, that's why I bought that book, not here, here I expected to find counter-axioms, cases of when you have to actually be pragmatic, take your head off the clouds and break those axioms when it makes practical sense. This chapter should really be called A Dogmatic Approach.

They continue:

The world is full of gurus and pundits, all eager to pass on their hard-earned wisdom when it comes to How to Design Software.

And finally confess:

And we, your gentle authors, are guilty of this too. But we’d like to make amends by explaining something that only became apparent to us fairly recently. First, the general statement:

Their "explanation", actually uses a dogma as an excuse (Good Design Is Easier to Change Than Bad Design), so it's circular. "I'm sorry we are not pragmatic and presenting you dogmas, but the reason is because of this dogma"

You can repeat yourself

Topic 9 is called "DRY — The Evils of Duplication", this is the part of the book I have the biggest beef with. The authors treat repetion as pure evil, and don't make any arguments on when it could be good, it's not pragmatic at all, pure dogma.

In fact, I've used in the past sentences like "they decided to take the more pragmatic approach and copy-paste the code for this situation", so it bothers me that the book that takes pride in it's title wouldn't even consider this as a possible sentence.

Just to be clear, I did many terrible code duplications in the past, and it helped me a lot learning the DRY concept at the start of my programming carreer and removing many duplications, but unlike the authors, I know that DRY does not hold true everywhere, and many many times it causes unintended harm, as you can read in Duplication is far cheaper than the wrong abstraction by Sandi Metz. It's not all black-and-white, a pragmatic programmer should know that.

Dogmatic Programmer: Duplication is evil

Pragmatic Programmer: It depends

They start with a code refactor example to try to make their point. This is the original code:

def print_balance(account)
    printf "Debits: %10.2f\n", account.debits
    printf "Credits: %10.2f\n", account.credits
    if account.fees < 0
        printf "Fees: %10.2f-\n", -account.fees
    else
        printf "Fees: %10.2f\n", account.fees
    end
    printf " ———-\n"
    if account.balance < 0
        printf "Balance: %10.2f-\n", -account.balance
    else
        printf "Balance: %10.2f\n", account.balance
    end
end

So they do this refactoring:

def format_amount(value)
    result = sprintf("%10.2f", value.abs)
    if value < 0
        result + "-"
    else
        result + " "
    end
end

def print_line(label, value)
    printf "%-9s%s\n", label, value
end

def report_line(label, amount)
    print_line(label + ":", format_amount(amount))
end

def print_balance(account)
    report_line("Debits", account.debits)
    report_line("Credits", account.credits)
    report_line("Fees", account.fees)
    print_line("", "———-")
    report_line("Balance", account.balance)
end

And they claim it a victory, apparently they removed most of the duplication.

What!? This is a terrible refactor, one function became four; it's way worse to read, now instead of reading just a bunch of plain printfs, easy to follow because it's a core function of the language, the reader needs to jump around functions and understand what the author meant; now instead of one function causing bugs you have 4 functions that can interact in weird ways causing weirder bugs; now if you want to change the format of just one like you need to add an if condition, a new method, or something, it actually violates their Easier to Change rule.

It does not work here because this code is so simple, this is the wrong example, on a full-fledge report some of those refactorings could actually make sense, you would probably have the logic and presentation separation already anyway, but giving the advice on this small piece of code actually makes things much worse, showing that they are not taking context into account, but just following DRY as a dogma.

Now there are many statements they make I'd like to comment on:

The alternative is to have the same thing expressed in two or more places. If you change one, you have to remember to change the others, or, like the alien computers, your program will be brought to its knees by a contradiction. It isn’t a question of whether you’ll remember: it’s a question of when you’ll forget.

Conversely, if you have everything expressed in a single, centralized place and you need to change it you are forced to remember all the places that depend on that. This can take a long time, time which may not even pay off for the inconsistency. It could be that the place you "forgot" to change actually was not that important at all, like a page nobody visits, a feature no one uses. If it is important, you will notice it. Again it depends on context, I touch on this on the Designed v Evolutionary Code post too.

Additionaly, no duplication at all can actually be pretty risky. Remember left-pad? Because everyone decided to not duplicate something that can be implemented in 1 line of code pretty much all javascript deployments in the planet were broken for a day. When you have central critical pieces without any duplication, you increase the chances of bringing the whole system to it's knees, like it happened many times recently with big tech.

DRY is about the duplication of knowledge, of intent. It’s about expressing the same thing in two different places, possibly in two totally different ways.

When I say duplication is good for resilience, some may think about server redundancy, but no, I really mean intent duplication, which the authors rejects. Why is "two totally different ways" necessarily a bad thing? Devs love redundancy of servers but despite duplication of code, problem is, having thousands of servers in hundreds of availability zones still won't prevent you from deploying a catastrophic commit.

If you think about it, different codebases doing the same thing is like redundancy for the overall system, for ideas, for bugs. There is a reason why evolution brought us so many different animals, which survive through different ways, there is a reason why we have anti-monopoly laws, causing different companies to do the same thing but in different ways, this way you get the chance that one of them is actually better than the other.

Dogmatic Programmer: let’s make everything abstract, what if we need to switch databases?

Pragmatic Programmer: we will never switch databases

On topic 11 - Reversibility, the authors go on about how we should decouple third-party services to keep our system flexible, and of course they give the classic example of the database switch:

“But you said we’d use database XYZ! We are 85% done coding the project, we can’t change now!”

In practice there are not a lot of reasons for switching one relational database with another, and even if you do, it will probably not be crazy hard, SQL is very standardized (much easier to learn than the thousands of ORMs APIs we have), on a small codebase find and replace can do most of the trick. Even if a big codebase decides to switch, it will be incremental with a migration plan anyway, it will never be “just change a config on the ORM”. Now if the change is for a completely different way to store the data to some domain specific NoSQL, or replace those MySQL queues with Kafka, then probably your abstractions won’t hold anyway, you will need to change your architecture to really take advantage of it.

An abuse of “reversibility” causes for example Generic Cloud Usage, from ThoughtWorks radar:

We see organizations limiting their use of the cloud to only those features common across all cloud providers—thereby missing out on the providers' unique benefits. We see organizations making large investments in home-grown abstraction layers that are too complex to build and too costly to maintain to stay cloud agnostic.

[...] which reminds us of the lowest common denominator scenario we saw 10 years ago when companies avoided many advanced features in relational databases in an effort to remain vendor neutral

Give a random number

The book then goes to Estimating, on topic 15, which is perhaps the most debated point in software development. As usual, they start with a bad example:

The Library of Congress in Washington, DC, currently has about 75 terabytes of digital information online. Quick! How long will it take to send all that information over a 1Gbps network? How much storage will you need for a million names and addresses? How long does it take to compress 100Mb of text? How many months will it take to deliver your project?

The last question is completely different from all the others, they are conflating estimations that have a super simple and linear models, where you can easily find all the variables, with a complex non-linear estimation where no good model exists. Deceiving the reader into thinking that estimating a software project is as easy and reliable as calculating bandwidth is an awful thing to do.

Then they go on some tips on how to improve your estimations, telling about building a model and breaking it in smaller pieces, they give a lot of focus on getting better at estimating, but the actual valuable thing is some communication tips in the middle, like saying weeks instead of days to convey less precision, and giving multiple range estimations instead of a point estimate.

Dogmatic 1: believes on estimations, and that one day they they will get very good at it (this day never comes)

Dogmatic 2: believes that estimations are never possible to get right anyway, so they are useless

Pragmatic: manages expectations, not accuracy

You see, developers are actually not that bad on estimating, getting around 60% of the estimations right[1][2], people hypothesize that developers are good at estimating the median, not the mean. So the problem really is on the blowup factor: eventually something will get very wrong, and not even the most experienced estimator will be able to predict it, when that happens (and it does), expectations management skills is what really counts.

Considering that, the True Pragmatic Programmer is not in favor or against estimations: it’s just a tool, which they know is flawed (sometimes exponentially flawed), still sometimes the most available one to align teams more or less on what to do, or as a way to get others thinking more deeply about a feature, so they use it when useful, and avoid it (or just give a random number) when useless.

When someone says “the deadline is October” for a pragmatic programmer, they ask “why?” (those five whys mentioned by the book). More often than not I've found out that those are just self-inflicted deadlines: no client knows about this feature you want to launch, other teams are unaware, it wouldn’t really affect much anyone, it’s just politics or a random date someone came up with.

Other times, the deadline actually makes sense, maybe some other team depends on that for a pressing issue, or maybe there is a regulation change, and even though you might use some fancy tricks of iterative development to avoid that, sometimes is just easier to agree on a date and that’s it. Pragmatic.

Well I think that’s enough about the book, this post is too long already, but I hope I manage to convey the idea of what a pragmatic actually means, the definition that is closer to the dictionary one.

Thanks for reading! Follow me on Twitter, and drop a message there if you disagree or want to discuss about any of the points.

Addendum 1: The version of the book I have and this blogpost is based on is the 20th Anniversary Edition, not the classic one.

Addendum 2: The article came out very harsh, so I need to clarify that I do in fact think The Pragmatic Programmer is a very valuable book for beginner developers, but not advanced as I hoped so. This made me want to write this blogpost, to express what I actually wanted to read, the book I wish existed.

Can you guarantee that a function has no bugs?

Rogério Chaves — Mon, 16 Aug 2021 18:28:57 +0000

Originally published on my blog

Software is famously known to have flaws, good practices and automated tests have helped us reduce issues, but most of the industry pretty much gave up on the idea of having software completely free of bugs, instead we moved our strategy to delivering early and iterating faster, letting the real world be our QA, and the real world is indeed the best QA there is. But is that really the only way? How come we always have gaps when testing our sofware? I mean it's computers, why can't we like simulate all the bits and be sure to check everything? What if we do unit tests? Integration tests? E2E tests? Contract tests? Have 100% coverage? Do some Mutation tests? If we apply all the tricks we can, can we feel safe with code which written by other humans? What if the code was not written by a human, but a machine, like Copilot?

The idea for this blogpost came from an insight by Nicholas Tietz, he suggests that maybe with AIs like Copilot evolving, it could be the case that we humans focus more on the specification and verification of what we want to be done, and the machine could automate the coding part.

If you think about it, it's kinda like doing TDD, in which on the first step you need to "write a test that passes iff the feature's specifications are met". Then, on the seconds step you see it fail, and on the third step you "write the simplest code that passes the new test". This is the step that could be automated by a machine.

Most importantly, and often overlooked by TDD beginners, it the keyword simplest. Write just enough to make the test green, nothing more, because this way you will be forced to write more tests for the functionality that you know is still missing, and then when you do, you are sure that functionality is tested as well. This is also a great way to think about what a machine will do, more often then not, given an objective, machines WILL find the simplest solution, often surprising us humans.

This means our tests need to ensure complete behaviour in order to prevent AI (or other devs) from taking shortcuts. Can we do that? Write our tests so well that it will force our code to behave properly? Let's try, let's start our project:

npm init

Unit Tests

Create a test file transaction.test.js with a sample test, just to check our tests are working:

test("it works", () => {
  expect(1 + 1).toBe(2);
});

Run it, all is working:

npx jest

 PASS  ./transaction.test.js

Alright, so now what do we actually want to build? My idea is to build a simple function to transfer money from Alice to Bob. To do that I wrote a test that creates a sqlite db, a balance table, put 10 pesos on Alice balance and 5 pesos on Bob balance. Then I call the function transferMoney, which doesn't exist yet but I intend to write it.

This function should transfer 10 pesos from Alice to Bob, we verify that by checking that the balances are correct, here is the the test:

const sqlite3 = require("better-sqlite3");
const transaction = require("./transaction");

describe("transferMoney", () => {
  let db;

  beforeEach(() => {
    db = sqlite3(':memory:');
    db.exec("CREATE TABLE balance (account TEXT, amount INT)");
    db.exec("INSERT INTO balance(account, amount) VALUES ('Alice', 10)");
    db.exec("INSERT INTO balance(account, amount) VALUES ('Bob', 5)");
  });

  it("transfers money from Alice to Bob", () => {
    transaction.transferMoney(db, { from: 'Alice', to: 'Bob', amount: 10 });
    const alice = db.prepare("SELECT amount FROM balance WHERE account = 'Alice'").get();
    expect(alice.amount).toBe(0);

    const bob = db.prepare("SELECT amount FROM balance WHERE account = 'Bob'").get();
    expect(bob.amount).toBe(15);
  });
});

We run the test again, with npx jest, and find several errors: first of all we don't have sqlite, please install it with npm install better-sqlite3; then it will say it cannot find module ./transaction, create transaction.js file and export a transferMoney function from it; with setup out of the way, now we need to implement the most simple thing to make the test pass:

function transferMoney(db) {
    db.prepare("UPDATE balance SET amount = 0 WHERE account = 'Alice'").run();
    db.prepare("UPDATE balance SET amount = 15 WHERE account = 'Bob'").run();
}

module.exports = { transferMoney };

 npx jest
 PASS  ./transaction.test.js
  ✓ transfers money from Alice to Bob (13 ms)

Our tests passes, we did TDD, and we have 100% coverage (you can check by using the --coverage flag), so our code must be perfect and free of bugs, right? Right!? What? Hard-coded values you say?

Well, obviously, our code sucks, we didn't even had to use the parameters as our current test suite does not force us to. How can we force parameter usage? One solution is to write more tests with a slightly different value, to force the parameters to be used -- but that feels a bit stupid.

This why TDD has a last step: Refactoring, where you can reorganize the code, preserving the functionality without adding anything new, while the tests still passes. So let's rewrite the code like something closer to what we expected (you might not like the result either, but bare with me):

function transferMoney(db, { from, to, amount }) {
    db.prepare("UPDATE balance SET amount = amount - ? WHERE account = ?").run(amount, from);
    db.prepare("UPDATE balance SET amount = amount + ? WHERE account = ?").run(amount, to);
}

Run the tests again, they pass.

You can argue that a machine would not refactor the code like we did, but to be honest, I don't think it would even need the refactor. Even though machines take shortcuts, on the current state of the AIs we have, it would most likely make use of those parameters from the start, as human developers usually do too. The reason for that it that AI is trained on gazillions of human code (for now), so it's more likely to replicate a longer, more common solution, than to come up with a simpler but creative one. So let's assume this is the case and move forward.

Cool, we have a feature to transfer money working, it seems that TDD helped us implementing the happy path, we can ship to production and see if any problems happen. Do you already know which ones? Take a look on the previous code and find at least two things you think might go wrong with it.

Let's start with an easy one, the famous billion-dollar mistake:

it("does not transfer any money when amount is undefined", () => {
  transaction.transferMoney(db, { from: 'Alice', to: 'Bob', amount: undefined });

  const alice = db.prepare("SELECT amount FROM balance WHERE account = 'Alice'").get();
  expect(alice.amount).toBe(10);

  const bob = db.prepare("SELECT amount FROM balance WHERE account = 'Bob'").get();
  expect(bob.amount).toBe(15);
});

Run the tests, we get an error:

npx jest
 FAIL  ./transaction.test.js
  transferMoney
    ✓ transfers money from Alice to Bob (56 ms)
    ✕ sending undefined does not transfer any amount (9 ms)

  ● transferMoney › sending undefined does not transfer any amount

    expect(received).toBe(expected) // Object.is equality

    Expected: 10
    Received: null

How can we prevent that? Plus all other nulls, undefineds or invalid types that can reach our function?

Type Checking

We could just add an if condition and return or throw if the amount is null, or undefined, an object instead of a number, etc, and also check on every other argument, not forgetting to write a unit test for each case or... we can just add types. Types constrain the solution space, it could guide a computer to generate code only within the types we added. Once you go very strict about it, some even say that if it compiles, it works.

On JavaScript world it's very easy to move from dynamic to static types with TypeScript, so let's rename our files to transaction.ts and transaction.test.ts, and type all the variables. This is the typed test file:

import * as sqlite3 from "better-sqlite3";
import * as transaction from "./transaction";

describe("transferMoney", () => {
  let db : sqlite3.Database;

  beforeEach(() => {
    db = sqlite3(':memory:');
    db.exec("CREATE TABLE balance (account TEXT, amount INT)");
    db.exec("INSERT INTO balance(account, amount) VALUES ('Alice', 10)");
    db.exec("INSERT INTO balance(account, amount) VALUES ('Bob', 5)");
  });

  it("transfers money from Alice to Bob", () => {
    transaction.transferMoney(db, { from: 'Alice', to: 'Bob', amount: 10 });
    const alice = db.prepare("SELECT amount FROM balance WHERE account = 'Alice'").get();
    expect(alice.amount).toBe(0);

    const bob = db.prepare("SELECT amount FROM balance WHERE account = 'Bob'").get();
    expect(bob.amount).toBe(15);
  });

  it("sending undefined does not transfer any amount", () => {
    transaction.transferMoney(db, { from: 'Alice', to: 'Bob', amount: undefined });

    const alice = db.prepare("SELECT amount FROM balance WHERE account = 'Alice'").get();
    expect(alice.amount).toBe(10);

    const bob = db.prepare("SELECT amount FROM balance WHERE account = 'Bob'").get();
    expect(bob.amount).toBe(15);
  });
});

And the typed implementation:

import { Database } from "better-sqlite3";

export function transferMoney(
    db: Database,
    { from, to, amount }: { from: string, to: string, amount: number }
) {
    db.prepare("UPDATE balance SET amount = amount - ? WHERE account = ?").run(amount, from);
    db.prepare("UPDATE balance SET amount = amount + ? WHERE account = ?").run(amount, to);
}

Assuming you have typescript installed (npm -g install typescript), you can now compile and run the tests:

tsc --strict *.ts && npx jest transaction.test.js

We get an error:

transaction.test.ts:24:63 - error TS2322: Type 'undefined' is not assignable to type 'number'.

24     transaction.transferMoney(db, { from: 'Alice', to: 'Bob', amount: undefined });
                                                                 ~~~~~~

Notice it's an error on the test file, but the test didn't fail, it didn't even run because sending undefined is not allowed. Now for our unwanted behaviour we cannot even write a test, it's actually forbidden, you are forced to delete that test, that's very cool!

We are so strict now, that certainly means we won't have issues with our code anymore right? No? We certainly closed more gaps now, and we can keep going, maybe write more and more unit tests covering things we can remember that can cause problems, maybe getting more creative about types, like introducing monads to handle side-effects and see if it helps.

But even if we go through this effort, what bugs may still be lurking out there? Let's say we wrote as much unit tests as we could and typed everything, how do we know we are still not missing some edge case? Maybe try random stuff and see if something breaks? Yeah! That's a good idea!

Property-Based Testing

In Property-Based Testing, you generate lots of samples to give to your function and write some assertions about it. Because of this sampling, Property-Based is also known as Fuzz Testing, but I like the name Property-Based Testing better, because it highlights the most important concept to grasp about this kind of test: that you should assert on properties that your output must hold*,* not on specific examples like unit tests, on properties. It requires a different mindset to think about properties, personally for me it was hard to do, since I was for many years writing just regular unit tests.

For example, if you are writing a simple sum function, to sum A and B. You cannot write a test to generate a thousand random numbers for A, and for B, and then just assert the result of sum(A, B) should be A + B. That's too easy! You are simply replicating the behaviour of your actual implementation, that doesn't help anyone, both your code and test could be wrong together. Instead you focus on properties, for example, no matter which of those 1000 random numbers are summed together, two positives should always return a positive result, if one of them is negative then is should have the sign of the greater of the two, or any other properties you can think about. If a counter-example is found, the test will tell you.

Let's write one:

import fc from "fast-check";

// ...

it('ends up with more money on the receiving account', () => {
  fc.assert(fc.property(fc.integer(), amount => {
    db.prepare("UPDATE balance SET amount = 10").run();

    try {
      transaction.transferMoney(db, { from: 'Alice', to: 'Bob', amount });
    } catch (_) { return; }

    const alice = db.prepare("SELECT amount FROM balance WHERE account = 'Alice'").get();
    const bob = db.prepare("SELECT amount FROM balance WHERE account = 'Bob'").get();
    expect(bob.amount).toBeGreaterThan(alice.amount);
  }));
});

Basically what this test does is using fast-check to generate random integers (with fc.integer) which is then passed to our callback to make and assertion. And what can we assert it's always true when transferring money from Alice to Bob? What is a property of a transference? Here they both start with 10 pesos, so the property we check here is that no matter how much money is being transferred, Bob should always end up with more money than Alice. We also ignore any errors, as we are only interested in the successful transfers for now.

Let's run it!

tsc --strict *.ts && npx jest transaction.test.js
 FAIL  ./transaction.test.js
  transferMoney
    ✓ transfers money from Alice to Bob (6 ms)
    ✕ ends up with more money on the receiving account (21 ms)

  ● transferMoney › ends up with more money on the receiving account

    Property failed after 1 tests
    { seed: -1111144902, path: "0:0", endOnFailure: true }
    Counterexample: [0]
    Shrunk 1 time(s)
    Got error: Error: expect(received).toBeGreaterThan(expected)

    Expected: > 10
    Received:   10

It found one counterexample: 0. Yeah, Alice is trying to transfer zero pesos to Bob, now it starts coming down to specs of the system you are building, is this supposed to be allowed? Let's say no, because we think it's too stupid to do a zero transfer, and we do a simple fix for it on the top of our transferMoney function:

if (amount == 0) throw "cannot transfer 0 pesos";

Great, the case for 0 is fixed, let's run again

tsc --strict *.ts && npx jest transaction.test.js
 FAIL  ./transaction.test.js
  transferMoney
    ✓ transfers money from Alice to Bob (5 ms)
    ✕ ends up with more money on the receiving account (53 ms)

  ● transferMoney › ends up with more money on the receiving account

    Property failed after 2 tests
    { seed: -1121527945, path: "1:1:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0", endOnFailure: true }
    Counterexample: [-1]
    Shrunk 26 time(s)
    Got error: Error: expect(received).toBeGreaterThan(expected)

    Expected: > 11
    Received:   9

Uh oh, Alice sent -1, we haven't though about that! Like a QA the property-based testing tries to reduce the counterexample to the simplest one that fails, telling us that allowing people to transfer negative amount of money breaks our property. Transferring negative money is not something our business want, so we improve our check:

if (amount <= 0) throw "only positive amounts are allowed to be transferred";

Run the tests again and you will see that they pass, great! What other properties can we ensure? Let's say our business is a good bank, the kind that do not want anyone to have debt. One of the properties we could check then is that Alice always have a positive balance after every transfer:

expect(alice.amount).toBeGreaterThanOrEqual(0);

Property failed after 2 tests
    { seed: -1313110683, path: "1:1:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:2", endOnFailure: true }
    Counterexample: [11]
    Shrunk 25 time(s)
    Got error: Error: expect(received).toBeGreaterThanOrEqual(expected)

    Expected: >= 0
    Received:    -1

The counterexample found is 11, this made Alice end up with -1 because it's more money than there is on the account balance. Solving that is a little more tricky, we need to first check how much balance there is left on the account, let's do it:

const balance_from = db.prepare("SELECT amount FROM balance WHERE account = ?").get(from);

if (amount > balance_from.amount) throw "there is not enough money to be transferred";

Run the tests again, they pass, no counter-examples found! Awesome, our code is stronger than ever.

You can go on and do some more assertions, like if the account does not exist when somebody send something other than "Alice" and "Bob", but I want to focus on something else, which you probably noticed from the first example already, as this is the classic database 101 example. "Where is the transaction? For God's sake wrap this in a transaction already!" - you may say. But why do we need it? How can we prove that we need it? Here is our current code:

export function transferMoney(
    db: Database,
    { from, to, amount }: { from: string, to: string, amount: number }
) {
    if (amount <= 0) throw "only positive amounts are allowed to be transferred";

    const balance_from = db.prepare("SELECT amount FROM balance WHERE account = ?").get(from);
    if (amount > balance_from.amount) throw "there is not enough money to be transferred";

    db.prepare("UPDATE balance SET amount = amount - ? WHERE account = ?").run(amount, from);
    db.prepare("UPDATE balance SET amount = amount + ? WHERE account = ?").run(amount, to);
}

So far our unit tests didn't require us to use a transaction, our types also didn't, neither did our property-based tests, no one found a counterexample for a transaction. We do know, however, from experience, that our code can fail at each of those lines of code, our server could die, leaving the database in an invalid state in the middle of the transfer, not to mention when there are multiple instances running this code in parallel and updating Alice and Bob balances at the same time. To solve it with unit tests we could mock each of those functions and see what happens if they fail, but if this was much larger, mocking every line of code and seeing what happens if they fail is very boring to say the least. Fortunately, there is a better way.

Formal Specification

If property based testing got you thinking about properties, formal specification will get you thinking about properties AND state. As you may have heard, state is the root of all evil. That's why functional programming folks want to push it away with pure functions, to a place we can reason better about it. Having stateless functions makes your program much easier to reason about, but you do need state at some point, otherwise your code is completely useless (like Haskell).

Our previous code have a bunch of hidden state, since if you really think about it, your database is really just a global mutable variable, if some function changing it fails in the middle, or two functions change it at the same time, you will get in an inconsistent state. Good luck debugging that.

Now, Formal Specification tricky part is that it requires us to model all state transitions we have, but then it gives you the power to verify all the states your code can reach, not just simulate random scenarios, but actually checking ALL possible paths. The idea is to leverage the power of mathematics to build a full specification, allowing you to check every intermediary step if you want to (helping to check for issues in case your program halts).

For that we are going to use TLA+, a formal specification language created by Dr. Leslie Lamport, he created it because he found that formal specification helped him to prove the correctness of distributed systems, the most famous of them being the Paxos Protocol. I will not go in details about TLA+ here (I'm a beginner myself), but if you want to learn more about it, I really recommend watching Dr. Leslie's video-tutorials, they are really fun, a bit weird, but mostly fun.

Enough talk, I will just paste the initial version of the specification here and we can go over it:

--------------------------- MODULE TransactionV1 ---------------------------

EXTENDS Integers, Sequences

CONSTANT accounts, initialBalances

VARIABLE balances, msgs

Init == balances = initialBalances
     /\ msgs = {}

TransferMoney(from, to, amount) == balances[from] >= amount (* Account needs to have enough balance, from property testing *)
                                /\ msgs' = msgs \union { [ account |-> from, amount |-> balances[from] - amount ],
                                                         [ account |-> to, amount |-> balances[to] + amount ] }
                                /\ UNCHANGED <<balances>>

DbUpdate == msgs /= {}
            /\ LET msg == CHOOSE msg \in msgs : TRUE
               IN msgs' = msgs \ {msg}
               /\ balances' = [ balances EXCEPT ![msg.account] = msg.amount ]

Next == DbUpdate
     \/ /\ \E from, to \in accounts :
           from /= to /\ \E amount \in 1..balances[from] : (* Send only positive integers, from property testing *)
             TransferMoney(from, to, amount)

(***************************************************************************)
(*                                 HELPERS                                 *)
(***************************************************************************)

RECURSIVE SumBalance(_, _, _)

SumBalance(accs, bal, total) == IF accs = {}
                                THEN total
                                ELSE LET acc == CHOOSE acc \in accs : TRUE
                                     IN SumBalance(accs \ {acc}, bal, total + bal[acc])

(***************************************************************************)
(*                                INVARIANTS                               *)
(***************************************************************************)

TypeOK == msgs \subseteq [ account : accounts, amount : Int (* Amount has to be an number, from static typing *) ]

BalancesAlwaysPositive == \A acc \in accounts : balances[acc] >= 0

TotalMoneyStable == SumBalance(accounts, initialBalances, 0) = SumBalance(accounts, balances, 0)

=============================================================================

At first this code looks very alien, but we can render it for a better visualization with LaTeX. If you have some experience with math symbols, that can help:

Okay let me explain. We are basically trying to replicate our code behaviour and all state changes here. We start with two constants: accounts, those will be Alice and Bob; and initialBalance, as we will put some initial money for each. Then we have two variables: balances, changing as the accounts transfer money to each other; and msgs, which we will use to model the queries that are being sent to the database, more specifically here only the UPDATE statements, we skip SELECT and pretend we can read the balances directly to make the spec simpler.

Then we have four functions. On Init, we define our initial state, nothing special.

On TransferMoney we do basically a replication of our current JavaScript implementation. The way to think about it is a bit different though, it's not imperative as in "do this update to the database", it's actually declarative like "we allow a state transition from wathever messages are already on the db queue union two new messages for updating to and from balances, as long as the from account has enough money.

On DbUpdate we are simulating the database actually changing the state, basically it's saying that if the msgs queue is not empty, we can switch to a state where the msg queue has all items but this first one, and the balances have this new amount on the specified msg.account

Now the Next is the interesting part of TLA+, it specifies what state changes your system can go through at each time step, in our case it's DbUpdate OR TransferMoney for each of the accounts we have, for any amount of money they might have in the balance, as long as it's positive. This is basically to say, the database could be updating and all people could be transfering money, back and forth, all the time, at the same time. Pretty much what happens with a system at scale, distributed or under high load, and we want to see what can happen there, what paths it can go through.

Then we have a helper, it's just a simple function to sum balances that we will use next.

Finally, we have our Invariants! Those are the things we want to verify. Those specifications need to remain true all the time, no matter where we stop on the execution, no matter how many computers we add.

Those, unfortunatly, won't come automatically, they will come from thinking. What properties do you want your system to hold? This depends on your goal, on business requirements, on, well, specifications, we are just formalizing them. Like unit tests, they might have gaps, but being at a much higher-level, a single specification can cover way more ground. This is what Nicholas Tietz meant, we can let Copilot or future AIs generate the code, while we humans focus on the specifications. A language like TLA+ may allow us to define them at the highest level that is still non-ambiguous.

Our first invariant TypeOK is a simple one, it's akim to type checks, we are making sure that on the msgs sent to the database account has always to belong in the constant accounts, and that the amount must be an integer.

Next is BalancesAlwaysPositive, we are a no-debt bank remember, so we assert that any point in time no balance is allowed to go negative, we already checked that with the property-based test and fixed it by introducing and if (amount > balance_from.amount) but could we be missing something?

Finally we have TotalMoneyStable, this is a new rule. Since we don't have deposits or withdraws yet, money cannot just appear out of nowhere right? If Alice started with 10, and Bob started with 10, and there is no one else in the system, but then suddently we have 25 pesos, then where did this money came from? So we add this invariant to make sure that at any point in time the total amount is the same as the initial one, since we have a closed system.

From TLA+ Toolbox, we can run our specification, and see if we violate any invariants. This is the model configuration we use:

Nothing surprising here, just one small detail is that we put just 2 pesos in each account, since if we put much more than that, the state space to verify grows exponentially large. Even though definition is in math, we are unfortunately still limited by mundane computing power.

Alright so let's run the model! 🥁🥁🥁🥁🥁

Oh no, we have an error!

The invariant TotalMoneyStable is violated, how did it happen? We can actually follow the steps the model found. First we initialize with Alice and Bob having 2 pesos each. Next we add two msgs, that Bob should have 1 peso and Alice should have 3, this means Bob is doing a single peso transfer. Next DbUpdate kicks in to update the balances and... ops, Alice has 2 pesos and Bob 1. One peso completely vanished of the system. Well, we know there is still a message left to be processed, but what if your system died at that point? Those invariants guarantee to be always consistent, no eventually*consistent, *always consistent.

The problem happens because our updates happen sequentially. What can we do to solve that? We can do both updates at once, this should solve our problems. Let's update our spec:

Okay now instead of sending 2 msgs to update the database, we are sending a single one, with from, to, and amount. Let's see what happens:

Oh no! Invariant BalanceAlwaysPositive was violated. How did that happen? Let's follow the steps. Initial balances, okay, then Bob did a transfer to Alice for 1 peso, but then before the database had time to process the update, Bob did yet another transfer to Alice, for 2 pesos this time, most likely it was another instance of our service running in parallel. As a result, once DbUpdate processes both msgs, Bob ends up with a negative amount, and that's terrible for our bank because we would have to charge interest and make money! No bank wants that I'm sure.

Alright, I have an idea for one final change:

Now additionally to updating both balances at once, we no longer send the amount to be transferred, but actually the final resulting amount for both balances. This way we are carrying the state inside the message, which means that even conflicting messages when processed will leave the state stable way, although it also means last-write-wins.

Let's run our model again:

Wow no errors! TLA+ could not find any issues on the 3.196 possible states it found. We solved folks, that's it, and we could do it without using transaction or locks! Great!

If you feel like it you can keep going, what would happen if we add another account? Say Alice, Bob and Carlo? Can you simulate the state changes that happen during a database transaction? Does it solve our problems?

Conclusion

Making a sure a function free of bugs is very hard, and the definition what "free of bugs" mean actually depends on your specification. On our every day jobs most of the issues are avoided by plain human experience and judgment, if you could not see the code, guaranteeing it is working is super difficult. Tools like property-testing and TLA+ allow us to get closer and closer, and while I have no idea if those tools will indeed help us bridge the connection with AI, I know they are fun to learn.

Thanks for reaching the end of this blogpost, let me know how you like it. I need to end up with a warning, though: we only tested the algorithm here, there is a lot more to software than that, this is just a tiny part.

Now I kindly ask you to not leave yet and save the links listed below to read or watch later, they are more interesting than this one, and modeled my thinking about testing.

Band-aid Driven Development

Rogério Chaves — Thu, 10 Dec 2020 17:25:42 +0000

When you do agile you don’t want to have silos, people learned long ago that having a separate UX team, then a frontend team, then a backend team, does not work well, ideally you want to split your product not by technical division, but by business area or product, and the team should be able to deliver new features, end-to-end

You can think this as a horizontal and vertical divisions, horizontal layers are like infra, backend, frontend, UX, and you can lay one on the top of the other, like a pie. Now, when you cut this pie you will want a vertical slice of it, not a horizontal one.

Nevertheless, as systems grow more and more complex, and you still need new features that crosses all those systems, it becomes pretty impossible for a single team to vertically own everything of it.

I will use one example I’ve been through: payments. We had to implement a new payments solution on a marketplace product, which involved many systems such as the shopping UI, pay-in, accounting and pay-out, you can see those as technical layers, and we needed to go through all of them. Each one of those is complex enough to require domain-specific knowledge, so it’s owned by a separate team. It would be very hard to split those people in more vertical teams, this technical division at some point of complexity makes sense.

All those systems needed new developments in order for the new flow to work, but since each is owned by a separate team, how do we do this? If we build one piece after the other then this would be quite waterfall, and bring all the problems waterfall has with it, lots of rework, bad feedback loop, communication problems, etc. We can ask each of those teams to develop it in parallel, but this doesn’t make things much better, at the end each team may have a ready system, but they actually do not talk well with each other, they don’t account for problems you only see during integration, communication is still poor. This, in fact, was what happened on the project I was.

My team specifically was tasked with implementing the UI part, as all the other parts were “ready”, although nobody knew for sure because we never integrated, teams had already moved on to other priorities. As we started to integrate, we noticed some problems, so we changed to instead of focusing on the UI bit, our priority number 1 became the integration of all the systems, end-to-end.

My excellent colleague Ivan Trapeznikov came with a genius idea: let’s just band-aid everything! We don’t care if things are not working properly, we just want to see at least something working end-to-end. To do that, instead of thinking of all possibilities, we focused only on one use case, only the most important happy path: customer puts credit card number and partner is paid out. No invalid credit card, no rejected transactions, no failed configuration, we can worry about those things later.

So on we went, and found many problems along the way, like the credit cards encrypting system was missing an extra field, what did we do? We hardcoded the field from a card we had and deployed it production. Parameters missing? Data not in correct format? Just hardcode it. Payment notification is not returning the expected status? Just pretend it was, deploy and… done! Wherever the flow stopped, we band-aided it so it could keep going.

After many band-aids we finally had something “working”. Of course only under a flag, with controlled test users, in a very specific scenario, but at least all systems were talking, end-to-end. Teams realized they were actually not “ready” as they thought they were. Then, because we kept track of all the band-aids, we could actually see which ones were more impactful or risky and needed to be fixed first, those are things you cannot find when focusing on development of a single thing at a time.

The band-aid approach worked really well, we manage to prioritize and fix things, removing the band-aids one by one, by collaborating with other teams and bringing their focus back to this flow until it was, actually, ready, which for me it means being used by real users in production.

One big advantage of this approach rather than fixing all integration issues at the end is that we can flag blockers early on, reducing the time the team would be idle or changing focus waiting for some other team’s development, reducing feedback cycle and risk to build upon integrations that don’t work (the earlier you find a problem, the cheaper it is to fix), leading to less wasted effort and a better time-to-market.

Today I still advocate for always trying to think in any project how could you go end-to-end first as much as possible. As a follow up, to achieve that from the start of the project and reduce the need for band-aids, an idea I like a lot is splitting stories using the hamburger method, I recommend the reading:

https://gojko.net/2012/01/23/splitting-user-stories-the-hamburger-method/

Cheers!

Is Neymar Better Than Average?

Rogério Chaves — Sat, 12 Sep 2020 17:52:34 +0000

Do people on dev.to like data analysis posts notebook style? Well, let me know if you like it or not.

I've been reading the book Doing Bayesian Data Analysis, and there is an example there with baseball, which I'd like to try for football.

Basically, we want to estimate what's the probability of a player scoring in a match, and which part of that probability is influenced by their position on the field vs other factors. We'll focus on Neymar but I'm actually using the data of all players from UEFA Champions League.

First let's load our data. I collected data from UEFA Champions League for 2018 and 2019-2020, each row is a participation of a player in a match, with which position they were on this match, and how much goals they scored. Players that were in substitution bench during the match but never entered the field are not included

import pandas as pd
import numpy as np

df = pd.read_csv("uefa_player_matches.csv", parse_dates=["match_date"])
df = df.dropna()
df = df[df['position'].isin(["G", "D", "M", "F"])] # excluded "SUB" and "-" positions
position_map = {'G': 'Goalkeeper', 'D': 'Defender', 'M': 'Midfield', 'F': 'Forward'}
df['position'] = [ position_map[x] for x in df['position'] ]
df.head()

	match_id	match_date	player	player_id	team	team_id	position
0	158963	2019-06-25 13:00:00+00:00	Mattia Migani	80722.0	Tre Penne	700	Goalkeeper
1	158963	2019-06-25 13:00:00+00:00	Davide Cesarini	56095.0	Tre Penne	700	Defender
4	158963	2019-06-25 13:00:00+00:00	Mirko Palazzi	56099.0	Tre Penne	700	Defender
5	158963	2019-06-25 13:00:00+00:00	Nicola Gai	80723.0	Tre Penne	700	Midfield
7	158963	2019-06-25 13:00:00+00:00	Nicola Chiaruzzi	80724.0	Tre Penne	700	Midfield

Let's take a look on matches that Neymar participated

df[df['player'] == 'Neymar']

	match_id	match_date	player	player_id	team	team_id	position	goals
4360	240579	2019-11-26 20:00:00+00:00	Neymar	276.0	Paris Saint Germain	85	Forward	0
5012	240603	2019-12-11 20:00:00+00:00	Neymar	276.0	Paris Saint Germain	85	Midfield	1
5192	292852	2020-02-18 20:00:00+00:00	Neymar	276.0	Paris Saint Germain	85	Forward	1
5451	292853	2020-03-11 20:00:00+00:00	Neymar	276.0	Paris Saint Germain	85	Midfield	1
5610	570966	2020-08-12 19:00:00+00:00	Neymar	276.0	Paris Saint Germain	85	Forward	0
5698	589000	2020-08-18 19:00:00+00:00	Neymar	276.0	Paris Saint Germain	85	Forward	0
5731	591151	2020-08-23 19:00:00+00:00	Neymar	276.0	Paris Saint Germain	85	Forward	0
8419	39151	2018-09-18 19:00:00+00:00	Neymar	276.0	Paris Saint Germain	85	Forward	0
8930	39170	2018-10-03 16:55:00+00:00	Neymar	276.0	Paris Saint Germain	85	Forward	3
9504	39191	2018-10-24 19:00:00+00:00	Neymar	276.0	Paris Saint Germain	85	Forward	0
9740	39199	2018-11-06 20:00:00+00:00	Neymar	276.0	Paris Saint Germain	85	Forward	0
10421	39224	2018-11-28 20:00:00+00:00	Neymar	276.0	Paris Saint Germain	85	Forward	1
10657	39232	2018-12-11 20:00:00+00:00	Neymar	276.0	Paris Saint Germain	85	Forward	1

Just to help us understand our data a bit, what's the distribution of goals per match?

df['goals'].hist();

Woah a Poisson distribution as expected, I love those. Okay and which positions scores more?

df[['position', 'goals']].groupby('position').sum()

position	goals
Defender	135
Forward	587
Goalkeeper	3
Midfield	449

Alright zero surprises so far. Let's move on and build a Logistic Regression model, which will give us a probability of a goal given a player and their position.

from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import OneHotEncoder

X = df[['player_id', 'position']]
one_hot_encoder = OneHotEncoder()
one_hot_encoder.fit(X)
X = one_hot_encoder.transform(X)

y = [ 1 if y > 0 else 0 for y in df['goals'] ]

model = LogisticRegression()
model.fit(X, y)
model.score(X, y)

0.9107711354793067

Logistic regression can fit our data with 91% accuracy, seems pretty good. Now let's take a look on what's the probability of Neymar scoring a goal in some future match:

neymar_id = 276
X_example = one_hot_encoder.transform([[ neymar_id, "Forward" ]])
model.predict_proba(X_example)[0][1]

0.3835193670312905

38%, not bad, what if Neymar were a Midfield player instead of Forward?

X_example = one_hot_encoder.transform([[ neymar_id, "Midfield" ]])
model.predict_proba(X_example)[0][1]

0.2424111808266389

24%, is it good? Is it above average? I think so, but let's check what's the average of player in this positions scoring at all. For that we will take the average of probabilities of all players in that position

def prob_for_position(position):
    X_example = []
    for player_id in df[df["position"] == position]['player_id'].unique():
        X_example.append([ player_id, position ])
    X_example = one_hot_encoder.transform(X_example)
    ys_pred = [ y[1] for y in model.predict_proba(X_example) ]
    return ys_pred

forward_position_probs = prob_for_position("Forward")
print("Forward position mean", np.mean(forward_position_probs))
plt.hist(forward_position_probs, bins=50)
plt.show()

midfield_position_probs = prob_for_position("Midfield")
print("Midfield position mean", np.mean(midfield_position_probs))
plt.hist(midfield_position_probs, bins=50);

Forward position mean 0.165252941308293

Midfield position mean 0.08585727912713484

It's interesting to notice that the histograms above remember somewhat a beta distribution, closer to the left side, never smaller than zero, this will be important for us later

But answering our question, yes, Neymar is way above average, as an average forward would score with 16.5% chance while Neymar gets 38% on forward position. It seems that being Neymar does put a hell lot of weight in the probability to score in a match.

Remember: this doesn't really mean that Neymar the person alone is way better than everybody else, because many other variables such as "Playing in PSG with good team members" are aggregated in "being Neymar", as it's not really possible for us to deconfound this variable right now. On the other hand the position variable (forward, midfield) is pretty deconfounded as we measure across all players

So what would be the probability of Neymar scoring, regardless of position?

def prob_for_player_id(player_id):
    X_example = []
    for position in df['position'].unique():
        X_example.append([ player_id, position ])
    X_example = one_hot_encoder.transform(X_example)
    ys_pred = [ y[1] for y in model.predict_proba(X_example) ]
    return np.mean(ys_pred)

prob_for_player_id(neymar_id)

0.18960539993655143

What about Messi?

messi_id = 154
prob_for_player_id(messi_id)

0.21051765637224326

Seems like Messi has a bit higher probability of scoring than Neymar, 21% vs 19%

Those are nice point estimations, but you shouldn't take them as absolute truth, because our data is just a small sample from all matches by those two players (only 2 UEFA championships), as our input is incomplete, how much can we really trust the data we collected?

One alternative approach that helps us with that is building a Bayesian model, where we can estimate distributions over our parameters, as you will see next

Measuring uncertainty with Bayesian Analysis

For this next part I’ll use R with rjags because that’s what I learned from the book. I even tried to do the same with PyMC3 but no success so far.

Anyway, first, let’s load and transform the data just like we did on previous python notebook

df <- read.csv("uefa_player_matches.csv", header = TRUE)
df <- df[complete.cases(df), ] # drop na
df <- df[(df$position %in% c("G", "D", "M", "F")),]
position_names <- list("G" = "Goalkeeper", "D" = "Defender", "M" = "Midfield", "F" = "Forward")
df$position <- factor(apply(df['position'], 1, function(x) position_names[[x]]))
df$player_id <- factor(df$player_id)
head(df)

	match_id	match_date	player	player_id	team	team_id	position
1	158963	2019-06-25T13:00:00+00:00	Mattia Migani	80722	Tre Penne	700	Goalkeeper
2	158963	2019-06-25T13:00:00+00:00	Davide Cesarini	56095	Tre Penne	700	Defender
5	158963	2019-06-25T13:00:00+00:00	Mirko Palazzi	56099	Tre Penne	700	Defender
6	158963	2019-06-25T13:00:00+00:00	Nicola Gai	80723	Tre Penne	700	Midfield
8	158963	2019-06-25T13:00:00+00:00	Nicola Chiaruzzi	80724	Tre Penne	700	Midfield
9	158963	2019-06-25T13:00:00+00:00	Enrico Cibelli	80725	Tre Penne	700	Midfield

Let's just setup some variable we will need to build our model: y, which means weather there was a goal or not, positions and playerIds

y = apply(df['goals'], 1, function(x) if (x > 0) { 1 } else { 0 })
positions = as.numeric(df$position)
playerIds = as.numeric(df$player_id)
playerIdsUnique = unique(playerIds)

playersMostCommonPositions = c()
for (playerId in playerIdsUnique) {
  playerPositions = df[(playerIds == playerId),]
  sortedPositions = sort(table(as.numeric(playerPositions$position)), decreasing=TRUE)
  playersMostCommonPositions[playerId] <- as.numeric(names(sortedPositions)[1])
}

Now the most exciting part, let's build and compile our hierarchical model!

modelString = "
  model {
    for (positionId in 1:length(positions)) {
      meanPosition[positionId] ~ dunif(0, 1)
      variancePosition[positionId] ~ dunif(0, 1)
    }
    for (i in playerIdsUnique) {
      # Creating a beta distribution based on mean and variance: https://en.wikipedia.org/wiki/Beta_distribution#Mean_and_variance
      mean[i] = meanPosition[playersMostCommonPositions[i]]
      variance[i] = variancePosition[playersMostCommonPositions[i]] / 10
      v[i] = ((mean[i] * (1 - mean[i])) / variance[i]) - 1
      alpha[i] = mean[i] * v[i]
      beta[i] = (1 - mean[i]) * v[i]

      probPlayer[i] ~ dbeta(alpha[i], beta[i])
    }
    for (i in 1:length(y)) {
      y[i] ~ dbern(probPlayer[playerIds[i]])
    }
  }
"

writeLines(modelString, con="TEMPmodel.txt")
jagsModel = jags.model(
  "TEMPmodel.txt",
  data=list(
    y = y,
    positions = positions,
    playerIds = playerIds,
    playerIdsUnique = playerIdsUnique,
    playersMostCommonPositions = playersMostCommonPositions
  )
)

So, let me try to explain everything. First of all, our goal is to model the probability of a goal happening or not in a match, so this is clearly a bernoulli trial, which is about true/false results, this is defined by the y[i] ~ dbern part. The argument inside dbern is the probability for the given player, but how do we define that?

Player probability of scoring is a beta distribution, as we saw on the previous python notebook. We want to find the probability of scoring for each player, for example, a player with a 0.5 mean probability would probably score every other match. A beta distribution is defined by parameters alpha and beta, but because those are very hard to intuitively interpret, I used a formula to be able to define it based on mean and variance. But who defines this mean and variance then?

The player position! Each position (Forward, Midfield, Defender, Goalkeeper) will have a mean probability of goal that fits for all players, simply defined here by a uninformative uniform distribution between 0 and 1. So the players influence the mean position probability, and the position in turn affects individual players probability to score. This is the magic of Bayes hierarchical models!

update(jagsModel, n.iter=500) # burn-in
parameters = c("probPlayer", "meanPosition", "variancePosition")
codaSamples = coda.samples(jagsModel, variable.names=parameters, n.iter=5000)

Alright, so finally, let's plot our posteriors and start asking questions! What's the mean probability of a forward player to score in a match?

mcmcMat = as.matrix(codaSamples)

positionLevel = which(levels(df$position) == "Forward")
positionMeans = mcmcMat[,paste("meanPosition[", positionLevel, "]", sep="")]
plotPost(positionMeans, credMass=0.95, xlab="Forward mean HDI")

Between 0.159 and 0.196, which is a cool advantage of using Bayesian methods, it's common to look at distributions rather than point estimates, so we can have an idea of our uncertainty. What about Neymar, is he in this range? Or is he conclusively better than average?

neymarId = 276
neymarLevel = which(levels(df$player_id) == neymarId)
playerProbs = mcmcMat[,paste("probPlayer[", neymarLevel, "]", sep="")]
plotPost(playerProbs, credMass=0.95, xlab="Neymar HDI", compVal=0.196)

Well the mean is similar to what we got using Logistic Regression, 0.38 vs 0.328 here. But is he conclusively better than average? Almost but no!* The lower bound for Neymar is 0.148, which is inside the average range, so it seems like he is probably better, but variance is quite high, maybe it was just luck all this time? What about Messi?

* considering 95% HDI

messiId = 154
messiLevel = which(levels(df$player_id) == messiId)
playerProbs = mcmcMat[,paste("probPlayer[", messiLevel, "]", sep="")]
plotPost(playerProbs, credMass=0.95, xlab="Messi HDI", compVal=0.195)

Yes! Messi lower bound is 0.204, considering 95% Highest Density Interval he is conclusively better than average forward position players! It does not overlap with the upper bound of 0.196 for average forward players.

That was it, just a small bayesian exercise, I hope it was as fun to you as it was for me.

Disclaimer: I'm not an expert so I may not be interpreting this results correctly, please let me know if I don't.

If you also want to play with it, you can find the dataset and the notebooks on my github.

Thanks for reading!

Designed v Evolutionary Code

Rogério Chaves — Thu, 30 Jul 2020 20:03:01 +0000

When I started coding I didn’t know any good practices, I just coded and coded and you can guess that the result was horribly hard to maintain and extend, I spent nights awake solving stupid bugs to meet stupid deadlines. That’s awful, there should be a better way.

Then I entered college, and I learned software analysis with UML, service management with ITILv2, among others. Amazing! So if I plan my software in advance it will be perfect and I won’t face the problems I was having before right? Wrong. The waterfall approach never worked, I could never predict things right, and I kept developing poor systems.

Next I started working at ThoughtWorks, they taught me Agile, and I threw almost everything I learned at college away. Instead of planning everything in advance, we should just plan small things, one at a time, but it doesn’t lead to crazy messy code like when I started because we ensure quality at each step, by using TDD, Pairing, Code Reviews, Refactoring, and so on.

In my project there we had a beautiful test suite pyramid, I saw zero bugs in production for the first 6 months. Honestly when I started I found it weird that people took days to deliver what I previously took hours, because of all this process, tests and refactoring, but they were surprisingly calm about it, following the plan one user story at a time, and after some weeks and months I could see how it paid off, how it was actually faster on the long run.

For many years that was it. It’s solved, that’s how we deliver software, it’s settled, I don’t need learn anything new. But then I joined Booking.com.

No tests, no merge requests, no pairing, huge perl codebase, code duplication, not enough refactoring. Are you crazy? I’m back at square zero, like when I learned to code, this doesn’t work. But it did, it did work, user stories were delivered at more or less the same pace as on the previous environment I was, how was that possible?

How did a company using this work style even got to be this big was beyond me, until Kent Beck, father of TDD, on a surprisingly similar experience at Facebook, solved the puzzle with his 3x theory: https://www.youtube.com/watch?v=FlJN6_4yI2A

I really recommend checking the video above, but here I’d like to focus more on the code.

A Different way of Working

The way I compare the two different ways of working is by thinking one as Designed Code and the other as Evolutionary Code. Both of them have the goal to save time, both of them are produced by agile, but the result is very different.

On Designed Code you don’t want to have bugs, you want strictly typed languages (like Elm, the language with zero runtime exceptions), a Test Pyramid, a well-though architecture and no tech debts, so that you are free to develop new features faster.

But guess what, those don’t come for free, and there is an alternative way: Evolutionary Code. On Evolutionary Code you always focus on the next thing that will bring the system more value, always considering the law of diminishing returns. You don’t guarantee that everything will work, instead you just release your code, measure where it fails, then you do the solution that solves 90% of the problems with the least effort, ad infinitum.

Basically on Evolutionary Code your code evolves not because you plan, but because the environment around it pushes it into certain directions.

Language

A strong type system will force you to solve every tiny possible issue, making you spend a while fighting with the compiler (anyone who played with Rust knows what I’m talking about). This gives a great feeling of safety, but some of those issues may actually never be a problem in real life, or rare enough that the business can afford while you spend your time on a bigger user pain.

That’s why I’d say Evolutionary Code matches better with interpreted languages, and it is better if it doesn’t try to fake safety at runtime. For example, if you have JavaScript (not Typescript!):

function trackClick(name) {
  if (!name) throw "track needs a name";

  ga.track(name);
}

Nothing guarantees that the code will be inside a try/catch, so do you really want to break your application just because a Google Analytics tracking didn’t work? No, you let it run. You should only throw/panic/die when it’s dangerous or impossible to continue, otherwise let it burn. There is a reason why JavaScript allows you to sum object with array, there is a reason why == compares number with string, it is all so your code can keep working, even if by accident. If you cannot get real safety from your language don’t try to fake it, embrace the speed boost of running without protections, fail fast and fix your code accordingly.

Likewise, when you are doing Designed Code, and you have all of the required boilerplate in place, and still gets a NullPointerException in your face it breaks your heart. It’s easier when you can either fully trust or fully not trust the system, when its in the middle it just causes more uncertainty and stress.

Code Changes

When your system grows complex, a Designed Code will have a single source of truth for the business logic, with a well architected system. So when you need to make a change on it, you have the advantage to be sure that all your system is in sync.

But this also means that a change on one place will break many others, so you are forced to fix everywhere, which can take you a long time.

On Evolutionary Code it’s common to end up having lots of duplicated code, this causes you to take a long time to keep all system in sync. On the other hand you can change one place at a time without disrupting others, prioritizing the most important ones (you will forget some, but hey, maybe they were not important anyway).

When you just go with the wind, and don’t refactor often, its also common to see some God Classes show up. Those are files with 10k+ lines of code, functions with 30 parameters, accumulating responsibility of basically everything. You cannot kill it, it’s too important, you also cannot avoid it because a big part of the code goes through there, so what you do when you need to add new functionality to it? You do the minimum possible.

Like the human body, it’s very risky to do interventions in complex systems that we don’t fully understand, so like a doctor we just do an incision as small as possible. In Evolutionary Code it’s common to create simple functions, with simple code (just ifs and for loops, no magic), that does one thing only (hey, that’s a good pattern!), you can test it, you can trust it, then you just do the smallest incision in the God Class to call your new small function.

Designed Code doesn’t have this problem, but it doesn’t mean it’s faster to add new code. Understanding a very engineered code also takes a lot of time, figuring out where your code should be in the middle of all those AbstractFactories, which Interfaces to use, reading docs and figuring out what was the team thinking when they developed a certain architecture takes time. Or, to be more up-to-date, figuring out how your new feature fits in between those five microservices takes time.

In general, Evolutionary Code tends to not have much abstractions, just basic syntax, so it’s easier to read, but you need to read lots of it. On the other hand Designed Code tends to have more abstractions, so you need to read less, but it’s not simple, you have to understand well those abstractions.

Tests

When you are doing Designed Code tests are essential, you need to know your system is fully working according to the specs. When you are doing Evolutionary Code monitoring is essential, you need to grab every piece of feedback you can from production to evolve your code accordingly. Resistant vs resilient.

Tests are good for regression, it’s great if you are doing incremental changes and you can have an early feedback of everything that broke, so you don’t need to later stop your development process to fix a bug in production. But if business decides to do a major shift, then you have not only to change a big chunk of your codebase, but also a big chunk of your tests, you cannot count with the regression guarantee anymore, and it can actually slow you down.

In Evolutionary Code tests and TDD are still very useful, when it helps you to develop faster, but not very useful as a regression suite. When you are writing Evolutionary Code, instead of running the whole system you build some unit tests or scripts to help you develop the new feature, but it’s common that after a while those tests will start breaking, with no real impact to the customer, causing people to ignore it on the CI. Other times things move so fast that the feature is no longer relevant so the test can simply be deleted. In this world you rely more on monitoring and data to understand what is working well and what isn’t, fixing and improving accordingly.

Likewise, Code Reviews can help you catch bugs that you wouldn’t have seen otherwise, but at the same time they delay integration and getting feedback from production, maybe the bugs you catch and optimizations you did on the review will actually not help in production while the ones you didn’t causes a major issue.

Which to Choose?

It depends.

I really don’t want you to read this and decide that one way or the other is better. The whole point is that I’ve seen both working, I now personally believe that Evolutionary Code can help us better on the Explore and Expand phases mentioned by Kent Beck, while Designed Code can help us better on the Extract phase, or on highly regulates environments.

I also think that piece of mind is absolutely important for something you will be doing 40 hours a week, and Designed Code can help you better there. But also, only now I notice how we can get comfortable with incremental changes and stop doing major shifts, which can be crucial for big companies to stay relevant and for startups to grow.

Thanks for reading!

Unit: universal test generator tool

Rogério Chaves — Mon, 27 Jul 2020 16:18:07 +0000

Do you know how to write tests? Cool! Can you write new test cases easily? Cool. But do you remember how did you set up the tests runner in your project? Me neither.

Once you learn, writing tests is pretty straightforward, and pretty standard across all languages, you have a way to declare the test cases, either with it "does something" or myTestCase() and you assert with something like expect(a).toBe(b) or assertEqual(a,b), so you can pretty much already test even in a language that is new to you. But how to set up, which test runner to choose and how to run them can vary a great deal.

I don’t know about you, but every time I start a new project I find setting up all the boilerplate, build tools and pipeline the most boring part, I just want to jump into the code, and I never know what’s worse: having to setup all this myself, or learning how to use an existing boilerplate from the internet and fighting it later. That’s why I start simple, as simple as I can in the language at hand, with just a main file and no dependencies if possible, and add things as I need. Problem is, I like doing TDD, so I need to at least setup the minimal boilerplate to run the tests, and a first example to look at so I can copy-paste for next tests.

“I never remember how to write tests in python, let’s copy from that old project again”, “How do I setup rspec again? I think I’ll just skip tests for this project”, “I know jest is easy to install but I feel lazy to go to their website and read the getting started for the 10th time”, “I’m starting to learn R, I wish I could write tests to it already, it would help me learn” — those were all thoughts I had, that’s why I created unit.

Unit is a CLI tool to setup and generate tests for all programming languages (if your language is not there pull requests are welcome ;)). Now there is only one command you need to memorize:

unit path/to/file

This file can be anything, file.rb, file.js, File.java, etc. Unit will try to identify the language, create a test file for it, add and setup the dependencies if needed and tell you the command to run it, for example:

Go to the github repo for instructions on how to install unit and all the languages supported. Feel free to open any issues there, feedbacks are very welcome.

Thanks for reading!

DEV Community: Rogério Chaves

Building a fully testable Lovable clone agent in Python with Scenario

Scaling from a Billion to a Million to One

The most useful command to clean your server hard drive

The Pragmatic Pragmatic Programmer

pragmatic

The Pragmatic Programmer: The Good Parts

The Dogmatic Programmer

Dogmatic Programmer: gullible, does what feels right, following idealistic theories

Pragmatic Programmer: skeptical, values scientific rigour (when possible)

You can repeat yourself

Dogmatic Programmer: Duplication is evil

Pragmatic Programmer: It depends

Dogmatic Programmer: let’s make everything abstract, what if we need to switch databases?

Pragmatic Programmer: we will never switch databases

Give a random number

Dogmatic 1: believes on estimations, and that one day they they will get very good at it (this day never comes)

Dogmatic 2: believes that estimations are never possible to get right anyway, so they are useless

Pragmatic: manages expectations, not accuracy

Can you guarantee that a function has no bugs?

Unit Tests

Type Checking

Property-Based Testing

Formal Specification

Conclusion

More Content

Band-aid Driven Development

Is Neymar Better Than Average?

Measuring uncertainty with Bayesian Analysis

Designed v Evolutionary Code

A Different way of Working

Language

Code Changes

Tests

Which to Choose?

Unit: universal test generator tool