DEV Community: Petr Muller

Book Notes: The Art of Agile Development, 2nd ed. Shore, Larsen, Klitgaard and Warden

Petr Muller — Tue, 28 Feb 2023 01:15:56 +0000

Summary

My copy of The Art of Agile Development (AoAD) was the second edition, released in 2021. I never read the first edition from 2007. Still, I understand it was an early influential Agile book that treated the subject less dogmatically and in a less process-oriented way than Scrum literature, which has gained much attention in the past 15 years - both positively and negatively. The second edition is similar. It is a combination of a cookbook and a handbook, emphasizing the advantages of this approach to software development.

The book’s structure follows the Agile Fluency model developed by Shore and Larsen. It begins with a section on implementing Agile in organizations that lack the necessary buy-in and support. The Agile Fluency competencies are Focusing (producing value), Delivering (efficiently delivering value), and Optimizing (increasing business value). Each fluency section contains three to six chapters on related topics. These chapters are further divided into short subchapters dedicated to individual practices related to the topics. The practices are primarily, but not exclusively, based on XP ideas. The authors usually discuss the sources that influenced the practices described in the book.

The subchapters on practices follow a standardized structure and are very practical. The authors link individual practices to related ones in the “Ally” boxes, hint at the practice’s target audience, and provide FAQs, checklists, alternatives, and further reading. The book is presented as a collection of various ideas and techniques that can be followed, modified, or ignored but do not form a strongly defined, prescribed methodology. The authors stress that the practices are synergistic, though. They recommend following them entirely at first and only modify or discard after some time when the team is familiar with them, and any shortcomings can no longer be attributed to the chaos of learning new things.

Thoughts

Compared to books on Scrum or Kanban, AoAD does not present a specific method to be followed but rather a loose collection of practices that work together synergistically. I appreciate this approach because it is more flexible in real-world situations. I am convinced that rigorous methods followed to the letter work well only when fully embraced. Since that rarely happens, the result is often dysfunctional (time-wasting ceremonies, absent product owners, Kanban without limiting the work-in-progress, etc.). The practices in AoAD are very pragmatic, low-tech, and low-process, true to the Agile Manifesto.

I appreciate that the authors mostly avoid preaching the usual Agile clichés about magical, universal improvements that are supposed to happen after an Agile method is adopted. Quite the opposite; the authors warn that adoption takes time and effort, and for a short period, the actual performance will be lower. Only as the team learns the new skills does it perform better.

The book can be a helpful reference for various situations. Initially, I was uncertain about the Agile Fluency model, but after further consideration, I concluded it provides a valuable framework for the team’s major competencies. A team can begin by evaluating which competencies need improvement (most teams will likely want to enhance their Focusing or Delivering abilities), then choose a relevant topic and try the suggested practices. Alternatively, the book can be consulted when a team is facing an issue and unsure how to resolve it: find the topic closest to the problem and review its practices or follow the Ally links to related topics.

The limitation of AoAD is that it heavily focuses on co-located teams. Many practices are designed for a team working together in the same physical space, such as using index cards and post-its. The book acknowledges this and attempts to adapt the practices for online tools, but these sections are often vague and shallow. I don’t think it works; even when the book suggests that all team members should use a tablet to interact with an online board service, I can’t imagine this being effective. The book would have been better off if it had omitted these attempts entirely. Distributed teams should look for books that address their needs more precisely.

Lastly, I occasionally felt that the depth and detail of individual chapters were not balanced. Some topics are described in great detail and contain much helpful information. Others read like the authors needed some content to be filled into the common structure, so they wrote one or two paragraphs of generic, bland content. These parts sometimes even regress to fluffing the text with generic Agile preaching about happier teams, more business value, etc. In several (few) cases, the authors go into menial details of a particular practice, providing a step-by-step example of a process that is not that complicated to deserve such detailed treatment. A little more strict editing could have removed these low-value items, resulting in a shorter, more readable book.

Interesting Bits

This (highly subjective) section contains the bits that I found interesting, surprising, original, or significantly different from similar works.

I like the suggestion to start adopting methods in a rigid, by-the-book fashion. I agree with the reasoning that most Agile techniques are counterintuitive and synergistic. Discarding ideas before trying them or without long enough learning time is a waste of opportunity. I am convinced that many teams do this. This is very hard to do without a dedicated coach because the peer pressure of skeptics often leads to abandoning experiments too early.

I enjoyed the parts that dealt with investments into agility and change. I appreciate the admission that Agile transformation needs time and dedicated effort. I find the systematic writing on organizational pathologies and the methods to overcome them helpful - especially the parts that predict the common “setbacks” that, without proper handling, could lead to abandoning progress because the new methods “do not work.”

There is a high focus on low-tech, highly efficient, and collaborative techniques throughout the book: many use simple cards and post-its on a whiteboard. All participants interact with the board: rearranging, writing new cards and discarding them. I find these intriguing, especially in contrast to trying to do the same in a behemoth like Jira, usually through the hands of a facilitator that furiously types words on a shared screen while the participants awkwardly wait until all words are typed. I do not know how to achieve the efficiency of a single whiteboard in a remote setting.

I liked seeking consent instead of consensus: people can be in favor, against, or abstain. The team proceeds if at least someone is in favor and nobody is against it. People who are against it must explain why. If they are not convinced, the proposal is not adapted. This gives power to individuals to pick their battles.

I really liked the suggestion that “stories are reminders to have a conversation”. Again, this is a contrast against the hideous, detailed Jira cards, with sections and checklists and acceptance criteria. I like how this follows the “pure” Agile Manifesto ideas. Stories that need all this likely should be broken down.

I also really liked the system of techniques that work with sizing the stories “just right.” This eliminates a lot of process (Fibonacci scale planning pokers of the world) and plays really well with team capacity, evening out small deviations in estimation if the team behaves consistently. I appreciate that the authors stress that teams have limited agency in setting their capacity. In the long term, the team will learn and be able to do more, but any efforts to pressure the team into achieving more are doomed.

I really like the concept of “slack”: the built-in unstructured time in a team’s iteration. The slack is a time set aside to iron out wrinkles in an iteration and allows teams to deliver the planned work despite surprises and changes. Otherwise, the slack should be used to improve the ability to deliver: pay the technical debt, improve automation, learn new techniques, etc. I find the notion of a capacity buffer that should be invested in the team’s long-term productivity useful.

I liked guiding impediment removal by identifying whether the source of the impediment is in the team’s control, the team’s sphere of influence, or “Soup .”The “Soup” is everything outside the influence of the team. Impediments coming from the “Soup” need to be addressed by changing the team response.

I have never worked in a team that would practice the more extreme collaboration techniques like swarming on stories (the team breaks stories into tasks, then the whole team simultaneously works on them, instead of everybody working on a separate story) and pair or even mob programming. I want to try that someday. It is probably harder in distributed teams, especially ones spread across time zones.

Standup is not a status meeting; it is a coordination meeting. Continuous Integration is a practice, not a tool.

Waffles, Fries, Beer and Developers; Notes from FOSDEM 2023

Petr Muller — Thu, 09 Feb 2023 01:09:56 +0000

Conferences are back! After pandemic-ridden years that felt like an eternity, I was excited to return to FOSDEM, one of Europe's most prominent open-source conferences. Unfortunately, I realized too late that the event was happening and missed the opportunity to submit a talk. Still, I decided to attend in person. The conference was held in Brussels, just a short and convenient 1.5-hour flight from Prague.

The train connection from Brno to Prague has been notoriously unreliable in recent years, even on its best days. With a weather warning issued for Friday, I went very early to compensate for any potential delays or surprises. I managed to get some work done, arrived in Prague, and went to the airport. I set camp in the airport coffee shop and waited for my wife to pick me up.

My wife works for the česko.digital NGO. They facilitate skilled volunteering projects that help Czechia with digital transformation. She went to FOSDEM to participate in the community and law and policy tracks and meet other community members in person. I'm a loner, so I just planned to attend talks.

Saturday

FOSDEM is tough for tech enthusiasts who like to explore a variety of topics. It consists of multiple mini-conferences (known as DevRooms) rather than one large event. Each DevRoom has different slot lengths and break times, making it difficult to switch between them. It's best to find a DevRoom of interest and stay there for an extended period. On Saturday, I spent most of my time in the Go Devroom.

Here is a list of talks I attended, along with my notes, links, and scribbles. Taking notes is not my strong suit, so I decided not to turn this into a coherent paragraph of English. Otherwise, it would have stayed on my computer and never been seen again. Apologies.

The State of Go

Speaker: Maartje Eyskens

Walk through the changes in 1.19 and 1.20. The walkthrough was speedy, and I could have taken better notes.
Structured logging proposal for the future (replace logrus etc.)

“Of course it’s impossible to not break kubernetes somehow”
(on Go history of breaking backward compatibility)

Interesting Links

Russ Cox: Updating the Go Memory Model

Recipes for Reducing Cognitive Load

Speaker: Federico Paolinelli

Federico is a Red Hatter working on Telco Network (MetalLB)

Line of Sight: Happy path indented to the left, weird cases are shifted right. Guard clauses, early returns etc.
Good package names
Something about errors, did not pay attention
Pure functions - a note about side effects but also side “inputs” via environmental variables
Function arguments
- boolean arguments making the functions modal: at least provide names with aliases
- functional arguments (see Functional options for friendly APIs)
“Reading like a newspaper”: order matters, main package file
Asynchronous functions: ideally have a synchronous function that does logic (testable etc) and

Overall, a nice talk. Somewhat basic (many blog posts like this) but well delivered.

Building a CI pipeline with Dagger in Go

Speaker: Márk Sági-Kazár

“Senior YAML engineer” from Cisco. Speaker asked who else is a YAML engineer. I proudly raised my hand. I heard great things about Dagger CI but never saw it in action, so I was curious.

Portable CI pipelines, written in a language of choice, so you have a power of a language to express advanced concepts, instead of learning a YAML DSL (Hello Prow! Hello ci-operator!). Live demo… that failed.

Debugging concurrency programs in Go

Speaker: Andrii Soldatenko

Many books on Go and many books on concurrency but only a few on debugging in concurrent programming.

Interesting Links

What's new in Delve / Tracing Go programs with eBPF

Speaker: Derek Parker

Derek Parker is also a RedHatter.

Delve now has an eBPF tracing backend. eBPF is much faster than ptrace (the “traditional” debugging syscall). eBPF was a “Berkeley Packet Filter,” but it kinda exploded, and now it is a universal tracing cool thing in the kernel (the ability to run sandboxed programs in the kernel).

DNF5: the new era in RPM software management

Speakers: Nicola Sella, Jan Kolárik and Aleš Matěj

I zoned out a bit here and used the opportunity to rest after my lunch sandwich.

Learning From the Big Failures To Improve FOSS Advocacy and Adoption

Speaker: Bradley M. Kuhn

I had half an hour of free time before Reconciliation Pattern, Control Theory, and Cluster API: The Holy Trinity, so I decided to see half of this and then leave. I decided to stay; this was a pretty good “soft talk” about open source past, state, and direction, taken from a firm, but not blind, ideological standpoint.

Five Steps to Make Your Go Code Faster & More Efficient

Speaker: Bartek Plotka

Bartek is an author of Efficent Go: Data-Driven Performance Optimization. He also did a small context giveaway, but I was not lucky enough to leave with a book.

Compactor runs, uses 23GB of memory, OOMs. Rinse and repeat.

Usual answers

Solution 1: “tune configuration”. --use-less-memory --use-less-cpu-please
Solution 2: vertical scale-up and give it MOAR RAM
Solution 3: horizontal scale-up (break into little things and scale them). Nope because complexity.

Usual culprit [source]

Ultimate solution: optimize algorithm and code

Previously we were all into optimization, but eventually called that out (”premature optimization,” right?) and maybe went too far into being lazy

TFBO: Test Fix Benchmark Optimize: Have tests that keep you in an unbroken state. Benchmark, improve performance while passing tests.
Understand current efficiency level: Set hypotheses about what you expect to see in the benchmarks.
Understand your efficiency requirements: Set a target.
Focus on the Hot Path: Do Profiling.
Try optimizing that part and repeat.

Lesson taken: I should learn the Go profiling toolset better. I know it is there, but I do not routinely use it.

It was a well-delivered scientific-method procedure of optimization, really enjoyed it.

Interesting links

benchstat

Headscale: How we are using integration testing to reimplement Tailscale

Speakers: Kristoffer Dalby and Juan Font Alonso

I was not really interested in this thing, but there was nothing that interesting at the time, and I wanted to stay in the Go room. Some funny memes and slides, but I do not know the domain. Otherwise, it was just a simple “we developed without tests for a while, started breaking things, started testing, and now things are better.”

Our Mad Journey of Building a Vector Database in Go

Speaker: Etienne Dilocker

“Optimizing the hell out of serving raw data from disk to somewhere.”

Is Go Object-Oriented? A Case of Public Opinion

Speaker: Ronna Steinberg

A “trial” about the question. I would vote yes before it. I voted yes at the end. Good fun talk.

Perspectives from the Open Source Developer

Speaker: Hilary Carter

This was a closing main track talk for the day, and I was tired, so I did not pay too much attention.

Sunday

I started the day with some Rust and spent the rest of the day in the Monitoring and Observability DevRoom. Most of the talks I attended were about OpenTelemetry and were very Grafana Labs-heavy. I knew Grafana and, to a less extent, Loki, and I had never seen Tempo (distributed tracing) and Phlare (profiling), and Mimir (backend for metrics, more backend-y than Prometheus?).

As usual, I took worse notes than I did on the first day.

Reimplementing the Coreutils in a modern language (Rust)

Speaker: Sylvestre Ledru

Application Monitoring with Grafana and OpenTelemetry

Speaker: Fabian Stäber

grafana/tempo: Grafana Tempo is a high volume, minimal dependency distributed tracing backend.

The OpenTelemetry standard is built on “spans” and defines three types of spans: server, internal and client. Traces in Tempo have labels (like metrics have) and are searchable.

Practical introduction to OpenTelemetry tracing

Speaker: Nicolas Frankel

There are options to “autoinstrument” applications with OpenTelemetry without any code changes: this is trivial for Java and Python (instrumented via runtime) and slightly more work (but not much) in Rust.

Exploring the power of OpenTelemetry on Kubernetes

Speakers: Pavol Loffay and Benedikt Bongartz

Adopting continuous-profiling: Understand how your code utilizes cpu/memory

Speaker: Christian Simon

grafana/phlare: 🔥 horizontally-scalable, highly-available, multi-tenant continuous profiling aggregation system

Loki: Logging, but make it cloud native

Speakers: Kaviraj Kanagaraj and Owen Diehl

grafana/loki: Like Prometheus, but for logs.

Continuous Documentation for Your Code

Speaker: Anastasiia Tymoshchuk

How to start with Continuous Documentation in Python

Closing words

It was nice to return to an in-person conference; virtual events are efficient, but the experience they provide is limited. Of course, there are annoying things like stale air, ambient noise during talks, and room entry crowds. But the freedom to participate in different activities with varying intensity and smoothly transfer between them is unique to in-person events. You run into friends and ex-colleagues. Have a small talk with a stranger while waiting in a queue. Enjoy a beer, coffee, or a piece of junk food. Or hunch over a laptop near an AC socket and do some work.

Returning to a conference after the pandemic felt a bit like attending for the first time: “oh, this is how it works...”. It was refreshing and encouraging. So I am all hyped for more, and I am already thinking about what to submit for DevConf.cz in June.

15 Year Carreer in Hindsight I: First Job!

Petr Muller — Thu, 05 Jan 2023 00:12:34 +0000

Series

I’ve reached the point where I am officially considered old in the IT industry. Over the past 15 years, my career path has not been straightforward. I didn’t follow the typical route of starting as a junior in a specific area, improving, and eventually becoming an expert or architect in that field. Instead, I’ve worked on many different things and started from scratch in various technology niches.

Recently, I’ve found myself talking to people about interesting things I did in the past that are unrelated to what I’m currently doing. In these conversations, I realized two things. First, I don’t remember well what I did 10 years ago. I never consistently kept a journal, so the fun stuff I did in 2008 with GCC is now spotty in my memory. It will be even worse in another five years, and I’d like to avoid that.

Second, I enjoy working with beginners and juniors. As a senior engineer, I am also a member of Junior Guru. This Czech community helps those starting or switching careers in IT. Writing down what I did at different career stages can help me with mentoring. It could also help others, as some of the things I did were clearly beneficial for me, and some were not.

That’s why I’m starting this series. I think about my career in stages. There was always a period when I was focused on something, a job, an area, or a goal. That took some time, and then it typically changed. So I will base this series on these stages, with each article covering one stage. This first article is about how I studied IT, how it went, and how I landed my first job.

How Did I End Up in IT?

Many good programmers start coding very early, captivated by the power to make a computer do what they want. I didn’t have that experience. When I was a child, all I did on our family computer was play computer games. However, in the era of 286/386/486 CPUs and DOS/Windows 95 operating systems, playing computer games often required some computing knowledge. So, I naturally learned things like hardware setup and LAN networking. I even remember translating some games to Czech by hand-editing strings in binary files using a hex editor. I grew up as a “computer kid” at school but never really learned to program. I wasn’t even that good at math!

When I was thirteen, I joined a youth club that taught 3D art and animation using Lightwave 3D. In reality, though, we mostly played Starcraft and HOMAM3. After a year, the club ended, but I still wanted to play Starcraft, so I joined another computer club that taught C programming. Fortunately, I managed to pick up at least some C between all that Starcraft.

Schools

I never actually planned a career in IT or otherwise. I didn’t know what it entailed and wasn’t particularly ambitious. I was smart enough to pass school easily without much effort, and I didn’t care too much about getting good grades. I put in a bit more effort when I was told that grades were essential for getting into a good school. I ended up applying to two excellent high schools in Brno, one that focused on IT and programming (because I liked computer games, right?) and the other more general. I got accepted to both, but I chose the IT school, which turned out to be a great decision.

The programming and IT curriculum at my high school were excellent. I had a slight advantage because I already knew some C, but we did a lot of programming over the four years. They also taught us topics from the computer science curriculum, such as algorithms, formal grammars, database concepts, and graph theory. The education was so good that I could get a bachelor’s degree at the university later with the education I got in high school, without much effort. I knew most of that stuff already.

After finishing high school, I wasn’t sure what my career would look like. But a university degree was apparently a good thing, so I applied for two computer science university programs and was accepted to both. I selected the more technical one (in Brno, where I lived and still live, multiple universities offer IT-related programs).

Testing Anti-Virus Software

In my second year of university, I found myself with a lot of free time because I didn’t have to put much effort into my studies. At the same time, like many students, I needed money for important things like beer and hobbies. I’m not exactly sure how I found the job ad for a part-time tester position, but I decided to apply and soon found out the position was with Grisoft. It may not be a familiar name today, but Grisoft was a well-known Czech anti-virus software vendor at the time. Their product was called AVG, and in Czechia, AVG was synonymous with anti-virus software. Grisoft was later renamed AVG and eventually acquired by Avast, another Czech AV vendor.

Finding out that I could work with a highly reputable company was a breakthrough for me. It’s hard to explain, but I never realized I could work on a big, well-known, prestigious software product or for a big name in the industry. I simply didn’t know that was possible. I wasn’t ambitious and had almost no idea what my career would look like. I just followed some path and focused more on beer and hobbies.

My first interview was a lot of fun. I had no idea what I was doing. I don’t remember it too well, but I answered half the questions with answers like “I don’t know, so I would google it.” I didn’t even feel bad after the interview. I didn’t know what an interview was supposed to be like, so I just winged it, as I did with most things. I showed up, used my brain, and something happened. In hindsight, it was hilarious. But they hired me anyway.

The tester job was as entry-level as they come. We were a cheap team of testers, all IT students. We showed up each day, received printed-out test scenario forms, and manually tested them on the day’s build. If we found a problem, we would file a Bugzilla item. Sometimes, our task was to validate that a reported bug was fixed in a new build.

It sounds menial, and it often was, but I learned a lot. Even in a low-level job, I saw how the software development process works: builds, bug reports, quality engineering, and communication between developers. I also picked up some technical skills, like working with virtual machines and a bit of Python programming. I learned a lot about Linux too. Checking the Linux builds was a task that most of my colleagues didn’t like. Most IT students at the time came from a gaming (therefore, Windows) background, and Linux was still relatively fringe. I didn’t mind and eventually became the “Linux guy” in our testing lab.

Winging School

The first job changed my life a lot. I made some real money for the first time. It was almost a laughable amount by today’s standard, but among students, I felt rich. I also learned a lot very fast. And even better, I learned a lot of things that I could use immediately. This was a big contrast to learning stuff at school, where it was much more abstract. It was often unclear why certain things were important, especially theoretical subjects like math. So I became a bit cocky and stopped investing effort into school beyond the basics necessary to pass the courses. I thought focusing on practical learning in my job was more valuable. My study results got worse.

One thing worth noting is that the job was full-time. After several months of balancing a full-time job, university study, and hobbies, I decided that work could wait. I did not prolong my contract and started to focus on school again. Some damage was done, though - I failed some courses, which meant I would need to repeat them after a year. Because of the university credit system, I knew I would need to prolong my study by one year. I did not mind, though; there were some exciting courses. I also picked up an interest in learning new programming languages. I continued learning Python, and I experimented with C++. I even enjoyed tinkering with some of the more arcane languages required by some courses. The AI class made us work with Lisp and Prolog, and the hardware courses required VHDL. I remember doing something in Smalltalk and Self as well.

Having an even modest IT salary changes your lifestyle as a student, though. After a short while, I ran out of money. I put up a listing on an online job board where I wanted to find a one-shot project for the summer break. I got two offers the next day. The first one was very close to the hardware (programming drivers for some industrial equipment), which I did not enjoy, so I declined. The second offer was from a recruiter hiring for… Red Hat, who was starting an engineering team in Brno. This led to my Red Hat gig. I ended up spending 10 years there, and it defined my career. I will dedicate the second post of this series to that.

Takeaways

It is hard to identify takeaways from how things looked twenty years ago. The world was so much different back then! But if I had a time machine and could send a short message back to my 15 or 18-year-old self, I would focus on two things:

Working in tech opens up a world of possibilities. With talent and hard work, you can achieve anything. It may sound like something from a Paulo Coelho book, but I believe it’s not far from the truth. Dream of space exploration? Give SpaceX a shot. Love games? Send your CV to a game studio. Want to get rich? Join an early startup. It’s like playing football and knowing that if you put in the effort, you can play for Barcelona. In football, this is just a dream for most players. But in tech, the dream is very much real.
Boring university courses don’t have to be a waste of time. Even if they don’t have an obvious application, investing effort into theoretical classes can pay off. Having a solid foundation of knowledge is valuable and applicable. You start to recognize patterns, relationships, concepts, and systems and learn how to think about them. People often say that learning the fundamentals is a waste of time because you never actually use them, and you can always look up what you need. But to do that, you need to understand the domain you’re navigating, think about it, and know what details to look for.

Alright, this is the end for now. I have been working on this post for a while. I’m still not completely satisfied, but I think it’s better to publish it and move on than to keep editing. It’s still a long post; I wonder how many people will actually read it 😀 If you did, I hope it was helpful!

My plan for the second post of this series is to describe my experience of going from junior to senior, leading a team, and then not anymore.

clitest: command line tester

Petr Muller — Tue, 29 Dec 2020 17:49:43 +0000

This post was originally published on 2018-05-02 on my old blog. I do not want to lose the content, so I migrated it here. The content may be more or less obsolete.

It’s like Python’s doctest but for CLIs. Given a text file containing snippets of shell sessions (prompts with commands and their expected outputs), clitest executes the snippets and verifies the actual output matches the described output. It is written in shell language. The project lives on GitHub and is available under MIT license. It does not see much development anymore, but that is fairly understandable given its simplicity. Nevertheless, the author continues to maintain deliverables and documentation. The project has comprehensive
documentation in its README.

aureliojargas / clitest

Command Line Tester

Installation and usage

Installation and usage are trivial: the whole clitest is just a single shell script with no external dependencies. Hence, it is easy to include copies of clitest in repositories. I was curious how is the little tool implemented in shell, and I discovered the code is nice and readable. I could not resist trying to run ShellCheck on it, and it only spat out few style issues: good job!

I tried to use clitest as a testing driver for pyff project examples so that I only need to provide expected output for each example. It worked nice, although the way how clitest executes the commands to test makes it a little sensitive about $CWD, $PATH and the like. Fortunately, these are quite straightforward to sort out. The test files for pyff are standard Python source code files, so I needed to embed the clitest instructions into Python comments (lines starting with the # character). I used the --prefix option for this; it tunes the way how clitest searches for testable snippets in files.

I have encountered a minor pitfall where pyff does not have entirely deterministic output. The output is correct, but the ordering of some elements may be different between runs. clitest fully matches the output against the expected one, so it is harder to use clitest in these cases (I have decided to make pyff deterministic, even if the order does not matter). There is a clitest feature which can alter the matching when using the inline output syntax, but my use case did not match the inline output feature well.

When would I use it?

I like clitest because it is a single file drop-in without dependencies. This makes it easy to drop it into a repository where it can serve as a lightweight integration test driver: all you need are some input files, define how to execute the program under test and specify expected output.

A second nifty use case is mentioned in the clitest documentation: it can be nicely used to test the code snippets in your documentation for correctness. Often, you will write the documentation in markdown or something similar, and it will contain snippets of how to execute the program and what is the expected output. This way, you can feed your documentation straight into clitest which can, without further work, validate your examples are still correct. You may even do this within a CI automatically, so your documentation is always up-to-date.

pyff: Python Diff

Petr Muller — Tue, 29 Dec 2020 17:44:43 +0000

This post was originally published on 2018-04-06 on my old blog. I do not want to lose the content, so I migrated it here. I worked on pyff during my 2018 sabbatical which I ended sooner than expected after being hired by Red Hat. I would still like to revive this work sometimes.

petr-muller / pyff

Python Diff

The idea of syntactic/semantic-aware diff tool was in my head since we neededsomething similar for a project we were working in Red Hat Lab together with VeriFIT research group. We wanted to connect code differences (git commits or PRs) with test results and build a “riskiness classifier”. The rough idea was something like ”whenever people change I/O code in method M of class C, test T tends to break”. We were missing the analyzer that would easily give us, in machine-readable format, what actually changed in the code, besides changed lines that a simple diff tool can give you. We somehow managed to build something ad-hoc for C code differences and continued, but since then I thought the smart diff could be an interesting project.

Comparing abstract syntax trees

I decided to start in a simple way: take two versions of a Python file as in input, and work over their AST to detect differences. There is an AST module in Python standard library that can parse Python code easily but I remembered a talk on Pylint which described Astroid as an improved module with more functionality (build for usage in Pylint). I wanted to use it but failed to find a current documentation link; for some reason I kept discovering www.astroid.org which is dead for some time (I discovered the current documentation later).

So I decided to go with “vanilla” ast module for a while. I discovered the very helpful Green Tree Snakes - the missing Python AST docs documentation for it and from there, the first steps were quite simple. I chose the approach of driving the development by examples: I selected a git commit from a different project, looked at the diff and asked myself “what changed in that code?”, then went to implement the necessary code.

I have started with detecting added and removed imports, classes and high-level methods in the module, followed by detecting simple changes of these entities such as added/removed methods and changed implementations. The entities are currently identified by name, which means renaming is not properly detected (it will be reported as one class/method removed and another added). At the moment, the only supported output is the natural language summary of the changes.

After I had this MVP version of pyff ready I went on to set up some necessary project infrastructure: README, tests and some helper code.

Further steps

I would like to implement a programatical API and a machine readable output format (probably JSON), then follow with implementing further change types detection. I will probably continue with the example-driven approach, but I would like to implement some “smart” detection soon: something like recognizing that the program was not semantically changed (for example, a simple variable rename) and not reporting an implementation change in that case.

Radamsa: A general-purpose fuzzer

Petr Muller — Tue, 29 Dec 2020 17:36:02 +0000

This post was originally published on 2018-01-05 on my old blog. I do not want to lose the content, so I migrated it here. The content may be more or less obsolete.

Radamsa is a general-purpose, black-box oriented mutating fuzzer. It is written in Scheme and is available on its GitHub page under the MIT license. While the project is not entirely abandoned (there are occasional commits on develop branch, but the last commit on the master branch is a PR merge six months ago), there does not seem much development to happen anymore. The project is a side result of the research done by Oulu University Secure Programming Group. The project has simple but straightforward and information documentation in the repository README file.

aoh / radamsa

a general-purpose fuzzer

Basics

Its documentation describes Radamsa as an "extremely black-box fuzzer": it does not need any information about neither the input format nor the internals of the fuzzed program. The tool starts with a given sample input for an application, on which it applies a mutation while trying to keep the general format valid(-ish). The root of Radamsa was a research on the automatic analysis of communication protocols.

Radamsa claims to be applicable, without any configuration, on programs processing any format of input - binary or text. Quick experiments (see below) show that Radamsa is quite successful. Although Radamsa's output cannot probably compare with format-specialized fuzzers (such as CSmith for C programs), the applied mutations go well beyond random garbage injection, leading to valuable testing inputs for a program.

Intuitively, I would not expect many successful bug discoveries in proven, battle-tested software, but Radamsa README file contains an impressive list of discovered CVEs, including curl, libxslt or bzip2.

Installation and usage

The instructions say building Radamsa is a simple clone-and-run-make process and I was able to build a single, dependency-free binary without any problem. Mimicking few examples from the documentation worked as expected: feeding my name to Radamsa's standard input yielded multiple mangled variants which I imagine can cause havoc to naive string processing routines.

The next experiment I tried was running Radamsa with a simple Python program as an input. Again, the results were interesting, the applied modifications varied a lot but kept a general structure of Python code. I saw lines removed or duplicated and tokens changed (for example, integer literals changed to a different value). I also encountered quite interesting, non-trivial mutations like replacing the whole expression in a parentheses with a recursive-ish expression (think something like a(a(a(a(a(b)))))). Again, several tries made me convinced these inputs would be valuable when trying to fuzz something that processes Python grammar.

As a last experiment, I tried an XML file. Specifically, xUnit result XML file. Again, Radamsa changed the file mostly in a way that kept the overall format, but the scale of applied mutations was similar to the Python input.

When would I use it?

Radamsa is extremely simple to start with - you only need the target system, few sample inputs and you are good to go. Set up Radamsa in a loop, feed its output to the system under test and detect bugs. Of course, the black box approach limits the rate with which Radamsa can penetrate deep into the tested system, especially compared to smart fuzzers guided by the instrumented target system, such as American Fuzzy Lop.
You also need to have a reasonable way how to detect error condition in the tested system, given an unknown input (but this holds for most fuzzers). Of course, you can usually start with some simple criteria like "the target should terminate and not crash".

I will certainly include Radamsa in my toolbelt. The fact that I can immediately, without any setup, run it for few hours against pretty much anything makes it useful in different situations, especially when instrumentation or specialized format fuzzers are not available or worth the effort to set up.

Randoop: Automatic unit test generation for Java

Petr Muller — Tue, 29 Dec 2020 16:58:51 +0000

This post was originally published on 2017-12-25 on my old blog. I do not want to lose the content, so I migrated it here. The content may be more or less obsolete.

Randoop is an automatic unit test generator for Java (and .NET). Randoop is written in Java and is available either from its project page or GitHub page. It is available under the MIT license. As of 2017-12-24, the project seems to be quite alive, although most of the commits are authored by a single developer (but the project accepts occasional PRs). Randoop appears to be driven by a research group at the University of Washington, but the overall quality of the project structure, supporting documentation, build system and other project artifacts is excellent.

randoop / randoop

Automatic test generation for Java

Basics

According to its documentation, Randoop generates tests using feedback-directed random test generation. It randomly (but smartly) generates sequences of constructor and method invocation for input classes. These sequences are executed, and the results are used to create assertions. This means the tests can mostly only capture the actual behavior of the tested class (possibly for future regression testing), not reveal many new bugs.

There is an exception to this, though -- Randoop can detect when the class under test does not conform to basic Java contracts (Object.equals() and the like) and several other likely-buggy behaviors, such as NullPointerException being thrown when no null values are passed as params to a method. The documentation states that it is possible to add more contracts for checking.

Installation and usage

I have cloned the Git repository and followed the manual to build Randoop from source using Gradle. The build went for about five minutes and produced a JAR file. I have tried to execute Randoop on a little library I developed when working on the static analysis of C programs, smg.

petr-muller / smg

Symbolic Memory Graph library in Java

I started with generating tests for the simple SMGRegion
class.
After a little fiddling with params, Randoop ran for a while, generating 9 files about 2MB each, with 4286 tests (so about 18MB total, which looks a bit excessive a ~60 lines long class). No "error revealing" tests were generated, just the regression tests. I have tried to execute the tests, and they all passed. Their total runtime was 0.105 seconds, which is good. I tried to introduce a change in the tested class and rerun the tests, and now 2506 tests failed as a result.

Afterwards, I have tried to include all public classes, and the results were about the same -- about ~4200 tests, no error-revealing (but still, Randoop can find just basic Java contract violations).

The generated tests are straightforward (just constructor and method invocation sequences) but quite long, with the usual appearance of generated code (numbered variable names, etc.). I was able to investigate the fails quickly, but of course, the generated code has no real semantic meaning that would hint the programmer about why the bug is there, not other than "this worked before".

When would I use it?

Randoop seems quite useful to me. It is mature enough, well-documented and quite easy to use. I also did not encounter any problems with the tool. Its error-revealing mode could be run as part of CI, being basically a simple fuzzer for Java contracts (but I think existing static analyzers could do the same job).

The generated tests usefulness is slightly more questionable. They could serve as regression tests, as they can only alert you later when you, perhaps mistakenly, change observable behavior. The good thing is Randoop can indeed create tests that you possibly need but did not write. You could generate a testsuite at a particular point in time and keep executing it: this way you would have a nice regression suite, but you would not test any code added after the suite was generated. Regenerating the suite after each change seems too expensive, but has some merit (of course, only if the original suite was run and passed first). Perhaps some discard-few-old, generate-few-new strategy might be employed there (I guess these strategies are probably discussed somewhere in the related scientific papers, such as the authors' Scaling Up Automated Test Generation ASE'2011 paper.

I can also imagine situations where Randoop generates tests that capture "undefined" behavior, like ordering or specific values that may change between execution. The user manual briefly discusses this, and the tool provides few techniques that can be applied to prevent such behavior.