DEV Community: Tim Gilboy

Chatting with Will McGugan: From Side Project To Startup

Tim Gilboy — Fri, 18 Nov 2022 11:55:55 +0000

Will McGugan is among the most well-known Python developers. He's the author of Rich, a library for formatting output in the terminal. It's used, among others, by pip, and has more than 40K stars on GitHub. In 2021, Will started building Textual, a TUI (text user interface) framework based on Rich. At the end of the year, he founded the company Textualize.

In this conversation, we discussed both technical and non-technical parts of this journey. He tells about the transitions between various phases: from side project to a popular open source library to an open core business. He also shares his thoughts on designing APIs, maintainability, and speeding up your Python code.

Building Rich in Public

Reka: Let's go back in time. How did Rich start? As I understand, it wasn't a linear story.

Will: Well, it's kind of a long story. It was in the back of my head for a few years. I was working on another project where I wrote some things to the terminal and hacked together a class. It did various formattings with ANSI escape sequences. Making text bold or magenta, wrap it.

It was a bit messy, but I realized there's potential to extract that to a library. So about three years ago, I started doing that. Based on the original code that I had written several years before, I built something. A bit tidier, a bit more elegant than that original class.

And it came together fairly quickly. There was a first version in just a couple of months of development. And it kinda took off very quickly. I pushed it in the usual channels, and it got a lot of stars.

People seemed to really appreciate it. Various formatting libraries existed that time, but they didn't work very well together. So you could colorize text but couldn't wrap it. You could create a table but couldn't colorize the text inside it.

So I figured I'd create a system where you can build all these things and have them nicely integrated. So that was pretty much the driving force behind Rich.

R: How did you begin creating this library? Did you extract code function by function from this old application?

W: I didn't use any code from that application. I only took the core idea from there. This core idea has two main parts. A Console object, which represents the output channel. And a protocol, which I can add to any object, to define how the output for this specific object looks like. So to print a table, you can construct the table, and then you can call console.print(table).

So everything follows this same pattern. I think that made a lot of sense to people. It just kind of clicked, because it's similar to printing. You don't call string.print(), you call print(string).

This pattern also enabled me to build various kinds of things which could be printed. Tables, panels, rich text, and progress bars. But the API is the same for all of them: You construct the object, and the Console can print it.

R: I definitely agree that the Rich API is very consistent and discoverable. What else did go into this besides the consistent pattern?

W: Rich uses SemVer. And I've always been ready to bump the major version number to improve the API. So, it's in version 12.6.0 now. And each one of those major version bumps was because of a change in the API, just to make it a little bit more elegant.

I think developers are prepared to accept quite a lot to do something they wanna do. They are prepared to accept clumsy, awkward APIs, if it does something that they need.

But I've been determined to make this API elegant. So, I've iterated over it quite a bit.

R: Have you felt any drawbacks of these iterations? Complaints because of breaking changes?

W: Not really, because I was very upfront about it.

Well, actually, early on, there was such a story. I made a breaking change, and I didn't increase the major version number. And then I got complaints. And I was quite surprised that there were so many people using it. And they cared enough to complain about it.

But after that, I've been consistent: I increased the major version for any breaking change. And I documented what changed, and people have been happy with that.

Basically, what developers don't want is surprises. They don't want things to just stop working suddenly. Which I can understand. But you can document all these changes, show where it's broken. And maybe explain why you changed it. So, people are happy that they can upgrade at their convenience. And also, they can be comfortable because they know it won't just break at some point in the future, seemingly randomly.

R: Switching gears, how did you manage the logistics of a full-time job and a demanding side project?

W: I was a Python contractor and working from home, which allowed me a little bit of flexibility and spare time. So I did my day job and then maybe an hour for Rich at lunch and maybe a little bit in the evening before dinner.

And it keeps interesting. Because you got a day job, which pays the bills. But I often find my hobby projects more interesting. So, I found the time for it.

I mean, I wasn't that consistent. There were gaps. Sometimes, there was a month where I didn't do anything, because I thought I'd finished it. And then I got a number of people asking for a particular feature. And I thought that's a good idea. Then I spent the next few weeks kinda working quite heavily on it.

So, yeah, I just kind of fit it into the slack time. Where I'd done the day job and didn't have anything else to do.

R: You said that Rich took off quite quickly. How quick was that?

W: It was released two and a half years ago. Now we're above 40K stars. If you look at the star history, it's been almost a straight line. There have been a few bumps, mainly due to social media. But generally, the growth returns to the same level.

Will's Twitter profile

R: Rich was also your first project that you've been building in public. How do you like that experience?

W: That worked really well, partly because Rich is quite visual.

When I got something working, I was pleased and posted it on Twitter with a few comments. I did this just for my own amusement, really. But people started following along and I built up a bit of a following. And I'm sure that contributed to the popularity of Rich. Because people would share it.

Rather than waiting, say, a year until the API was very solid before I announced it, I announced it early on. And just kept releasing new content, whenever I had a new feature or fixed a bug. Or sometimes even when there was a fail, when something went wrong, I would post about that.

So people could follow along with the development. And it was beneficial to me. This helped my motivation. Knowing that there were people out there, who were genuinely interested in what we're doing.

Because development can be a very solitary activity. You just tend to be on your own, hacking away. It's nice to know that there are other people that care about the thing that you're working on.

R: What kind of reaction did you get? How much did the feedback influence it?

W: A fair bit. So in the beginning, I didn't have many followers on Twitter. So, it was just me posting something. But occasionally, I'd get a star or a like or a comment. But as it went on, I got more and more comments, more and more feedback.

And that did influence the direction that Rich went. People would ask for features. At several points, I thought, oh, it's finished now. But people would ask for something and I realized that the library would be better, if I implemented that. So as time went on, there was more and more feedback from people. And a lot of that feedback went into the library. So it did very much influence the direction that it took.

And it's quite a positive thing. If I was just building for myself, I might just build a very lean core set of features that I needed. But then its use would be limited to just myself or to people who are doing exactly what I'm doing. But by listening to feedback, I built something which was broadly useful.

Now, Rich has become kind of standard for doing something more sophisticated at terminals. And it's because of listening to all that feedback I've got for the last two years.

R: Do you have some specific examples in mind? Features that you didn't plan to add, but the community asked for?

W: Yeah, quite a lot of it. Progress bars are a good example. I had imagined that Rich would just purely be writing static content, nothing animated. But progress bars were requested quite a few times, and I looked into that.

I figured out how to do the mechanics, and I enjoyed the technical challenge. So I built the whole progress bar thing, even though I had no use for it. And now, it's one of the most loved features.

R: I definitely like the progress bar, as well, so thank you. 😀 Regarding the feedback, have you developed some strategies how to evaluate it? When to accept it and when to ignore it?

W: One individual feedback on its own probably wouldn't make that much an impression with me. When multiple people give the same feedback or request the same feature, that's a sign where I should focus my efforts on.

It was almost like polling the community. The most asked for feature is the feature that I would implement next.

It's quite a strange thing because when you start project like this, it's purely for your own amusement. And you're the boss, it's not like a day job. No one tells you how to write code. You write whatever you want and do it however you want. And when it gets more popular, something shifts. You start working for yourself and you start working for other people.

So rather than being your own boss, you've now got ten thousand bosses. But that makes it more useful for more people, and so it's beneficial.

R: Do I recall correctly that Textual also came from the input from the community?

W: Yeah. That's right. Some friends on Twitter took Rich and built a dashboard, which shows live events happening on GitHub. Pull requests and various things. They created a nice table and it had emoji icons. It was kind of cool. They were using Rich under the hood, but they've added the interactive stuff themselves.

And then I thought such apps could be really useful, but there wasn't a nice way of creating them. So that's when the idea of Textual came. Could I take Rich, add interactivity on top of it, and create an application framework?

And I started doing that again. Just as a hobby project. I had no thoughts of turning it into any kind of business. But after a few months, it kinda came together.

I had a strong tech demo. And I kinda convinced myself that, yeah, this is gonna work. That this could be the, like, the beginning of something, which could be very successful. Fingers crossed.

Photo by Textualize

From Side Project to a Company

R: When did this happen? When did you decide to focus on Textual?

W: Towards the end of 2021. I figured that I could take this application framework and build a web service around it. Which I could then sell as a service.

It was only the very beginning of an idea but I was prepared to take a year off. I was going to live on savings and implement this. And at the end of that year, I'd either have a business or I could get funding for it. In the worst case scenario, I would just have to go back to the day job.

That didn't last very long, because the opportunity for funding came along. And I took that, which is probably smarter than burning through a year of savings. So at the beginning of this year, we founded Textualize. Since then, we've been working on Textual and Rich full time.

R: What motivated your decision to create a separate project? Instead of adding all this new functionality to Rich, which was already a well-known brand that time.

W: There's quite a nice division of labor between the two libraries. Rich is more for static content, for writing stuff to the terminal. It has a few dynamic things, like the progress bars. But I figured it was better for Rich to just focus on static content, as it's reason for being.

Because it's already quite a large library. And if I added too much to it, it runs the risk of being too big. People would just look at it and think: Well, it does way more than I could possibly need. Plus, it would have a lot of other dependencies.

So, I figured that I would then separate it into two libraries. I would build something new for the dynamic, interactive stuff and that would have Rich as a dependency.

So, if you don't need a full application framework, you won't need to pull everything in. You can just use the bits that you need.

I think that was a good move, because it means that there's a nice division of labor. There's Textual for the dynamic stuff and Rich for the rendering.

R: You mentioned in an interview that Textual's first prototype was a bit messy and you cleaned it up afterwards. Can you tell more about that?

W: Sure. When I started Textual, it was kind of just exploring what could be done. So first, I just built a prototype using asyncio and Rich and see if it would work, first of all.

And a lot of that was built over a number of weekends where I was drinking lots of coffee. And it came together. But the code was not as tidy as I'd want it to be.

So when I got the funding, I could just purely focus on that. I could apply all the best practices and try to make the code as neat and tidy as possible and address some technical debt.

I could also be more ambitious. Because if it was like a hobby project, working on the evenings and weekends, I'd have to be quite pragmatic. Because there's only so much a single developer can do.

But once I got the funding, we can do a lot more because I've got employees to work on it with me. Very smart people and we can do some work in parallel.

So, we took the Textual source. We didn't restart. There was lots of good stuff in there. We just took the original tech demo, built on that, and used that as a platform for the features. And now, we've tidied it up. Most of the code, I'm quite proud of: It's quite neat and tidy.

And I encourage people to look inside the code. And now, it's a much better platform going forward. So we're gonna make lots of releases quite rapidly in the future.

R: Which release cadence do you strive for?

W: We released 0.2.0 recently, which had the CSS support. That was on a feature branch for well over six months. This is not how I like to work.

I like having rapid releases. Now, we're gonna aim for something like a week to two for each new release.

So that if you're using it, you can keep following the latest fashion. That means you can identify if there's any problem early on.

R: Why was CSS support a special case?

W: There was very little to build on. No library to do CSS that I could just import. First, I had to write a CSS parser. Then I had to write a CSS renderer. Then we had to integrate CSS all the way through the project. These all took a while.

Here and there, I ported back some fixes to the main branch. So that I wouldn't leave early adopters completely in the dark.

But once the CSS stuff was done, now we can just keep iterating. Which is a nicer approach for everyone involved.

R: How was the transition from a side project to a company?

W: It took a lot of adapting. I'd been sort of working freelance, from home for the previous twelve years. Now, I'm working in an office. And that was quite hard to adapt to: having to get up and go into work every day. Given I hadn't done that for ten plus years.

The other difficulty was giving up control. I suffered from wanting to do everything myself. Even though I've got great, talented guys working for me. So I had to learn to delegate and allow other people to work on it. Because up to that point, it was just my project. But now, I'm just a member of the team.

So that was quite hard to adapt to. But other than that, it's not been too bad. It's like I can have a bit more control over my life. I don't have someone telling me exactly where to work. I can make those kind of higher level decisions, which I prefer. So my day-to-day life hasn't changed all that much, really.

R: What helped you with giving up control?

W: I think it's just acknowledging that when you're not the only person working on it, things will go faster and smoother. And also getting feedback from other people is always beneficial.

We'll probably be distributed going forward. But for now, we've got a small office in Edinburgh, and it's been quite useful to bounce ideas backwards and forwards. So when you get used to that, you realize that ultimately, it's gonna be better for the project.

Textual is gonna be like the work of many talented people rather than just one individual. So I think it's acknowledging that it's better for the project.

R: How did you choose your team? Which roles to hire for and who to hire?

W: My first employee's just sitting over there. I knew of him via GitHub. He's got several other projects, one of which was a testing framework. So, we sort of corresponded on GitHub a few times, and I was impressed by his work. And he just, by coincidence, happened to be in Edinburgh. So I just reached out. And I convinced him to join me. So that is employee number one.

At the moment, we have three developers including myself. I'll have a fourth by the end of the year. Actually just in a couple of weeks.Next year, we're probably hiring again, probably in the middle of the year.

Last but not least, I've got a PA and bookkeeper and she's also my wife. Very overqualified for what she's doing currently, but she's a member of the team.

R: What came into the office vs remote decision?

W: It's hard to say. Initially, I thought we would be just distributed. But when I talked to Darren, he was asking about an office, and I started thinking about an office.

And there are some benefits. I think it feels more like work. If you have to get up and physically go to a new location, it kinda puts your head in a different place.

And it has been beneficial in that if you got people around you where you don't have to arrange Zoom calls. You can just lean over and talk to them. I think that has been very beneficial in bouncing ideas backwards and forwards. And creating quick iterations.

I think working remotely can work very well. And as we will have more and more employees in the future, we'll get distributed. But I would like to keep a core team in the same office, because I think that's a very good way of just gathering feedback very quickly.

R: How was the technical part of this transition from side project to a company?

W: I always tried to write code, even if it's just for myself, that other people could pick up and read. But when you are working with other people, that has to be like first and foremost in your mind.

You have to add docstrings and you have to name things very carefully. You have to make it, so that someone else could contribute. I find, a good rule of thumb is: If you can explain it to someone easily, then it's good. If you struggle to explain something or if you feel the need to justify some code you've written, then that probably means that it's not such a good code.

With good code, there should be very little that you have to defend and justify. It should be obvious that this is a nice algorithm, a nice way of doing it.

R: Were there surprises? Stuff that you thought was obvious but was difficult to grasp for your new teammates?

W: Yes, there were. Sometimes, you can lose sight of what's obvious and what's not. Because everything you write builds on something you've written in the past.

So, for you, it might feel obvious because you've got this last two months of work in your mind.

I think, what you have to keep in mind is that when someone comes to the project completely fresh, they don't necessarily have that background. In their mind, they haven't got that mental model.

That's why you have to structure your code in a way that makes it independent of previous work. So that you don't have to, again, justify the code that you've written based on the last three months work. It should stand on its own.

I think, keeping that in mind is important when you're working with other people.

Technical Lessons

R: You also wrote a blog post about 7 things you've learned building Textual. One of them was: "Immutable is the best". Has this been your preference longer or was it a lesson from Textual?

W: I've been thinking that way for a while. But working on Rich and Textual made it clear to me that immutable objects in Python really simplify a lot of things.

One of the problems with Python is, it's a bit too slow. It's reasonable to acknowledge that.

But if you cache things, you can make it a lot quicker. And things are easier to cache, if they're immutable. If objects can't change, you can just stick them in a cache. So, that's one benefit of immutability.

Another benefit is testability. If you have pure functions, which take immutable objects and return an immutable object, it makes testing much easier. Immutability also makes code easier to reason about.

So, I can understand why in some languages everything's immutable by default. In Python, nothing's truly immutable. I think, it's still beneficial to work with immutable type objects. These are, for all intents and purposes, immutable. So, I try to do that whenever I can.

Usually, the exceptions are large objects. These would be more expensive to copy. Bigger objects also tend to have more state and change it more frequently.

Also, there are things which can't be conceptually immutable: input, output or files, etc. Those can't be immutable, because the actual data can change underneath.

For me, immutable is the default idea. I only make objects mutable if I absolutely need to.

R: You've mentioned Python's slowness. Which brings up the question: Why did you choose Python?

W: I chose Python, because it's the language I prefer to work with. I love the elegance of it and the fact that it's very readable, and you can iterate very quickly.

But I've always known it was slow. It's not a compiled language. It doesn't generate machine code. At the machine code level, assigning an attribute is almost instant. It's like less than one cycle. But in Python, you gotta look up a dictionary and execute a bunch of code. I always knew it was slower, but generally it's not a problem.

When it came to Textual, there was a number of optimization issues.It got faster and faster. And most of those improvements were kind of algorithmic. But even once you got the algorithm going as fast as you can, there were a number of micro-optimizations. I think, most code can be halved in the time taken through micro-optimizations. So, in general, the code got faster as I worked on it.

And also, each Python version gets faster. I mean, the newest version 3.11 is, on average, 30% faster. And they're aiming for five times faster.

That's encouraging. I think, in the future, Python won't be seen as a slow language. Similar to JavaScript. If JavaScript can be fast, Python can be fast.

So, I think, in the in the future, speed will be much less of an issue. At the moment, it takes some work. Especially for things which are doing a lot of number crunching, a lot of processing. You might have to really focus on that code to make it fast enough. But, I think, in the future, it might just go fast enough by default. Which I'm really encouraged by.

R: Did I understand that correctly? The execution time can be halved by micro-optimizations for a lot of code?

W: Yeah. I think, that's an average.

If you take any loop, that's more than a couple of lines, you can apply a bunch of micro-optimizations. Just reduce number of operations that have been executed. Doing things like hoisting attribute access outside the loops. I think that most code can benefit from that, and it can go, like, twice as fast.

That's just my estimate. I haven't done any grand tests or anything. It's not an academic thing. But I found that almost all code can be sped up by two times.

R: How did you approach these optimizations? Did you start with profiling?

W: The general advice is that you should profile the code before you optimize. That's not bad advice. But I've found that most of the time, the slow code is really obvious.

Inner loops, loops doing the most iterations and doing the most work are the usual suspects. And it tends to be a piece of code that does more than one thing. It tends to be doing a lot of things. So, I think, often, the critical paths that slow your code down are really easy to identify.

If you understand the code, you have an idea where things can be slow. But you do get to a point eventually where you're not sure. Then you do have to profile and it tells you where to focus your efforts.

But, I think, if you know your code well, it's generally fairly obvious. At least from the outset, which bits are quite slow and which bits are fast.

R: Can you mention an example of a speed-up?

W: Yeah. So when you render things in Textual, it starts off with a high level representation of byte code segments. Which is a piece of text, plus a style. And then Rich will transform that into ANSI escape sequences.

And I could tell that was doing a lot of work: transforming this Style object and the ANSI escape sequences. And this will be done for, like, every piece of text. So, probably every word, we'd have to do this kind of work. So, that was a point where it was kinda clear that was slow, and I spent a lot of time optimizing that.

And one of the ways I did that was make the Style object immutable. I love immutable things, and then I could cache that. So, when you're generating ANSI escape sequences, you don't have to calculate the Style objects again. You can just look them up in a dictionary. And that made it a lot faster.

So that's kind of cool. I think, the new version is twenty times faster than the original code.

Terminal as Platform

R: How is Textual used currently?

W: We have a gallery, showcasing Textual applications. They seem to be more developer focused at the moment. People also like it for cryptocurrency, because they can display tables of numbers that update very quickly.

There's a few productivity tools, but I think it'll be interesting going forward to see further usages. Including apps intended for non-technical users.

The CSS branch was merged just two weeks ago. So I'm really excited to see what will happen in the near future.

R: Which further usages do you expect?

W: There's quite a broad number of use cases. In essence, it's any application that fits in that kind of terminal form factor.

But I think initially, it'll be for internal tools. Managing configuration, managing servers, editing files, browsing files, that kind of thing. Things that work with your local file system or network.

And because it's terminal based, it means you can go from the command line, go into a Textual app, and then go back to the command line. And it's kind of like seamless.

It also means much less distraction than opening a web browser. The terminal can keep you in this frame of mind where you're working and interacting with your computer and doing productivity tasks.

So, I think initially internal tools or things like network automation. But after that, I would like to break out into almost anything. Because if you're a developer or someone quite technical, you are home in the terminal. Most developers open a browser and a terminal at the beginning of the day, and they'll have them open all day.

So I think people would prefer terminal apps to web apps if they were feature complete. Because it just integrates better with our workflow.

Ultimately, there are very few restrictions. Terminals can't do videos or sophisticated graphics. But for everything else, I think, there's a fairly large set of people who would actually prefer content to be in a terminal.

Which is quite an odd thing because I'm running this in a MacBook with M1. But my preferred way of interacting with the computer is with a grid of text in a in a little box. And I'm not alone, so I think, there's probably a fairly sizable market for it.

R: How does the open core business model of textualize.io look like? What will be the paid product?

W: The paid product will be a web service. You can take a Textual app and turn it into a web application running in a browser.

That means you can distribute it to other people more easily. One downside of terminal apps is that they are generally used only by technical people. Which is quite strange to me, because terminals are installed on virtually every desktop, which has ever been shipped.

But they're only used by a relatively small number of people. So, I think, to make Textual truly useful, you have to be able to build apps for everyone, not just for developers. So that's where this web service comes in. If you build an app just for yourself or a small group, maybe it's just in a terminal.

But if you want other people to use it, you can use our service. Create an account, in the command line, run textual serve providing a Python file, and you get a URL. You can host it anywhere on a virtual private server, and you can run it in the browser.

Roadmap

R: How does the roadmap look like? What are the short and middle term plans?

W: Short term is we're building more widgets for Textual. At the moment, it's got quite a small set of widgets.

We've got a checkbox, a button, scrollable panels. But we want more sophisticated widgets. We want a better tree control, markdown viewer, and also the list control, that kind of thing.

We're trying to build a nice big library of built-in widgets. So that if you're building an application, it's generally got what you need.

In the midterm, we're kinda moving away from development to developer relations. So, we want to evangelize Textual by building tutorials and videos. We can distribute them via YouTube and even TikTok and go to conferences and things.

R: So, you're currently focused on widgets. Is it an area where you also expect contributions from the community?

W: Definitely. We would love people to start building widgets. So, you're just going to PyPI, you'd search for "textual-", and then you get like a number of third party libraries.

I think that's very important because, ultimately, we can only build a subset. We can't think of or implement everything. So, we want to allow people to build and distribute their own widgets. That's why we're documenting that quite thoroughly.

R: It seems that you've generally put quite a lot of emphasis on documentation.

W: Yeah, we've put a lot of effort into the documentation. That's one of the reasons why we delayed the CSS branch. This feature needs a thorough documentation.

Just giving someone the code and saying you can use CSS wouldn't be very useful. It needs a bit of introduction and explanation. Once folks have seen the reference documentation, it's quite straightforward. But without documentation, it wouldn't be useful.

So, we've spent a lot time on the documentation website, and I think it's quite nice. We've got nice diagrams and screenshots. We've got reference pages and API level documentation. So we're quite pleased with it.

R: Yes, we encourage people to check out the docs and build something with Textual. Thanks a lot for the conversation.

Resources

Textual

Rich

Further

7 things I've learned building a modern TUI framework blog post
7 lessons from building a modern TUI framework interview with Will at the Talk Python To Me podcast
The terminal as a platform interview with Will at The Changelog podcast

Chatting with Sebastian Witowski - Part I: Code Standards, Tooling, and Working in Teams

Tim Gilboy — Mon, 31 Oct 2022 10:19:09 +0000

_Sebastian Witowski is a freelancing Python developer and consultant. He has given over 20 talks at various PyCons and also blogs regularly. In this conversation, we focused on cooperation in developer teams: agreeing on common standards and respecting individual preferences, setting up a well-balanced CI. In the 2nd part (coming next week), we'll talk about his activities in the Python community and his content about developer tools.
_

Reka: We can find lots of talks and blog posts from you online. But what do you do in your day-to-day job?

Sebastian: I'm working as a Python consultant / freelancer, mainly in web development. Usually, I join projects for a long time and work in small, remote, and distributed teams. In my current team, I hold the role of a lead developer, so I'm also shaping the whole project. Occasionally, I also do consulting for companies. It can be about migrating older codebases (e.g. in R) to Python or setting up CI/CD pipelines.

R: That's very versatile. How did you choose this career?

S: After university, I worked at CERN for six years, and that was pretty exciting. Then I decided to move back to Poland with my wife and to work remotely. That was 2018. Working remotely was not that common yet, but I gave it a try anyway.

I enjoy freelancing, working with different products and projects. This way, I also have the chance to work with companies located wherever, although I'm based in Poland in a small town.

R: How do you like remote working?

S: It's definitely different from working for a company that is located in a specific town. When most of the employees are from the same town, they've probably worked for similar companies.

But when you work with someone from Norway, someone from Romania, someone from Spain, everyone has completely different backgrounds. They all have different ways of developing and managing a product.

It takes more effort to get everyone on the same page. Even about things like how we write comments, which types of docstrings we use.

But also everyone has something to bring to the table. Each time I start working on a new project, I learn a lot from other people, which is a really, really nice thing.

R: You mentioned getting everybody on the same page. Can you elaborate on that? What are the most important things the team should agree about?

S: There are those no-brainer things like using Black for formatting. Years ago, I worked on a project where code reviews often contained comments about single vs double quotes. And we didn't have tools that would pick this up. So reviewers manually put in these comments, and someone had to change their pull request to change the style of quotes. You shouldn't waste time doing that.

The baseline is to start using the tools that are industry standard in Python. Like Black for formatting, flake8 for picking up the common problems, isort for sorting your imports.

When I join a new project, I usually try to figure out: Are they using some tools? Can we improve the code review workflow by introducing some pre-commit hooks or some CI pipelines?

R: Can you tell more about the pre-commit and CI setup?

S: CI is great, even if it can take a while to set it up. I sometimes struggle to set it up in GitLab from scratch, but I can often reuse some template with the usual suspects like Black and flake8.

pre-commit is great as well, and contrary to the CI, it doesn't need to be uniform accross the team. I use several static analyis tools in my editor or pre-commit. Mostly because I don't want to impose those more specific tools on other people.

There are things like code formatting you should all agree about. But then when it comes to other tools, people might have their own opinions. The more senior people are, the more fierce discussions about tools can get ;)

And this is always tricky because I don't want to get into endless discussions. They have been using one tool for five years, I've been using a different tool for five years and we would just waste each other's time. So I prefer to use tools for myself and try to nudge people towards trying them out. For example, during a code review, saying something like I saw this and this. And by the way, I'm using this cool tool that helps spot this.

Sourcery is another no-brainer for me. I have ten years of experience writing Python, and Sourcery still finds simple refactorings that I missed.

R: Where do you draw the line? What should a team agree about, and where is more room for individual opinions?

S: I'm happy with people having different opinions and even following different coding styles. I draw the line at PEP 8 and PEP 257. That's what we should implement straight away.

Beyond that, it's mostly about reaching an agreement with the team. So sometimes we review our CI/CD pipelines and we say, okay, there is this another tool that we could use. And if everyone agrees, or if the majority agrees, we just add it.

Of course, you shouldn't overdo it. At some point, the CI can become noisy, especially if all these checks can also block a merge request.

For example if you have a problem in production and you need to push a hotfix. If a stupid linter tells you, look, you have to move those imports while you have clients who cannot use your product, then you're absolutely infuriated. And you're going to get rid of all the stuff.

So you also have to figure out that balance where all those tools are useful, but they don't get in your way. Of course, you can disable those things and merge the merge request anyway, but the tools should help you.

R: Speaking of overdoing. In the intro of your talk Python Versions and Dependencies Made Easy, you mentioned that you've met both kinds of teams: some who are still in the 90s and some who are overengineered. How would you describe these two extremes? How can we notice if our team falls into one of these categories?

S: A sign of too few tools is lots of discussions during the code review. Especially about issues that could be fixed automatically.

At the other end of the spectrum, we have the teams who jump on every new tool they find. When you have a new tool, it’s usually an evolution of some old tools. It has cool new features and it solves new problems that the old tool doesn't. At the same time, not many of those tools survived the test of time. Let's pick ten tools from the most upvoted HackerNews posts in the past few weeks. I bet in ten years, maybe two of them will still be around.

Also, there will be some bugs and corner cases and some things that the tool won't do for you. And if it's a new tool, you don't have people who have experience with that so we can't really compare these to some other tools.

So what I learned some time ago is to strike a balance. And the talk that you mentioned shows three tools that I use almost on all my projects: pyenv, virtualenvwrapper, and pipx. They have been working for years. They will still work for years in the future. I've also been using pip-tools since I don't remember when and I think it will still be supported in the future.

These are also tools that do one thing. You could instead choose a tool that does a lot of things like Poetry, which is, of course, a pleasure to work with. But I prefer this Linux-style philosophy where you pick up a tool that does one thing, but does it very well.

Photo by Larissa Oliveira on EuroPython 2022 official photos

R: What are the advantages of this kind of focused or scoped tools? Why do you prefer them?

S: Because you can easily mix and match them. They tend to produce text-based output that is easy to combine with other tools. For example, pip-tools creates a list of dependencies that can be just read by pip. This compatibility is very handy: if something breaks or I need more advanced features, I can introduce a new tool without changing the whole workflow.

R: What are your strategies to transfer some lessons & best practices from one project to another one?

S: I like having some kind of regular technical discussion. It's good to give developers a chance to discuss things that they find interesting. Or discuss some pain points that they are facing. Some companies do a round of lightning talks within the company. That works really well.

Currently, my team uses a custom commercial framework. And when we struggle with something, it often turns out that another team member ran into the same issue two days ago. So it's nice to have a periodic meeting where people can share their knowledge, ask about things and so on.

Screen sharing and pairing sessions also work really well. You can notice if others use a new tool, recommend some setups and so on. The more people interact with each other, the better it is for spreading knowledge.

We can also always document things. But for me, documentation alone never worked that well for knowledge sharing. It's much better to have a discussion or a demo and regard the documentation as complimentary.

Documentation is helpful for answering specific questions, but not for discovering new stuff. If you don't know something is possible, you will never think about searching for it.

R: How do you share your knowledge when you have the role of a consultant?

S: You really need to understand what problems the company is facing. Usually, the people who reach out to you are higher in the hierarchy. And sometimes they say “we want you to fix this”. And when you start talking to them, it turns out that the underlying problem is completely different. So you need to spend some time discovering the real problems.

The way of communicating possible solutions also matters a lot. You can't force some strangers to do things your way. Or tell them that we're just scrapping your whole workflow. It's important to convince people with benefits and try to address the pain points they're facing.

I can tell them “this is how some other companies do this”. Or “this is what you can do to make it more efficient” and so on.

R: Do you have some strategies for this?

S: It definitely helps to prepare very well on the first meeting. And to talk to multiple people from the organization.

You have to take good notes and often get back to them to verify that you understood everything correctly. Especially before you start creating a roadmap. If you've misunderstood their problems and their expectations, you're basically wasting time and the client is not gonna be happy. So you have to be very thorough with your communication.

R: What are usually the situations when a company hires you for such an ad-hoc consulting?

S: A frequent case is when companies want to migrate an older project (for example in R) to Python. And I help them figure out which packages or tools they need to reproduce the same stuff.

Sometimes, companies struggling with tooling reach out. Or they have spaghetti code. And they would like to figure out how they can reorganize their project and how some tools can help get all their developers on the same side.

R: I'm surprised by the spaghetti code topic. I would assume that several companies have this issue, but only a small part recognizes it.

S: That's a good point. Like, companies don't recognize this often as a problem. I think I've had only one client that asked explicitly for that.

Most of the time, it just comes up as part of a different request. Usually, companies don't want to spend money to tackle a technical debt. They don't care how their code looks like as long as it brings the money. Only when there are too many bugs to fix and it takes too long to implement new features. That's when they think maybe we have to invest in someone to help us with this.

R: What are the first steps? How will you start paying back this tech debt?

S: It's always nice to have some tests. Usually, companies that have messy code, don't have tests. That's the main problem. If you have enough tests, you can rewrite parts of the code and make sure everything still works.

So, the first steps would be to add some happy path tests to the old codebase. Then you will have at least some level of certainty that after you do the migration, it still works.

Then you have to sit down and talk with developers and try to figure out: Have I understood what this project does? As a consultant, it's an additional difficulty that you don't know the project well. It's a bit easier if you're in a team and you internally decide that you want to reduce your technical debt.

For example, in the project that I'm currently working on, we accrued some technical debt, and we obviously didn't have dedicated time to tackle this debt at the beginning. So the idea was to use the "boy scout" method that whenever you're working on a new feature, try to refactor a bit of this code. Maybe extract some things. Maybe create a small new module. This is a really nice approach if you can't devote a lot of time to refactoring. And suddenly, after like a year of all the developers doing this, you get much less technical debt.

This approach is much easier to sell to the business users. Because if you tell them, look, I need a week to rewrite this, they will hear “I’m gonna waste a week doing something that won’t result in any new features”.

So instead, you can say “I need two days to implement this feature because I need to write a test and I need to make sure everything works”. Then you spend one day writing the feature, a second day adding some tests.

R: Do you also recommend this step by step approach in your consulting?

S: As a consultant, it's more about creating a roadmap. A big difference is that I won't be there to write the code myself. So it's rather about figuring out what's not working now, what's the main problem, how their ideal solution would be. Or if they could solve their one biggest problem, what would that problem be?

Maybe you show them what could be a reasonable way to structure things: create some test folders, use pytest, conftest.py for the fixtures and so on.

It's often about showing them some possible tools that could make their life easier. They might not like some of them, but they might discover something very useful as well.

R: What other technologies do you use besides Python?

S: Most of my client work is in Python. In a project two years ago, I used a bit of Vue.js, and that was fun. Right now, everything is Python, because we have this framework that also generates the React frontend from Python code.

When I have time for some side projects or hobby, I try to use different stuff. For example, I've built my personal website with 11ty JavaScript is here to stay, so even though I'm using it from time to time and I always have to remind myself how to write a for loop, I still try to keep my knowledge up-to-date.

I want to learn Rust. But I've wanted to learn Rust since, like, I don't know, over a year. I looked at some tutorials, and it seems to be really nice. Also, the tooling is great. That's a huge advantage of this language.

I didn't study computer science. When I was studying automatics and robotic control, I had C in the first semester and then two semesters of C++. And it's great to know C and C++, because some principles stay with you, but that's not how you get people interested in programming. People don’t get hooked up on programming because you show them pointers.

My first semesters with C and C++ felt quite like a waste of time. Then in the last semester, I forced myself to learn programming, because I thought that could be useful. So I learned just enough C++ to do the basic stuff.

But then later, I started building websites, and this I enjoyed really much. With a website, you change a bit of CSS and immediately, your website changes colors. You don't have to go through, like, fifty compilation errors to see the results. And that I found fascinating.

This is the first part of our two-part interview with Sebastian Witowski. Stay tuned for part two, which we’ll be posting shortly.

Sebastian Online

Resources

Sebastian's talk Writing Faster Python 3 at EuroPython 2022
Sebastian's talk Python Versions and Dependencies Made Easy at EuroPython 2021

Tools Mentioned

Dissecting the Google Style Guide

Tim Gilboy — Fri, 15 Jul 2022 08:50:46 +0000

Every developer has their own unique style and approach to writing code. Our styles come from how we’ve learned to code, our previous experiences, and a bit of our own personalities. But, we also need to consistently work as in teams and often in large and complex organizations. This means dozens, or hundreds, or thousands of unique coding styles working together and potentially clashing in a single codebase.

If your team is all working with different approaches, different styles, and different preferences this can start to create problems. In order to operate effectively as a team, you need to be able to unify these divergent approaches so that all of the code you’re dealing with is relatively easy to understand, easy to extend, less prone to bugs, and more maintainable.

Most teams already have systems in place to help standardize their code - typically informal rules and standards emerging during code reviews within smaller teams. But, as teams and complexity grow, these informal best practices aren't enough and you need a more explicit approach. At Google, they’ve explicitly laid out their approach to creating consistent Python code for their open source projects in their Python Style Guide.

Let’s take a look at what the Google Style Guide is, how they’ve structured their rules and guidelines, and what different engineering teams can learn from it. And then we’ll quickly take a look at how you can build your own style guide and best practices into code with Sourcery.

Before we dive too much into the Style Guide itself, I want to call out a quick difference between what counts as Style and what counts as Formatting. There is some overlap between them, but as a general rule of thumb, Formatting is specifically focused to the aesthetic aspects of code - ranging from line length, to spacing, to use of quotes and brackets, etc. A Style guide may include some of these elements (eg in the Google Style Guide, large pieces of section 3 are focused on Formatting elements), but is primarily focused on the more structural aspects of your code as we’ll discuss below.

For handling Formatting issues there are several great Python Formatting tools including:

AutoPEP8
Black
Prettier
YAPF

Google’s Style Guide is made up of more than 40 sets of “do’s and don’ts” for use within Python and is broken into two key sections - Python Language Rules & Python Style Rules. Broadly speaking the Language Rules are used to define how Google approaches using (and importantly not using) and structuring different elements of the Python language within their code. This is a critically important aspect to a Style Guide for teams that have different levels of experience in a given language, because it helps provide guide rails around the code structure and on what type of advanced language feature your team should and shouldn’t use.

For example - section 2.19 says you should not use any Python Power Features while 2.10 says you can sparingly use Lambda Functions (provided that they’re relatively short).

The Style Rules section is more focused on formatting and stylistic practices that the code should follow - ranging from section 3.2 focused on line length to section 3.16 focused on naming practices.

Across both sections of the style guide there are three main reasons or classes of recommendations:

Recommendations to Avoid Danger - Focused on cutting down on severe bugs, extreme forms of technical debt, or regulatory/security risks.
Recommendations to Enforce Best Practices - Focused on making sure everyone is using the same best practices to ensure maintainability.
Recommendations to Ensure Consistency - Focused on making sure the code is readable, extendable, and structured similarly.

Rules vs Guidelines

Like most style guides, the Google Style Guide creates two types of recommendations for approach when coding - rules for how to structure your code and guidelines for how to structure your code. A rule is an explicit mandate around how to structure your code with minimal if any exceptions, while a guideline is a strong suggestion which may have exceptions.

Having a blend of rules & guidelines within a style guide is important because there are aspects of code structure which could potentially lead to huge headaches if there is variation. But for most recommendations, it’s useful to give your team leeway to move outside of them in specific situations. Otherwise, if everything is a strict rule, then there may not be enough flexibility for your team to operate and it can lead to frustration.

Section 3.1 - recommendations around semicolon are examples of explicit rules

Section 3.18 - recommendations around function length is an example of a guideline that gives broad suggestions for length, but leaves flexibility for individual functions.

Recommendations to Avoid Danger

The Google Style Guide only has a few recommendations that would fall into the “avoiding danger” category and they tackle a blend of areas. They are primarily around preventing bugs from being introduced, but section 3.8.2 also introduces some interesting rules around licensing and IP requirements, which can be critical standards within open source projects.

2.5 Global Variable [Guideline]

While not fully banned, Global Variables are strongly discouraged from being used because of their risk to changing module behaviour during their import. This is a prime example of a recommendation in a style guide being used to discourage a style of coding that could significantly increase the bug risk in the project.

2.12 Default Argument Values [Guideline]

Putting in place simple guide-rails (but not blanket bans) throughout the style guide can help to reduce the risk of critical errors. It’s ok to use default argument values, as long as you’re not using mutable objects as default values.

2.18 Threading [Rule]

This is another section of the style guide designed to limit bug risk, but rather than creating a requirement because the primary use case is a risk, Google has a rule not to assume the atomicity of built in types because of the risks around edge cases.

2.20 Modern Python: from future imports [Guideline]

Many of Google’s open source projects need to be compatible with a wide variety of Python versions. They recommend using from future import statements to help make runtime version transitions smoother. This recommendation is probably somewhat between “Avoid Danger” and “Enforcing Best Practices” as it both helps with reducing the risk of potential bugs and is reinforcing a best practice around using new functionality.

3.8.2 Modules [Rule & Guideline]

Critical requirements in a Style Guide don’t just relate to how the code needs to be structured, but also can relate to business requirements, IP requirements, and managing security/reputational risk. As part of the broader Comments and Docstrings section of the Style Guide, Google lays out a clear requirement for every module to include a license boilerplate.

3.11 Files, Sockets, and similar Stateful Resources [Rule]

Beyond bug risk and business risk, another important parameter to consider in your Style Guide is ensuring best practices around code performance. Here, Google requires explicitly closing files and sockets (and other similarly closable resources) when finished using them to make sure you don’t inadvertently eat up resources.

3.17 Main [Rule]

Both for testing and automatic documentation, the Google Style Guide requires that all the functionality of an executable file live within a main() function and that the code always check if if name == 'main' before executing the code. This carries over an important aspect raised in 2.5 that you should not be executing code directly on import.

Recommendations to Enforce Best Practices
2.1 Lint [Guideline]

If you require or recommend that your team use certain tools or linters it is useful to explicitly outline that in your style guide. Here Google lays out the recommendations (and limitations) for using Pylint.

2.2 Imports [Rule]

We all rely on imports (both internal and external) frequently and having inconsistent approaches to handling import structures can lead to confusion and make it more likely that we introduce bugs into our code in the future. Google lays out explicit requirements for how to import modules and packages and in which limited situations you can import a module as an alias.

2.3 Packages [Rule]

Similar to the above rule, the Google Style Guide also has explicit rules for how to import packages clearly. In order to eliminate confusion, they require that you import each module by the full package name.

2.4 Exceptions [Guideline]

Defining how you handle cases around exceptions, errors, and logging are all important aspects to a style guide because inconsistency here can very quickly lead to your code becoming overly complicated and tricky to understand. Here Google has laid out a series of explicit conditions that exceptions need to follow.

2.6 Nested/Local/Inner Classes and Functions [Guideline]

Nesting functions can cause serious issues for code readability, but there are also use cases where they can be quite useful. This recommendation is an interesting example of how it can be important in a style guide to take a neutral position on issues, because it eliminates debate around whether you should or should not be always doing something. In this case - it is at the developer’s discretion for when to use nested local functions or classes, with a few explicit exceptions where they are discouraged.

2.7 Comprehensions & Generator Expressions [Guideline]

This recommendation helps create very explicit guidelines of when comprehension or generator expressions are ok to use - they need to be simple cases and need to fit in one line, otherwise they should not be used.

2.8 Default Iterators and Operators [Guideline]

I mentioned earlier that one key piece of a style guide is which built in features of a language you should leverage and which you should avoid. When it comes to default iterators and operators for supported types, the Style Guide strongly recommends that you use them because of their simplicity and efficiency.

2.9 Generators [Guideline]

Similar to the above rule, the Google Style Guide encourages the use of generators because they give simpler code.

2.10 Lambda Functions [Guideline]

As with nested functions, the Google Style Guide says it’s ok to use lambdas, but only in specific situations and doesn’t necessarily encourage their use. It suggests that anything that is longer than one line is a bad use case for a lambda.

2.11 Conditional Expressions [Guideline]

Throughout the Google Style Guide a fairly consistent theme exists. You should use Python functionality that helps to make your code shorter and more convenient for simple, one line use cases, but probably not for more complex situations. The same holds true for conditional expressions.

2.13 Properties [Guideline]

Like the above few recommendations, the guidelines around Properties are focused on making sure that they are not used in cases where they would make things significantly too complex (eg where they wouldn’t match the expectations of typical attribute access and not implementing your own property descriptor). But, like the above recommendations, they don’t discourage the overall use of properties

2.14 True/False Evaluations [Guideline]

All of the sections of the Python Language Rules have a Pros and Cons section around why you may or may not want to use a given feature of the language - and this recommendation might have my favorite con - “May look strange to C/C++ developers”. Overall they recommend using the “implicit” false in Python wherever possible, but the con they lay out is an important one to think about when you’re structuring your own style guide. Make sure you consider who is going to be contributing to your codebase and what their backgrounds are and how that might influence how you should structure your best practices.

2.16 Lexical Scoping [Guideline]

Compared with some of the previous recommendations which had heavy caveats, this one is relatively straightforward - it is ok to use lexical scoping.

2.17 Function & Method Decorators [Guideline]

Rather than spending much of the section focus on whether or not to use decorators (it says they’re ok to use judiciously), most of this section of the Style Guide focuses on how to handle decorators when you are using them - especially on how they fit into other guidelines within the Style Guide.

2.19 Power Features [Rule]

Within Google there are thousands of developers writing code in a variety of languages and they have different levels of expertise and comfort with Python. Here, developers are explicitly told not to use “Power Features” of Python because they can be difficult to understand for developers who a) aren’t familiar with them or b) are revisiting the code later. When you’re building your own style guide it’s important to consider what variation in experience you will have within your team and what types of features you want your team using.

2.21 Type Annotated Code [Guideline]

Interestingly, type annotation is a topic that’s covered in two separate sections of the Style Guide - here in the Python Language Rules section and again in the Python Style Rules section. This piece of the guide is relatively straightforward, simply stating that type annotation and type checking is strongly encouraged.

3.15 Getters & Setters [Guideline]

This is another section that encourages the general use of certain type of functionality in your code - in this case getter and setter functions - but only where the access is complex (not more broadly).

Recommendations to Ensure Consistency

I won’t go into as much detail on some of the consistency points and why they’re useful as I have for the previous sections, because many of the more formatting focused recommendations are relatively straightforward.

3.1 Semicolons [Rule]

Don’t terminate lines with semicolons or use semicolons to have 2 statements on the same line.

3.2 Line Length [Rule]

With a few exceptions, the max line length should be 80 characters.

3.3 Parentheses [Guideline]

In general you should use parentheses sparingly

3.4 Indentation [Rule]

Use 4 spaces for indentation.

3.5 Blank Lines [Rule]

Have 2 blank lines between top level definitions, 1 blank line between method definitions and the class line and first method, and 0 blank lines after a def line.

3.6 Whitespace [Rule]

Here the Google Style Guide lists a series of restrictions on when there should not be whitespace.

3.7 Shebang Line [Rule]

Only include the relevant shebang line (#!/usr/bin/env python3 (to support virtualenvs) or #!/usr/bin/python3) on files that are going to be executed directly

[3.8 Comments and Docstrings Guideline](https://google.github.io/styleguide/pyguide.html#38-comments-and-docstrings)

Structuring comments and docstrings consistently are important to driving the overall consistency and readability of a project. Here the Google Style Guide goes into detail on the formatting and structure of how to approach docstrings for a variety of cases, comments.

3.10 Strings [Guideline]

Inconsistency around string formatting can quickly make code cluttered and a pain to try to read through. In this section, the Google Style Guide sets out clear guidance around consistency of structuring strings (both long and short strings and f strings) and lays out clear guidance around how to structure and format logging and error messages.

3.12 TODO Comments [Rule]

An important piece of managing a complex codebase is thinking about how you manage known issues or known future improvements. Here is an explicit definition of how you should use TODO comments - especially who is referenced in the TODO and what that means.

3.13 Imports formatting [Guideline]

In section 2.2 there is a series of clear recommendations as to how to handle imports and how to structure them. This is instead focused on import formatting and sets definitions that imports should be defined on separate lines, that imports should be at the top of files, and that they should go from generic to specific.

3.14 Statements [Guideline]

Similar to the last several recommendations, this section is focused specifically on formatting and structuring your code so that you only have a single statement on a given line.

3.16 Naming [Guideline]

Names are hard. And inconsistent naming can create havoc within a project. That’s one of the reasons I really like this section of the Google Style Guide. It outlines what types of names to avoid (single character names, offensive terms, specific characters to avoid, etc), gives naming conventions to ensure consistency, and gives guidelines for naming different aspects of your code.

3.18 Function Length [Guideline]

You could create a wide variety of recommendations in your style guide around code quality requirements (eg length, complexity, working memory, etc.). Here, Google lays out a suggested limit to function length (40 characters) but also recognizes that there are exceptions where you may need to violate that. I like this approach of giving a strong suggestion, but not making it a strict rule for any code quality based section of a style guide.

3.19 Type Annotations [Guideline]

Not to be confused with 2.21 Type Annotated Code, this section sets out a series of guidelines around how to annotate your code and some broad guidelines on when and when not to annotate your code

Do I need a style guide?

Google has tens of thousands of software engineers on their team - so you might be thinking that having an explicit style guide is clearly good for a company with that level of complexity, but doesn’t make sense for smaller organisations. But, regardless of company or team size, the same truth is going to hold - the time spent reading, interacting with, and maintaining a section of code is going to be orders of magnitude more than the amount of time that you’re going to be spending writing it. This means that any gains your team gets from initial speed in writing your code in whatever way works best for each individual quickly gets eaten up by the long term costs of dealing with those clashing approaches.

If your team doesn’t already have a set of explicit (or consistent, but implicit) code standards, best practices, and/or guidelines that you are following then the Google Style Guide is a great jumping off point. Look at the full set of recommendations and see which fit well into your team’s approach, and then remove or add in any rules that would be more relevant for your team. Remember, there isn’t an objective set of right and wrong recommendations to have in your team’s Style Guide, it’s more important that your team is all going off the same playbook and it is helping your code become more consistent and effective.

Turn your team’s best practices and style guide into code → Get a Demo

What Is Technical Debt (and why a bit of it is a good thing)

Tim Gilboy — Wed, 09 Mar 2022 17:10:11 +0000

This is part 1 in our overview series on tech debt.

Technical debt isn't a tangible entity in software development. But it's a great analogy to let us think about how the decisions we make today in how we structure our code impact our future ability to effectively develop. Any time you've been slowed down because you were dealing with old code and can't make heads or tails of how it's structured, when you realise certain things just can't be extended how they were written, or when you've been slowed down by overly complex code are all examples of the costs of tech debt. But why do we use debt as an analogy? And show we be looking to constantly go back to our old code to try to eliminate technical debt from our codebases?

As best I can tell, tech debt first cropped up as a term in 1992 when Ward Cunningham laid out the basics of the idea. He had 3 main points about what tech debt is and how it parallels financial debt:

"Shipping first time code is like going into debt."
"A little debt speeds up development as long as it's paid back promptly."
"Every minute spent on not-quite-right-code counts as interest on the debt."

Let's take a look at the analogy in more depth by looking at a (semi) practical example. Let's imagine we're looking to start a lemonade stand, but only have $5 to start out. We need to buy lemons and sugar so we can start making lemonade and start making money, but $5 won't buy us that many ingredients.

The Git Bank offers loans to any aspiring lemonade stand owners. Their standard terms are a $50 loan with 5% weekly interest (pricey, but lemonade stands are risky investments!). We could take the loan (and go into debt) so we can buy more supplies, start selling more lemonade, and get the business off the ground faster, but is it the right decision?

Let's say we make $1.10 for every $1 we invest into supplies and that we choose to reinvest all of our profits back into more supplies.
At the end of the first week we would have had $9.74 if we don't take the loan vs $44.94 if we do take the loan and repay it.
- $97.44 in last day earnings
- ($50) for the loan principal
- ($5) for the interest

We can view tech debt that comes into play when we're developing new software in the exact same way. We could spend the extra time to fully refactor and fine tune every aspect of the code, but that will make it take longer to release the code and get the benefits from it. Instead we could ship it earlier, even if it's not perfect. At some point, we'll have to go back and refactor that code - the time this will take is the principal of our technical debt. We're also going to be slowed down in future development by some of the shortcuts we took - this time lost on future development is the interest we keep getting charged on our technical debt.

It seems pretty clear that taking on some technical debt can be worth it in the short term since it allows us to move faster, but how quickly should we be aiming to pay it back (ie refactor and clean up our code)? Cunningham suggested that we should pay it back immediately - and if that's easy and quick and we're going to work on that code frequently then that's a great idea. But, if we're dealing with sections of our code base that we don't actually touch all that often then it can be more efficient to actually leave some of that debt unpaid. Just like financial debt, it's not necessarily optimal to target zero tech debt.

To quickly recap:

Technical debt is a nice analogy we can use to think about the cost/beneift tradeoffs we make with our decisions when we are developing.
Technical debt can be split into a principal component (the cost in time it will take to refactor or fix the underlying code) and an interest component (the ongoing slowdown to future work).
While we want to keep the ongoing costs of tech debt low, it's not necessarily optimal to target having no technical debt.

Looking to reduce your tech debt or prevent it from building up in the first place? Learn how Sourcery can help your team improve all your Python code.

Python Refactorings Part 3

Tim Gilboy — Fri, 14 May 2021 16:58:52 +0000

Writing clean, Pythonic code is all about making it as understandable, yet concise, as possible. This is the third part of a series on Python refactorings, based on those that can be done automatically by Sourcery. Catch the first part here, and the second part here.

The focus of this series is on why these changes are good ideas, not just on how to do them.

1. Add guard clause

Deeply nested functions can be very difficult to understand. As you read them you have to remember the conditions that hold for each level of nesting. This can be even more difficult in Python, given that there are no brackets to help define conditional blocks. An easy way to reduce nesting is to convert conditionals into guard clauses.

As an example let's look at this function:

def should_i_wear_this_hat(self, hat):
    if isinstance(hat, Hat):
        current_fashion = FASHION_API.get_fashion(FASHION_TYPE.HAT)
        weather_outside = self.look_out_of_window()
        is_stylish = self.evaluate_style(hat, current_fashion)
        if weather_outside.is_raining:
            print("Damn.")
            return True
        else:
            print("Great.")
            return is_stylish
    else:
        return False

This is quite hard to parse, given the two layers of nesting. When I get to the else at the bottom I need to flick back and forth a bit between that and the if test at the top before I've understood what's going on. This if condition is a check for an edge case, where something that isn't a hat is passed in. Since we just return False here it's a perfect place to introduce a guard clause:

def should_i_wear_this_hat(self, hat):
    if not isinstance(hat, Hat):
        return False

    current_fashion = get_fashion()
    weather_outside = self.look_out_of_window()
    is_stylish = self.evaluate_style(hat, current_fashion)
    if weather_outside.is_raining:
        print("Damn.")
        return True
    else:
        print("Great.")
        return is_stylish

We add a guard clause by inverting the condition and returning immediately. The edge case is now taken care of at the start of the function and I don't have to worry about it any more.

Having reduced the level of nesting, the rest of the function is now easier to read. Whether to add a guard clause is sometimes subjective. For really short functions it may not be worth doing. Where functions are longer or more complex it can often be a useful tool.

2. Swap if/else to remove empty if body

One pattern we sometimes see is a conditional where nothing happens in the main body, and all of the action is in the else clause.

if location == OUTSIDE:
    pass
else:
    take_off_hat()

In this case we can make the code shorter and more concise by swapping the main body and the else around. We have to make sure to invert the conditional, then the logic from the else clause moves into the main body.

if location != OUTSIDE:
   take_off_hat()
else:
   pass

We then have an else clause which does nothing, so we can remove it.

if location != OUTSIDE:
    take_off_hat()

This is easier to read, and the intent of the conditional is clearer. When reading the code I don't have to mentally invert it to understand it, since that has been done for me.

3. Merge append into list declaration

When declaring a list and filling it up with values one way that can come naturally is to declare it as empty and then append to it.

hats_i_own = []
hats_i_own.append('panama')
hats_i_own.append('baseball_cap')
hats_i_own.append('bowler')

This can be done in place, shortening the code and making the intent more explicit. Now I just need to glance at one line to see that I'm filling a variable with hats, rather than four.

hats_i_own = ['panama', 'baseball_cap', 'bowler']

Doing it this way is also slightly more performant since it avoids the function calls to append. The same holds true for filling up other collection types like sets and dictionaries.

4. Move assignments closer to their usage

The scope of local variables should always be as tightly defined as possible and practicable.

This means that:

You don't have to keep the variable in your working memory through the parts of the function where it's not needed. This cuts down on the cognitive load of understanding your code.
If code is in coherent blocks where variables are declared and used together, it makes it easier to split functions apart, which can lead to shorter, easier to understand methods.
If variables are declared far from their usage, they can become stranded. If the code where they are used is later changed or removed unused variables can be left sitting around, complicating the code unnecessarily.

Let's take another look at the earlier example.

def should_i_wear_this_hat(self, hat):
    if not isinstance(hat, Hat):
        return False

    current_fashion = get_fashion()
    weather_outside = self.look_out_of_window()
    is_stylish = self.evaluate_style(hat, current_fashion)
    if weather_outside.is_raining:
        print("Damn.")
        return True
    else:
        print("Great.")
        return is_stylish

Here the is_stylish variable isn't actually needed if the weather is rainy. It could be moved inside the else clause. This means we can also move the current_fashion variable, which is only used here. You do need to check that these variables aren't used later in the function, which is easier if functions are kept short and sweet.

Moving the assignment to current_fashion also avoids a function call when the weather is raining, which could lead to a performance improvement if it's an expensive call.

def should_i_wear_this_hat(self, hat):
    if not isinstance(hat, Hat):
        return False

    weather_outside = self.look_out_of_window()
    if weather_outside.is_raining:
        print("Damn.")
        return True
    else:
        print("Great.")
        current_fashion = get_fashion()
        is_stylish = self.evaluate_style(hat, current_fashion)
        return is_stylish

We could actually then go one step further and inline the is_stylish variable. This shows how small refactorings can often build on one another and lead to further improvements.

def should_i_wear_this_hat(self, hat):
    if not isinstance(hat, Hat):
        return False

    weather_outside = self.look_out_of_window()
    if weather_outside.is_raining:
        print("Damn.")
        return True
    else:
        print("Great.")
        current_fashion = get_fashion()
        return self.evaluate_style(hat, current_fashion)

5. Use items() to directly unpack dictionary values

When iterating over a dictionary a good tip is that you can use items() to let you access the keys and values at the same time. This lets you transform this:

hats_by_colour = {'blue': ['panama', 'baseball_cap']}
for hat_colour in hats_by_colour:
    hats = hats_by_colour[hat_colour]
    if hat_colour in self.favourite_colours:
        think_about_wearing(hats)

into this:

hats_by_colour = {'blue': ['panama', 'baseball_cap']}
for hat_colour, hats in hats_by_colour.items():
    if hat_colour in self.favourite_colours:
        think_about_wearing(hats)

This saves us the line that we used to assign to hats, incorporating it into the for loop. The code now reads more naturally, with a touch less duplication.

6. Simplify sequence comparison

Something we often do is check whether a list or sequence has elements before we try and do something with it.

if len(list_of_hats) > 0:
    hat_to_wear = choose_hat(list_of_hats)

A Pythonic way of doing this is just to use the fact that Python lists and sequences evaluate to True if they have elements, and False otherwise.

This means we can write the above code more simply as:

if list_of_hats:
    hat_to_wear = choose_hat(list_of_hats)

Doing it this way is a convention, set out in Python's PEP8 style guide. Once you've gotten used to doing it this way it does make the code slightly easier to read and a bit less cluttered.

Conclusion

As mentioned, each of these is a refactoring that Sourcery can automatically perform for you. We're planning on expanding this blog series out and linking them in as additional documentation, with the aim of turning Sourcery into a great resource for learning how to improve your Python skills. You can read the next part in the series here.

If you have any thoughts on how to improve Sourcery or its documentation please do email us or hit us up on Twitter.

Python Refactorings Part 2

Tim Gilboy — Mon, 15 Mar 2021 17:39:32 +0000

Writing clean, Pythonic code is all about making it as understandable, yet concise, as possible. This is the second part of a series on Python refactorings, based on those that can be done automatically by Sourcery. Catch the first part here and the next part here.

The focus of this series is on why these changes are good ideas, not just on how to do them.

Convert for loop into list/dictionary/set comprehension A pattern that we encounter time and again when coding is that we need to create a collection of values.

A standard way of doing it in most languages would be as follows:

cubes = []
for i in range(20):
    cubes.append(i ** 3)

We create the list, and then iteratively fill it with values - here we create a list of cubed numbers, that end up in the cubes variable.

In Python we have access to list comprehensions. These can do the same thing on one line, cutting out the clutter of declaring an empty list and then appending to it:

cubes = [i ** 3 for i in range(20)]

We've turned three lines of code into one which is a definite win - it means less scrolling back and forth when reading methods and helps keep things manageable.

Squeezing code onto one line can make it more difficult to read, but for comprehensions this isn't the case. All of the elements that you need are nicely presented, and once you are used to the syntax it is actually more readable than the for loop version.

Another point is that the assignment is now more of an atomic operation - we're declaring what cubes is rather than giving instructions on how to build it. This makes the code read like more of a narrative, since going forward we will care more about what cubes is than the details of its construction.

Finally comprehensions will usually execute more quickly than building the collection in a loop, which is another factor if performance is a consideration.

If you need a deeper dive into comprehensions check out my post here. It goes into exactly how they work, how to use them for filtering, and dictionary and set comprehensions.

Replace assignment with augmented assignment Augmented assignments are a quick and easy bit of Python syntax to include.

Wherever there's code like this:

count = count + other_value

we can replace it with:

count += other_value

This is a bit shorter and clearer - we don't need to think about the count variable twice. Other operators that can be used include -=, *=, /= and **=.

One thing to be slightly careful of is that the type you're assigning to has to have the appropriate operator defined. For instance numpy arrays do not support the /= operation.

Inline variable that is only used once Something that we often see in people's code is assigning to a result variable and then immediately returning it.

def state_attributes(self):
    """Return the state attributes."""
    state_attr = {
        ATTR_CODE_FORMAT: self.code_format,
        ATTR_CHANGED_BY: self.changed_by,
    }
    return state_attr

Here it's better to just return the result directly:

def state_attributes(self):
    """Return the state attributes."""
    return {
        ATTR_CODE_FORMAT: self.code_format,
        ATTR_CHANGED_BY: self.changed_by,
    }

This shortens the code and removes an unnecessary variable, reducing the mental load of reading the function.

Where intermediate variables can be useful is if they then get used as a parameter or a condition, and the name can act like a comment on what the variable represents. In the case where you're returning it from a function, the function name is there to tell you what the result is - in the example above it's the state attributes, and the state_attr name wasn't providing any extra information.

Replace if statement with if expression It often happens that you want to set a variable to one of two different values, depending on some program state.

if condition:
    x = 1
else:
    x = 2

This can be written on one line using Python's conditional expression syntax (its version of the ternary operator):

x = 1 if condition else 2

This is definitely more concise, but it is one of the more controversial refactorings (along with list comprehensions). Some coders dislike these expressions and find them slightly harder to parse than writing them out fully.

Our view is that as long as the conditional expression is short and fits on one line it is an improvement. There's only one statement where x is defined as opposed to having to read two statements plus the if-else lines. Similarly to the comprehension example, when we're scanning the code we usually won't need to know the details of how x gets assigned, and can just see that it's being assigned and move on.

Replace unneeded comprehension with generator One tip is that functions like any, all and sum allow you to pass in a generator rather than a collection. This means that instead of doing this:

hat_found = any([is_hat(item) for item in wardrobe])

you can write:

hat_found = any(is_hat(item) for item in wardrobe)

This removes a pair of brackets, making the intent slightly clearer. It will also return immediately if a hat is found, rather than having to build the whole list. This lazy evaluation can lead to performance improvements.

Note that we are actually passing a generator into any() so strictly speaking the code would look like this:

hat_found = any((is_hat(item) for item in wardrobe))

but Python allows you to omit this pair of brackets.

The standard library functions that accept generators are:

'all', 'any', 'enumerate', 'frozenset', 'list', 'max', 'min', 'set', 'sum', 'tuple'

Simplify conditional into return statement The last refactoring to look at is where we reach the end of a method and want to return True or False. A common way of doing so is like this:

def function():
    if isinstance(a, b) or issubclass(b, a):
        return True
    return False

However, it's neater just to return the result directly like so:

def function():
    return isinstance(a, b) or issubclass(b, a)

This can only be done if the expression evaluates to a boolean. In this example:

def any_hats():
    hats = [item for item in wardrobe if is_hat(item)]
    if hats or self.wearing_hat():
        return True
    return False

We can't do this exact refactoring, since now we could be returning the list of hats rather than True or False. To make sure we're returning a boolean, we can wrap the return in a call to bool():

def any_hats():
    hats = [item for item in wardrobe if is_hat(item)]
    return bool(hats or self.wearing_hat())

Conclusion
As mentioned, each of these is a refactoring that Sourcery can automatically perform for you. We're planning on expanding this blog series out and linking them in as additional documentation, with the aim of turning Sourcery into a great resource for learning how to improve your Python skills. You can read the next part in the series here.

If you have any thoughts on how to improve Sourcery or its documentation please do email us or hit us up on Twitter