DEV Community: Aquiles Carattino

Difference between conda environments and virtual environments in Pyhon

Aquiles Carattino — Mon, 10 Jan 2022 08:57:06 +0000

What is Anaconda and the conda package manager

Anaconda is a full-fledged package manager that works on most operating systems.

For people coming from a Linux background, this is not novelty, but for Mac and Windows users, package managers are unknown. A package manager is a tool that allows to install programs AND their dependencies. This removes a lot of duplication (programs shipping dependencies) and makes the system easier to maintain (a single source of truth for versions and requirements).

Anaconda brought the idea of a package manager to windows, focusing on scientific/engineering Python developers. Many libraries such as numpy have dependencies that go much beyond what the default Python installer can take care of. Not so many years ago getting a decent development setup on Windows was a total nightmare, and Anaconda solved that problem.

Installing packages is only half the problem, being able to isolate project dependencies is the other half.

When developing several projects on the same computer, managing dependencies can become crucial to ensure quality and compatibility. That's the root idea of a conda environment: isolating the dependencies for each project in separated spaces. An update of one of the libraries will have an impact only on that specific project structure, and not on the others.

Conda environments are so specific that they allow to select the version of Python we want to run, for example. We can track the version of the underlying libraries, for example those on which numpy relies to perform calculations. Conda environments are ideal for ensuring reproducibility. You know you can bring the same environment to the next computer you work on, and the results will be the same.

How do virtual environments compare to conda environments

Python virtual environments are the light version of the conda environments.

Virtual environments are a native tool to Python developers, and they provide a functionality similar to that of the conda environments. The main difference is that they rely on the Python package manager. Libraries and programs that do not belong to the Python ecosystem can't be installed with these tools. In these cases, one must rely on the operating system's ability to install programs.

The Python package manager improved a lot in the past decade. Today, no one has issues installing numpy, pyqt, or scikit with a simple pip command. However, many of these packages still rely on libraries available at an operating-system level. For example, Python itself is grabbed from the operating system and not installed purposefully on each environment. Virtual environments are a very good tool to isolate dependencies across projects that rely mostly on Python packages.

How to choose between conda environments and virtual environments

The choice between one and the other comes down to understanding how much control we have on the underlying operating system, and what would the risks be.

One of the largest risks when deploying application is that an un-intended software update generates a cascading effect. For example, Windows updates itself giving very little control to the user. In these update cycles it may happen that there's a new release of Python, which in turn makes our program misbehave. However, thinking about ourselves as the developers and consumers of the same code hides the real problem.

If we are delivering software, either by deploying it on a server accessible by others, or as part of an embedded system, or as a program that others will install in their computers, we probably want to minimize the maintenance burden. If we have control on the packages installed on a server, for example, then I could argue that conda environments are not necessarily adding value. We could rely on the operating system's package manager.

If our projects are exclusively Python-dependent, virtual environments are so lightweight that there's only upside in using them.

Even if we have control on the computer where the code is going to be deployed, adding the layer of a virtual environment gives plenty of peace of mind. It prevents problems with incompatible cross-dependencies, allows careful updates and testing. Even if we don't have too much control on the underlying libraries, it may be just what we need.

How I combine virtual environments and conda environments

I normally work on two types of problems using Python: scientific work that requires a high degree of reproducibility across computers, and side projects that are mostly web-based.

For scientific-based projects, I always rely on Anaconda. It gives me peace of mind that there's an authoritative body overseeing which packages are being released. If a user asks for support, I can be quite confident that a conda install will not expose them to unnecessary risks and burdens. It also limits the problems tracking down dependencies, and troubleshooting version mismatches (especially on Windows).

My side projects include maintaining websites written with Python. Python for the lab and the Climate Helpdesk being the two most prominent examples. In these cases, the code is deployed to a server on which I have control on the dependency installation. In that case, I use virtual environments both locally and remotely in order to test the outcomes of the modifications I want to implement.

More than a combination is the selection of the proper tool for the job.

It is important not to follow trends and broad advice without thinking first. Conda and virtual environments are powerful tools. They simply serve different purposes and is up to each one to decide what tool to pick.

Bear in mind that all this discussion does not even consider the next big thing of nowadays: Docker Containers. Virtualization is a completely different realm compared to virtual environments. That would be the topic for another discussion.

How do you pick fonts?

Aquiles Carattino — Tue, 06 Oct 2020 08:46:14 +0000

I am going through a redesign phase of my personal space, and I am trying to pick fonts for it. Given that there is such a large selection (both free and paid), how do you pick fonts for your projects?

How to Make Cool Underlines for Links with TailwindCSS

Aquiles Carattino — Mon, 05 Oct 2020 20:53:15 +0000

Making underlines for links instead of changing the color of the letters improves a lot the readability of documents. However, underlining itself does not allow to control the thickness of the line nor its color. There are two ways to achieve control on how the underlining looks like: using the boxShadow property of TailwindCSS, or tuning the background image.

Box Shadow

Tailwind provides some basic shadows, but if we want to add custom shadows to our links, we must edit the tailwind.config.js file, and add the following:

    theme: {
        extend: {
            boxShadow: {
                link: '0 -4px 0 0 rgba(178, 245, 234, .7) inset',
            },
    },
}

The syntax is exactly what box-shadow takes. Now, if we want to underline a link, we can use the following in the HTML:

<a href="target.html" class="no-underline shadow-link">target</a>

We can also apply it globally to all links, by adding the following to the style.css file:

@tailwind base;

a {
    @apply no-underline shadow-link;
}

@tailwind components;

Background Image

Using a background image as an underline requires slightly more tuning but allows for much more stunning results. We can add to style.css after the @tailwind utilities line:

@layer base {
    a {
        background-image: linear-gradient(to top, orange, yellow, transparent);
        background-position: 0 110%;
        background-repeat: no-repeat;
        background-size: auto .75rem;
    }
}

We use the @layer base to create a global style that will be applied to all the a tags. The rest is CSS code to create a linear gradient that goes from orange, to yellow to transparent, from bottom to top. The result would like in the image below:

So, if you see those fancy underlines and want to create them with Tailwind, you know now where to start!

Show off your personal landing page

Aquiles Carattino — Sat, 26 Sep 2020 06:25:23 +0000

I am curious to see how people present themselves online, how does your personal blog/portfolio/landing page look like?

I am gathering inspiration for updating mine. I created it in a couple of minutes just because I needed to get something quick out there and haven't updated it yet and I really don't like how it looks.

Differences of Python Multiprocessing on Windows and Linux

Aquiles Carattino — Sat, 26 Sep 2020 06:18:27 +0000

Multiprocessing is an excellent package if you ever want to speed up your code without leaving Python. When I started working with multiprocessing, I was unaware of the differences between Windows and Linux, which set me back several weeks of development time on a relatively big project. Let's quickly see how multiprocessing works and where Windows and Linux diverge.

The quickest way of showing how to use multiprocessing is to run a simple function without blocking the main program:

import multiprocessing as mp
from time import sleep


def simple_func():
    print('Starting simple func')
    sleep(1)
    print('Finishing simple func')


if __name__ == '__main__':
    p = mp.Process(target=simple_func)
    p.start()
    print('Waiting for simple func to end')
    p.join()

Which outputs the following:

Waiting for simple func to end
Starting simple func
Finishing simple func

The output is what we were expecting. Let's go to the core of the problem at hand by studying how this code behaves:

import multiprocessing as mp
from time import sleep


print('Before defining simple_func')

def simple_func():
    print('Starting simple func')
    sleep(1)
    print('Finishing simple func')


if __name__ == '__main__':
    p = mp.Process(target=simple_func)
    p.start()
    print('Waiting for simple func to end')
    p.join()

If we run this code on Windows, we get the following output:

Before defining simple_func
Waiting for simple func to end
Before defining simple_func
Starting simple func
Finishing simple func

While on Linux we get the following output:

Before defining simple_func
Waiting for simple func to end
Starting simple func
Finishing simple func

It does not look like much, except for the second Before defining simple_func, and this difference is crucial. On Linux, when you start a child process, it is Forked. It means that the child process inherits the memory state of the parent process. On Windows (and by default on Mac), however, processes are Spawned. It means that a new interpreter starts and the code reruns.

It explains why, if we run the code on Windows, we get twice the line Before defining simple_func. As you may have noticed, this could have been much worse if we wouldn't include the if __main__ at the end of the file, let's check it out. On Windows, it produces a very long error, that finishes with:

RuntimeError: 
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.

While on Linux, it works just fine. It may not look like much, but imagine you have some computationally expensive initialization task. Perhaps you do some system checks when the program runs. Probably you don't want to run all those checks for every process you start. It can get even more interesting if you have values that change at runtime:

import multiprocessing as mp
import random

val = random.random()

def simple_func():
    print(val)


if __name__ == '__main__':
    print('Before multiprocessing: ')
    simple_func()
    print('After multiprocessing:')
    p = mp.Process(target=simple_func)
    p.start()
    p.join()

On Windows, it would give an output like this:

Before multiprocessing:
0.16042209710776734
After multiprocessing:
0.9180213870647225

While on Linux, it gives an output like this:

Before multiprocessing:
0.28832424513226507
After multiprocessing:
0.28832424513226507

And this brings us to the last topic and the reason why I lost so much time when I had to port code written in Linux to work on Windows. A typical situation in which values change at runtime is when you are working with classes. Objects are meant to hold values; they are not static. So, what happens if you try to run a method of a class on a separate process? Let's start with a straightforward task:

import multiprocessing as mp


class MyClass:
    def __init__(self, i):
        self.i = i

    def simple_method(self):
        print('This is a simple method')
        print(f'The stored value is: {self.i}')

    def mp_simple_method(self):
        self.p = mp.Process(target=self.simple_method)
        self.p.start()

    def wait(self):
        self.p.join()


if __name__ == '__main__':
    my_class = MyClass(1)
    my_class.mp_simple_method()
    my_class.wait()

The code works fine both on Linux and Windows. And this may happen for a lot of different scenarios, until one day you try to do something slightly more complicated, like writing or reading from a file:

import multiprocessing as mp


class MyClass:
    def __init__(self, i):
        self.i = i
        self.file = open(f'{i}.txt', 'w')

    def simple_method(self):
        print('This is a simple method')
        print(f'The stored value is: {self.i}')

    def mp_simple_method(self):
        self.p = mp.Process(target=self.simple_method)
        self.p.start()

    def wait(self):
        self.p.join()
        self.file.close()


if __name__ == '__main__':
    my_class = MyClass(1)
    my_class.mp_simple_method()
    my_class.wait()

On Linux, the code above works fine. On Windows (and Mac), however, there'll be a very nasty error:

[...]
    ForkingPickler(file, protocol).dump(obj)
TypeError: cannot serialize '_io.TextIOWrapper' object

Pay attention to the fact that we don't do anything with the file. We just open and store it as an attribute in the class. However, the error already points to an interesting feature. The way Spawning works is by pickling the entire object. Therefore, if we have a class or an attribute that is not pickable, we will not be able to start a child process with it.

And, for people working with hardware, most likely the communication with the device, in pretty much the same way that a file is non-pickable. It does not matter how much you try to make it multiprocessing safe by implementing locks or whatnot. The root problem is at a lower level.

Is there a way of solving it?

Sadly, there is no way of changing how processes start on Windows. You can, on the other hand, change how processes start on Linux. It would allow you to be sure your program also runs on Windows and Mac. We just need to add the following:

if __name__ == '__main__':
    mp.set_start_method('spawn')
    my_class = MyClass(1)
    my_class.mp_simple_method()
    my_class.wait()

By using set_start_method, the program will give the same error on Windows and Linux. Whether you need to add this line or not depends on what do you want to achieve.

So, if you ever encounter these discrepancies, you will have to re-think the design of your program. I had objects with non-pickable attributes, especially drivers for devices and ZMQ sockets.

Speed is another factor

Even though processes usually speed up the speed of a program by leveraging multiple cores on a computer, starting each process can be time-consuming. The fact that on Windows and Mac Python needs to pickle the objects to create child processes adds an overhead that may offset the benefits of running on separated processes. It is especially relevant when you have many small tasks to perform, instead of a couple of long-running ones.

Therefore, when using processes, improving the speed of the program is not a granted outcome. You should always benchmark your application to understand where and how different components can affect its behavior.

Python Slots to the Rescue

Aquiles Carattino — Thu, 30 Jul 2020 16:09:37 +0000

Not too long ago I discovered that Python implements the slots pattern. They allow you limit the attributes that a class can have after instantiating it.

The code below works, but it may give raise to un-intended behavior:

class Camera:
    exposure = '1ms'

cam = Camera()
cam.exposure_time = '10ms'

Since Python allows creating attributes of classes at runtime, the user of our Camera class may, inadvertely, use a wrong name and never realize of the mistake. There are many ways of avoiding this problem, but today I'll discuss about slots. We can change our class:

class Camera:
    __slots__ = ['exposure', ]
    exposure = '1ms'

If we repeat what we did earlier, we will now get an error:

>>> cam = Camera()
>>> cam.exposure_time = '10ms'
...
AttributeError: 'Camera' object has no attribute 'exposure_time'

Which, I believe is a very descriptive error.

Using slots has also the advantage of reducing the memory footprint of our objects. If we are going to create millions of camera objects at the same time (thinking about a server handling requests, for example), it may be smart to optimize the attribute declaration.

Did you know about slots? Have you ever had to use them in your code?

I started a digital garden and so should you

Aquiles Carattino — Thu, 30 Jul 2020 16:00:00 +0000

A month ago I decided to start a digital garden, which is, in a way, an evolution to plain note-taking. The core idea is that you write down your thoughts and create links between the notes. Each note is then a seed that can sprout and create new insight, new knowledge. Generally speaking I liked the idea, but what really made me take the plunge was that it was the approach I was looking for to lower my barrier at pushing content online.

Over the years I have started, growth, and abandoned many blogs. A problem I always had (and didn't realize) was that blogs force a certain structure and quality for each article. Therefore, if I wanted to write about Python I created a blog for that. If I want to write about my vacation trips, I had a blog for that. But every time I wanted to put something online, I had to spend countless hours polishing the content.

Then it enters into scene the idea of Learning in Public. If you and your readers agree that what they are looking at is work in progress, that mistakes, unfinished thoughts are allowed, then the self-imposed barrier for quality drops.

For many developers I think this can be of great help. Instead of pushing themselves to write tutorials, and very extensive articles, a digital garden allows for code snippets, or half-baked ideas. Perhaps you don't cover all cases, or all applications, but it can still be a stepping stone for something bigger once connected to other thoughts.

Even though I could have always taken notes, knowing that I am making them public gives me joy. That feeling helps me dedicate few minutes every morning, before starting my daily work duties to write down the ideas I have in my mind, the things I've read, the podcasts I've listened to.

It is too early to tell if it will have any lasting impact on my learning process or my overall content generation flow. For sure I had fun building it and I still have fun pushing content to it as often as I can.

If you want to see it in action, head to aquiles.me

How to Write a Programming Book (A practical but not technical guide)

Aquiles Carattino — Wed, 20 May 2020 19:54:52 +0000

This article first appeard on my blog.

I didn't plan to write a programming book. This means I followed a very inefficient path, which led me to a position where I was forced to re-write the entire content. However, in this process, I have learned a great deal about what I should have done to be more efficient and to deliver more value to my readers. Some of the things I outline in this article are general to any kind of writing, not only technical books but also for shorter pieces such as this one.

Why

If you ever start the path of writing a book, you should ask yourself why you are doing it. There are many reasons, and none is more valid than the other. You could write a book because you want to earn money by selling it. Perhaps a book is an excellent strategy to position yourself in a field. I have seen many people releasing free e-books to collect e-mails for their newsletters, or to try to become known thinkers in an area foreign to them.

Whatever your reason is, the best thing you can do is write it down, so you don't forget it. The reason you embark on a relatively complex project such as writing a book should guide your decisions. If you know you want to sell your book, you must be sure you meet the market's needs. If the book is mandatory reading for the students of a class, you should be sure you cover all the needed topics. If you are entering a new field, be sure you are up to date with the area.

The Outcome

Knowing why you write a book only sets the north. The next step is to think about your readers. Think about what message you expect to pass to someone who finishes the book. Depending on the subject and the type of book you want to write, you can think about a list of topics that you should cover. In my case, the book is a guide through an entire project. The outcome for someone who reads the book is a complete program.

I have seen many programming books that tackle different small projects. And with each project, there is a bit of new knowledge transmitted. Sometimes there is cohesion between the chapters, but often they can be read independently. Particularly, books aimed at more advanced programmers tend to be more atomic. Chapters are independent and focus on specific topics, without necessarily building upon each other.

The Base Line

The next crucial thing to know is where the book is starting once you know where the book is going. In my opinion, this may be one of the most challenging tasks. Is your book targeted at people who know how to program, but don't know a specific language? Or they already know the language but don't know about programming patterns? Are they comfortable using the command line?

If you don't set a baseline somewhere, you risk starting a book that grows backward.

There is always something else that needs to be explained earlier to understand what is coming afterward. It is possible to develop the feeling that your book should include everything. When this feeling appears, you should challenge it. Ask yourself the kind of readers you want to have. Are you targeting those who are just starting or those with few years of experience?

Once you fix the baseline and the outcome, you will have an idea of the size of the challenge you are up to. This is the moment to judge whether it is within your reach or better think about something else.

Outline of the Content

Once you know where the book starts and where it should finish, it is time to decide what goes inside. Most books are structured in chapters because they give a clear break to the reader. Once a chapter is finished, they can take a break before jumping to the next. For technical books, chapters should be cohesive and with a narrow scope. Remember that there is no limit to the number of chapters a book can have.

Ideally, if you already know the topic reasonably well, you can write the book's outline as a series of bullet points. Each element on the list could be a chapter or a section. You can also be slightly more specific and write down some ideas for content, what do you want to transmit, or some keywords.

Sometimes, however, you need to do some research to write the book. Many times, when you decide to write a book, you do it based on what you already know. However, the way you acquired that knowledge may have been very unstructured. Writing a book is an excellent way of challenging that knowledge and filling your own knowledge gaps. If this is your case, the outline of the book becomes fluid.

When you write a book, you should never forget where you are starting it, and where you are finishing it.

When your book's contents become fluid, you should always remember the two anchor points you defined earlier. Those are the only two places to keep you contained, preventing you from going off the rail. Once you start piling content, always check that every step brings you closer to the outcome you expect. If something off-topic appears along the way, store it safely for when you are looking for inspiration for your next book.

Execution

Once you have a project with a clear start, goal, and path, it all comes down to execution. Writing is not only a skill, but it is also a habit. The more you write, the easier it becomes. Writing a bit every day is better than writing a lot once per month. However, you must also acknowledge that writing is something that happens in between other things.

If you manage to spare some specific time per week for your book, it will be a great help. Part of the reasons for losing motivation is the lack of progress. If today your book is not closer to being complete than it was 6 months ago, it is hard to keep the spirits up. If that is the case, first remember why you started writing in the first place. Then, focus again on the outline and think about what is missing.

Separate research from writing and editing.

I, personally, tend to start writing until the point I realize there is something I am doubtful about. I start reading about it, which blocks my writing. While researching, I roam towards topics I didn't need for clarifying my doubt. What was supposed to be a writing session ends up showing little progress. If you are writing a programming book, the research could be replaced by developing code. Developing while you write is a quick way of diverging from your real goal.

Before starting to write, try to think about the things you need to know. One thing is looking up, another is researching on a subject. When you start writing, just write. Perhaps you make mistakes, maybe some sentences could have been re-written more beautifully. If you keep typing without looking back, you will make significant progress. Remember, progressing is key to keep the motivation.

Once you are done with writing, it is time for editing. Re-read what you have written. Correct what you can, mark the things you are doubtful about, or you are not inspired to improve.

Separating research from the rest is more or less straightforward. It is such a different task that most people won't really struggle. Separating writing from editing, however, is much harder and takes a lot of discipline. Especially when you see the red marks under the words you type, or the warnings that some text editors generate. Each one should find their own path. Some people prefer to spend time on each paragraph as they write. I prefer writing several paragraphs and going through them on another day.

Planning

If you are one of the lucky ones who can spend a fixed amount of time per week on the book, you can also try to plan for each chapter. If it is your first book, you won't be able to calculate exact times. You can try, at least, to have clear objectives that you can keep updating as time passes by. You will quickly learn how many words you can write per hour and how many words are there in a chapter. From there, calculating is easier.

If you are not spending a fixed amount of time on the book, you can still plan but without dates. You can always add milestones such as finish researching for the chapters, writing X chapter, or editing. Having a list in front of you that gets shorter after every time you sit to work on something can give you a big motivation push.

Finishing

Obvious as it may seem, finishing the book is an important step. It doesn't matter how much you correct it. Every time you read what you have written, you will find something to change. Knowing where to put the final dot and deciding that the book is ready takes a lot of courage. Publishing a book entails a degree of risk: what would happen if there is a mistake, or if people don't like it.

You can always ask feedback from people you trust to gain confidence. The reality is that until you don't make your book public, there is no way of knowing for sure. Mistakes can be corrected. Especially if you are distributing e-books, improved versions of the text can be shared very quickly with your readers. Especially if you are writing on a foreign language, chances of making mistakes that go undetected by automatic checkers are high. However, if the content offsets these issues, readers are still going to be happy.

Finishing a book also includes deciding what the cover will be, the typography, the page size. It can be a lot of fun to tweak these details until it looks exactly as you want. But you should always be practical. Everything is perfectible. It is up to you to decide when the result is satisfactory to make it accessible to others.

Conclusions

Writing a book can be a lot of fun. It can position you in a niche, or it can help you become known in an area outside where you are already known. However, writing a book is not a project that can be tackled without careful thinking. I made the mistake of not thinking ahead of time, writing happened organically. It wasn't only a mistake, it was also a big one. I had to change the contents while writing, at some point, there were some inconsistencies in the finished book.

For the second version of Python for the Lab, I completely re-wrote the chapters, using what I learned from the first version. I also made a conscious plan and stuck to it. I was delighted with the outcome, to the point of sending it to print so I could actually hold it in my hands.

If you want to stay up to date, you can always follow me on Twitter.

From Ph.D. to Independent Software Developer

Aquiles Carattino — Tue, 28 Apr 2020 07:56:20 +0000

Life as a Ph.D., even with its ups and downs, is a relatively comfortable life. There is quite a reasonable job security, a project, colleagues, a supervisor. But doctorates have hard deadlines which can't be extended. It forces us to think once more about what do we want to pursue for the future: academic life, a job in the industry, a sabbatical, an entrepreneurial path.

I knew academic life was not for me because of several reasons, which I hope to be able to put in writing one day. Getting a job in a company was a possibility, and I was actively applying to some roles. However, something that always characterized me was the pulsion of learning and starting new things. Starting something on my own was an aspiration, but without a concrete plan until the very last few months of my contract.

Identifying Options

One of the hardest things to do when you want to start something on your own is to identify the options you have. And once you identify them, isolate your value proposition. Of course, back then, I had no idea what the meaning of the words value proposition was. During my Ph.D. I had developed several skills, some directly related to my work, some in my free time.

I knew how to build websites. It was something I always enjoyed, and over time I kept updating my skills. My first option was to create a service to develop websites for small shops in my area that didn't have an online presence yet. I developed a website, built a couple of sites to have on my portfolio, including my group's website. However, it was a very time-consuming task, with little joy. Customers are always complaining, I have no design skills, and the competition is fierce.

I also knew how to work with Raspberry Pi's and Arduinos. I started thinking about building a consumer product such as an intelligent thermostat, or a private VPN to use at home. I even contacted friends overseas to discuss options. I had the base code for the thermostat, and I could monitor the temperature of my house at different points and stream the data. I also set up a VPN to work at home, with a private cloud accessible from anywhere else in the world. It was fun, but it was just a hobby. Lots of people could do it much better than me.

I knew that I had to keep thinking about what I wanted to do, time was still passing, and the end of my contract was getting closer and closer. I started contacted startups and small companies to try to understand how someone with my profile could fit into their needs.

The Serendipity

One of my former colleagues, Sanli Faez, was starting his group when we met for a beer. He told me he needed a hand developing software for controlling experiments. During my Ph.D., part of what I did was building instrumentation software to do experiments that were not possible to do by hand. I automatize a lot of repetitive tasks, and that allowed me to build statistics much faster, with measurements running overnight or even for several days uninterrupted.

It was just a random talk that made all my choices crystallize. In that talk, I understood that there was something I could do that was a skill hard to find. Programming software for a lab is not just about being a good software developer. It is also about understanding the experiment itself. Very few people have those overlapping skills. I was not only one of those people, but I was also available to start a new project.

Becoming an Expert

When I started developing software for research labs, I started with what I knew. I was behaving like I saw the postdocs act. I was participating in group meetings, offering feedback. I even participated in a conference showing the software I had developed and the new things I had learned over time.

One day, out of the blue, someone wrote to me asking for advice on a program they were writing. I didn't know the person, but they found me on Github, and we had someone in common on LinkedIn. The first time, it was just a coincidence, the second time, it was a pattern that was starting to emerge. Perhaps I was more than just a person developing software.

It is a strange feeling when it hits you. People are seeking your advice on a topic that you were giving for granted. At that time is when you realize that you have become an expert. It is not a choice. It is something that happens when you dedicate day and night to learning about a topic on which very few people are focusing.

Building a Brand

When people start asking for advice, when they seek your opinion or your approval on a topic, you might want to separate yourself from the product. Developing software was allowing me to live, but I didn't want people to know me just because of it. I started to build a brand, Python for the Lab, to separate me from my product. In my long term vision, it would allow me to hire people or to diversify.

I am not sure whether it was a good move or not, but I know people started perceiving what I do as a product. I have heard people talking about my website or my book, without actively realizing I was behind them. I still don't know how to feel about it. I know I should feel proud of having a successful product, but not being recognized for it still feels strange.

Value Proposition and Unique Selling Point

Understanding what a value proposition is, took me a long time. Still, it is something that can be applied to many different situations. In any transaction between people, one person delivers value, and the other acquires it. The perception of the value being exchanged, however, can be very different. In my case, developing software for the lab was a natural task that I learned out of need. I hadn't realized that it could be of any importance for someone else.

It took me a while to find a way of formulating what I was doing. What I realized was that I could save hundreds of hours of a Ph.D. or a postdoc who had to learn what I already knew. Because of my experience, I was much faster at developing solutions than someone starting from scratch. Those hours that the group was saving was my value proposition. Hours are a quantifiable measure. The other aspect is the sheer interest of a researcher in learning or not how to do what I could do. Very few experimentalists want to spend so many hours behind a computer screen.

Once I could formulate the value of what I did, I had to wrap my mind around is why someone would pay me. That is also called the unique selling point. In the case of software for the lab, the answer is straightforward: no one else can do what I do. Better said, no one else is available to be hired for a specific project. In labs, most people are doing a Ph.D. or a postdoc. Very few groups have access to support departments that can develop custom software. There is almost no academic gain in developing just software and not producing results.

I was not looking for scientific achievements. That was my unique selling point setting me apart from people pursuing a career in Academia and who could learn to do what I was doing. My other unique selling point was the brand I was building. Python for the Lab is years ahead from anyone else who would start now. That generates a reputation that helps to justify why me and not someone else.

Summary

Looking in retrospect, I could have saved much time if I would have known what I was looking for. Better said, if I would have had the tools to identify what I was able to do. I spent much time focusing on topics in which I was no expert just because it seemed like the easy path to follow. We all have heard the stories of successful websites and apps. But what I was trying to achieve was a sustainable way of life, one that could give me both joy and enough money to live.

Being able to identify what made me unique was a journey. It is hard to understand at first until you validate the feeling with others. Your life experiences accumulate, and that is what makes you unique. It is not only the technical skills that set you apart but every choice, every experience from your past.

If you want to know more about the things I do, follow me on Twitter