DEV Community: Florian Polster

How to install only the CPU version of pytorch in pdm

Florian Polster — Thu, 03 Jul 2025 17:13:32 +0000

I've been chasing how to do this for days and Google couldn't help me. So here's me hoping I'm helping the next poor soul.

In pyproject.toml:

dependencies = [
    "torch>=2.7.1",
]

[[tool.pdm.source]]
type = "index"
url = "https://download.pytorch.org/whl/cpu/"
name = "torch"

Long-running jobs in Temporal.io

Florian Polster — Mon, 17 Mar 2025 17:00:00 +0000

TL;DR

A long-running job should be implemented as one long-running activity that
uses heartbeat for resumption.

What I needed to do

There was a data-sync backfilling to be done where we backfill data from one
service's database into another one. It was about 5.5M DB entities that needed
to be copied, so we expected the total running time to be some number of hours.

We wanted to write a job with some kind of progress record, i.e. a "cursor".
Such that, should the job fail at any point, it could, on retry, continue where
it left off.

First instinct: Parameterized activity

After reading the Temporal docs, my first instinct was to implement one workflow
with one parameterized activity which the workflow would call in a loop.

This idea was dismissed after talking to my colleagues. Because each activity
invocation results in events being added to the Temporal Events. Temporal works
under the assumption that there will be few events per workflow, not tens of
thousands. We didn't want to risk exhausting Temporal's disk space.

The good solution: One long-running Activity with Heartbeats

I'll just show you some code here.

At the beginning of the Activity we use GetHeartbeatDetails to get the cursor
that was recorded by a previous, unsuccessful activity execution. After each
batch we record the cursor via RecordHeartbeat.

You'll want to do some info logging throughout the activity execution. Errors
don't need to be logged, just return them to the runtime, it will surface them.

As you can see, the batch size is configurable. We ran it with a size of 100 and
that worked well.

package workflows

import (
    "context"
    "fmt"

    "go.temporal.io/sdk/activity"
)

type BackfillingActivities struct {
    BatchSize int
}

type Heartbeat struct {
    Cursor string
}

func (a *BackfillingActivities) CommentsBackfillingActivity(ctx context.Context) error {
    cursor := ""
    if activity.HasHeartbeatDetails(ctx) {
        var heartbeat Heartbeat
        if err := activity.GetHeartbeatDetails(ctx, &heartbeat); err != nil {
            return fmt.Errorf("error getting heartbeat: %w", err)
        }
        cursor = heartbeat.Cursor
    }

    var err error
    for {
        cursor, err = a.processBatch(ctx, cursor, a.BatchSize)
        if err != nil {
            return err
        }
        if cursor == "" {
            break
        } else {
            activity.RecordHeartbeat(ctx, Heartbeat{cursor})
        }
    }
    return nil
}

// processBatch processes one batch starting from cursor and returns the next cursor or an error.
func (a BackfillingActivities) processBatch(ctx context.Context, cursor string, batchSize int) (string, error) {
    logger := activity.GetLogger(ctx)
    logger.Info(fmt.Sprintf("Starting processing batch with cursor \"%s\"", cursor)

    //-> batch := SELECT FROM WHERE primary_key > {cursor} ORDER BY primary_key ASC LIMIT {batchSize}

    logger.Info(fmt.Sprintf("Found %d items", len(batch))

    //-> Transform into target data structure

    //-> Load into target DB

    if len(batch) < batchSize {
        return "", nil
    }
    lastItem := batch[len(batch)-1]
    return lastItem.Id, nil
}

JSON in ClickHouse

Florian Polster — Sat, 14 Jan 2023 10:12:38 +0000

I just watched this talk about storing JSON in ClickHouse. It's quite insightful. My takeaways:

There are performance-trimmed json-value-extraction functions that avoid parsing the entire blob in favor of performance. JSON in String columns compresses pretty well.
There're functions to convert simple JSON to maps and when you store the same data from the talk's example as a map instead of JSON, queries are 4-5x faster.
You can insert data into a table with engine=null. Such a table doesn't store any data but it can be the source of a Materialized View.
It's pretty common for users to store the JSON in one column and then extract specific values into columns next to that. That will already boost performance if you're smart about it. You can further enhance performance by defining Data-Skipping Indexes on those columns.
All the approaches explored in the talk are documented here.
There's a beta feature for the future of JSON data in ClickHouse with many under the hood optimizations (essentially, inserted data is schema-inferred and clickhouse will automatically create columns for all fields (even nested) and it will even create new columns as json object with new fields flow in).

How to host internal static websites protected by single-sign on (Oauth2, OpenID, or SAML) for free

Florian Polster — Sat, 21 Nov 2020 14:28:04 +0000

In this brief tutorial I will show you how to create a log-in protected static website using the Google Cloud Platform. We'll use App Engine to host the static (or dynamic if you want) website and we will add a log-in shield using Google's Identity-Aware Proxy. At Staffbase we use the G Suite so I'll set IAP up to only allow access to the website to people from our GSuite domain.

Prerequisites

The eventual website will cause no costs as long as you don't have huge amounts of traffic. Nevertheless, even to create just a free account, Google requires you to provide a credit card number. They do this avoid fraud and misuse. So that's something to keep in mind - you need a credit card to begin.

You also need to install the Google Cloud SDK since you need the gcloud CLI tool which comes with it.

Lastly, you need a GCP project, so create one if you haven't already.

Setting up App Engine

The only thing you need to turn your code into an app is adding an app.yml file. You can reuse my app.yml without having to make any modifications.

My app.yml

runtime: python38

handlers:
  # resolves example.com/blog/ to example.com/blog/index.html
  - url: /(.+)/
    static_files: static/\1/index.html
    upload: static/(.+)/index.html

  # resolves example.com to example.com/index.html
  - url: /
    static_files: static/index.html
    upload: static/index.html

  # resolves example.com/blog to example.com/blog/index.html
  - url: /([^\.]+)([^/])
    static_files: static/\1\2/index.html
    upload: static/(.+)

  - url: /(.+)
    static_files: static/\1
    upload: static/(.+)

Credits for the above YAML: https://github.com/mattgartner/appengine-static-sites. This configuration assumes that you have all your static files in a directory called static.

Don't worry about the fact that Python is configured as the runtime. Specifying a runtime is required and while all runtimes are capable of serving static files there is no dedicated runtime for static sites.

You don't need to understand the handler definitions. They might look complicated but in essence all they do is making sure that urls like domain.com/url/path and domain.com/url/path/ get translated to domain.com/url/path/index.html.

There is also a guide from Google on this. Once you have an app.yml file you only need to run gcloud app deploy --project=<project-id>

Configuring Identity-Aware Proxy

Besides Google Accounts, you can use a wide range of additional identity providers, such as OAuth, SAML, and OIDC.

Configuring IAP is very easy. I had no problems following the official guide. There is this role called IAP-secured Web App User that you have to assign to anyone you want to be able to see the pages. I assigned this role to the entire staffbase.com G Suite domain but theoretically you can also assign it to individual Google accounts.

The Reasons Why People Do Mob and Pair Programming Even Though It's Less Effective Than Working Alone

Florian Polster — Sun, 22 Mar 2020 13:50:24 +0000

Explaining how mob programming works is not within the scope of this article. You can find that out elsewhere. You might know pair programming. That's a special case of mob programming where mob.size() == 2 so to speak.

I read an interesting article (in german) yesterday and it made me realize something that I want to share with you. It's the justification of mob programming which the article got down really well IMO. The benefits weren't so clear to me before.

TL;DR
Yes, in terms of throughput mob programming is less efficient than working independently and you should keep that in mind. However, mob programming enables faster time-to-market for single tasks/features. It's also perfect to spread knowledge among the team and that has valuable long-term benefits.

If you've ever wondered 'why would anyone do mob programming at work, isn't it inefficient to have multiple people work on the same thing instead of everyone working in parallel?' - You're right. If every person is working on separate tasks you get more work done. That modus operandi does yield higher throughput of work items.

What mob programming optimizes for is time-to-market -- getting one specific task done completely in the shortest amount of time possible. Here's how:

If one person were to work on the task alone they might come to a point where they need some information from a co-worker. Getting that question answered will introduce some latency. In mob programming that co-worker is right there and can respond immediately.
Where one person alone might have to look something up or google something. In mob programming there might be someone able to help instantly.
Mob programmed code is already reviewed! If people are working separately, they submit their code for review resulting in more latency especially if further changes are requested from the reviewer. We do code reviews because individuals tend to suffer from tunnel vision, forget to write tests/documentation or are too lazy to do quality refactorings. Basically the problem is that one person can only look at the problem from one angle. With mob programming all these problems disappear as multiple people provide multiple angles to look at the problem.

Another benefit is knowledge sharing. While working on the same problem, knowledge about that particular domain is spread across the team. The first result of this is that in the future team members become more independent when working alone because less questions pop up. The second result is that everyone in the team can respond to requests from the stakeholders. And if any of the members go on vacation no hand-overs are necessary.

The last upside is that by spending time together the team grows together possibly with shared moments of accomplishment which is probably the most effective bonding activity you can do.

Curiously, if you mix in mob programming from time to time the team becomes more effective in working independently.

If you have programmed in a mob before please let me know how it went in the comments!

The authors of the article I mentioned in the beginning have made a website ... in case you want to check it out.

How to work with git - An overview of git workflows

Florian Polster — Sun, 05 Jan 2020 19:43:34 +0000

If you're here looking for a recommendation on what workflow to use, I strongly support the recommendation Microsoft hands out to their customers.

Git workflows differ from dev team to dev team and from git server to git server (Github, Gitlab, etc.). Git as a technology is so solid and flexible that people have found multiple viable workflows. Here is an extensive overview. It is assumed that you know git and feature branches.

I plan on updating this post based on your feedback. So please, if you think something's wrong or missing, or if a link becomes out-dated leave a comment or contact me!

Another guide about the same topic: https://www.codingblocks.net/podcast/comparing-git-workflows/

Forking vs Centralized Workflow

The centralized workflow - i.e. all developers on a team have read and write access to a shared repo.

This is especially common for professional teams. Team members fetch from the central repo, work in topic branches and push their branches to the central repo. Code is reviewed by other devs in Pull Requests. Accepted PRs merge the branch into master.

The forking workflow - i.e. only core developers can change the repo.

This is the most common form of collaboration for open source projects. The goal of this workflow is that the publicly visible repository sustains a high quality. Only trusted people (i.e. the maintainers of a project) can modify it. The way non-maintainers contribute to such projects is by a process called forking. It clones the entire git repo to your personal user account.

Say, if I want to make a contribution to some open source project like the programming language Scala. I would go to the official repo github.com/scala/scala and click on the Fork button. That would create a copy of that repo under github.com/pofl/scala. I would have full control over that clone. I can push to it how ever I want.

When I want to submit a contribution to the official repo, I will first push my changes to my fork. Then I can create a Pull Request in the upstream project. Such a PR is essentially a request to merge a branch in my repo into the upstream repo. It's called Pull Request instead of Merge Request (which it is called in GitLab btw) simply because 'pull' is the gitnically correct term for fetching a branch from a remote and merging it into HEAD.

The day-to-day workflow is:

Fork the repo.
Clone your fork to your computer (origin).
Add the official repo as a remote (upstream).
Fetch upstream.
Create a new branch from upstream/master.
When you're done, push it to origin/topic-branch.
Then you can create a PR in the upstream project where you can request that your topic-branch be merged as if it was a branch in the upstream repo.
Resolve all requested changes that the maintainers pose.
The maintainers accept the PR and the branch is pulled form your fork into upstream.

GitHub's guide on forking

Git history hygiene

Developers strive for clean code. Clean and consistent code is easier and faster to work with. To accomplish that teams author a code style guide and enforce it. The addition of Git as a tool to your development setup brings with it a whole new dimension of things that need to be kept clean. This section explains these things.

If you are following the developer community on the internet you'll come across a few articles and blog posts talking about the git history. Many people have string opinions about what a clean history is. Curiously, when people say Git history what they actually mean most of the time is the shape of the commit graph of master and other permanent branches. We'll cover that in this section. What we'll also cover is under what conditions a commit is 'clean'.

Linearity of Git history

When you merge a branch into another, usually a merge commit is created but not always. Git defaults to not create a merge commit if it's not necessary (this is called fast-forward merging). A merge is what happens anytime a code contribution gets accepted into master, and that's the point where some people go religious. Luckily, most people just accept the default behavior and live with it.

Linear history

There is one philosophical tribe of people who find that the commit graph should be linear. They want to be able to have a look at the Git history and immediately be able to see what has been happening and who's been doing what. With a linear history all you see is the unnoisy master and the feature branches that are in progress. A linear history is achieved by rebasing the feature branch onto master before merging/pulling and enforcing fast-forward merges eliminating merge commits.

The downside of this approach is that rebasing is quite an overhead as it involves more steps than merging. It also becomes limiting when many people work on the same repo, as there will be situations where devs will have to re-rebase because master changed before the PR got reviewed. The advantage is that you really do get a cleaner look at you history.

Here is a link that discusses and explains this philosphy further. Here you can see the Git history of a project following the this philosophy.

Maximum historic information

The other extreme is one where people disable fast-forward merges and enforce the creation of merge commits. The philosophy behind this is that the history is a source of information and the fact that code was created on feature branches would be denied when you create a fast-forward merge. "A linear history is a lie" is their saying.

Here is an article comparing the two approaches with favoring the latter. Here's a project following the second approach. Here is (a section of a) talk proclaiming the second philosophy.

Squash merge

This is another approach how to keep the history linear. This article explains everything.

TL;DR a squash merge is a merge where the entire merge source branch is condensed to a single commit which is created on top of the merge target branch without creating a merge commit. Some teams like to do this because it keeps the Git history slim.

Permissions

Many Git servers allow restricting permissions to some extent. Here are a few use cases:

Suppose you have a branch called live or prod which, on every new commit, gets automatically deployed into the live production environment that is used by your users. You probably don't want everyone to be able to make changes on these branches.
Let's say you want to maintain a very high code quality in master, it should not be possible for people to bypass the code quality assurance measurements.

So in both examples what you want to achieve is, basically, to restrict write permissions on certain branches. Here are a few ways some Git hosting solutions allow you to do this:

Pretty much every one of them allows you to protect branches against changes like pushing or merging. You are able to grant this permission to a select set of users.
Pretty much every one of them allows you to make it a requirement that some CI/CD pipeline succeeds before PRs can be acctepted.
Some of them enable you to make it so that some branches can only be modified by accepted PRs. This effectively makes code review mandatory and is therefore very cool. It is also sometimes possible to require the approval of PRs by specific persons.

Here are some links that go into more detail: GitLab on the benefits of branch protection, GitLab documentation of allowing branches to be modified only through PRs (Merge Requests in the GitLab language) and GitHub on their branch protection features

Clean commits

First, a link: Anatomy of a perfect Pull Request

Many teams have rules that determine under which conditions the commit history is 'clean'. Firstly many teams have rules for what the commit message should look like. One very popular set of rules is this. Another write-up of rules.

Besides those I've heard of projects which mandate that the commit title starts with the name of the module or the file in which the changes were made (in such a case there is usually a not so restrictive rule on title length). Some teams also mandate that there must always be a body.

One very popular rule is: for every commit compilation has to succeed and the code needs to be free of errors (minimum: tests have to pass). The reason for this is that if somehow at one point a regression emerges, you can easily find out what commit caused it with git bisect (Link1, Link2).

The reason why many teams have rules to enforce a clean history is to simplify code reviews. When all commit messages look the same, they can be processed faster. When all commits have a body detailing the changes the reviewer can grasp the context faster. It's easier for a reviewer to review three commits in succession as opposed to one commit containing three different logical changes.

We all do however make that "WIP" commit from time to time. We do commits that don't build, that have an ugly commit message etc. With git you can always go back and clean your history up with git rebase --interactive. See this guide to learn how to manipulate the Git history.

PS: Linus Torvalds on clean history

Branching strategies

In this section I'll discuss guidelines for what branches to create and when to create them.

Release handling

How you handle releases depends on whether you have to support multiple software releases. If you always only release one version and then develop the next version, then simply adding tags to the master branch is sufficient.

If you have to support multiple releases, create a release branch for every release. On this branch you integrate bugfixes. You also prepare and release updates from it. Even if you only support one release release branches can be made great use of. They are an excellent way to perform a soft code freeze where on the release branch the code doesn't change during bug-hunting while on master new code can be added at full speed.

Read this great guide on release branching.

Sometimes you detect a bug that affects multiple releases or your master. There are different approaches how to tackle this situation.

Git Flow recommends creating a branch from the release and merging the fixing commits into every affected release and the develop branch.
You can implement the fix for every release manually
When you have fixed the issue in one branch you can cherry-pick the commits onto the other branches.

Popular Branching Strategies

There are a few branching strategy guides out there. Git Flow was one of the first written detailed branching guidelines on the internet and gained a lot of popularity because of that. It's an interesting read, but in my humble opinion it's an overkill for a branching strategy. Here are two reddit threads [1] [2] where this strategy is discussed and where the downsides of it are layed out.

Another very popular workflow is Trunk-based development (TBD). The website has many great articles about the details of the workflow. Some parts of it can be applied to other workflows. Microsoft has published a recommendation for Git workflows that is very very similar to TBD while being a much shorter read. That article and TBD in general is my personal recommendation to be a basis for your workflow.

Once in a while some company invents a new Git workflow, gives it a fancy name and releases it as an article. Here are those that I'm aware of.

Release Flow by one team at Microsoft
GitHub Flow
GitLab Flow

However, it's not very satisfying to read these articles because you'll realize they're mostly the same as TBD. Which again speaks for TBD IMO.

Long-lived topic branches

Topic branches should be short-lived. This means that ideally only one or a few days should pass between branching and merging back. If branches live longer, problems arise from the fact that potentially the code in master changes, resulting in a divergence between the current master codebase and the version on which you created your branch. Merging/rebasing becomes very complicated.

Ways to prevent long-lived branches:
- Reduce the scope of the branch/issue (try to split the problem you are trying to solve into smaller problems)
- Use feature toggles [1] [2] ... TL;DR: Add some form of toggle (an if statement most of the time) to dis- and enable the new code. With this approach the new code resides next to the old and both evolve close together.
- Use branch by abstraction, a variant of feature toggles where you first introduce a new abstraction that wraps the part of the code you want to change. Then you implement the new code as an implementation of that abstraction while the rest of the team uses the old implementation

E-Mail based workflow (Linux kernel)

The Linux kernel is the birht place of Git. The lead developer of Linux, Linus Torvalds, made Git because he needed a source code management system that suits the development workflow of the kernel. That workflow is centered around mailing lists. Code changes ("patches") are shared via mailings lists and there is no central Git server like GitHub. Here is an overview of distributed Git workflows. Here is a guide explaining how to use Git with E-Mails.

So far for this workflow guide. Thanks for reading and please give me feedback :)

My vision for the missing LaTeX dependency manager

Florian Polster — Mon, 29 Apr 2019 06:43:08 +0000

Abstract

In this article I claim that the LaTeX developer experience can be improved by learning and copying from programming languages. Especially automated package management (including automatic package acquisition based on \usepackage directives) would be a big gain.

My user experience with LaTeX

TL;DR

I shouldn't have to install the packages that my document requires for compilation one by one manuallay. This should be automated.
The Tex Live installer asks too many questions.
The LaTeX compiler is not user-friendly

I'm a Linux user and I use LaTeX for university assignments. I acquired texlive through my distro's package manager. Installing Texlive was a cumulative download of about 2GB. I got a ready-made LaTeX template from a fellow student. I ran pdflatex. It found that some .sty file could not be found and entered some weird interactive mode. I figured out how to prevent this weird interactive mode for good so that the compiler just terminates upon error. I then had to manually install all packages that provide all the .sty files that were not present on my system.

These two are the first two problems I want to point out. The user should never be confronted with this strange interactive mode of the compiler. The worst part of the interactive mode is that it is so hard to exit (is it Ctrl+C, q, quit or exit?). Also the retrieval of packages should be automated. Installing them manually is tedious. An alternative is installing all texlive-* packages, so that none would have to be insalled manually. But that is a download of over 6GB...

Then I wanted to work on my thesis on university lab computer. It did not have a package installed thorugh the distribution that was necessary to build my thesis. Lacking admin privileges I could not install them regularly. What I could have done was download the package manually from CTAN and put it into my source path.

At that time I was not aware that texlive could be installed manually, independently from the system package manager. I could have installed texlive in my home directory and could have made use of tlmgr to install all packages. That would have worked. However, having recently tried this out, manually installing texlive is also tedious: The installer gives the user many choices to make. Installation location, the size of the set of packages to install and many other less relevant things. Also, after installation, all programs (compilers, tlmgr etc) need to be added to the PATH environment variable so as to make it exectuable from the command line.

Dependency management in programming languages

Programmers prefer using libraries and frameworks over writing everything from scratch, because this approach has many obvious advantages. Relying on externally developed code has some consequences: The foreign code needs to be locally available to (depending on the type of language) build or run the software. Not only that, but the foreign code may rely on other libraries to perform its tasks. Manually downloading, integrating and setting up external code and its dependencies is tedious. And the process needs to be repeated for every new PC that is set up to build/run the software and for every version update of the dependencies.

Long story short: There exist dependency management tools for most programming languages. As an example let me describe what the JavaScript dependency manager and build tool npm does for the user.

NPM

There is an online registry/repository where libraries/frameworks (the more general term being 'packages') are published. http://npmjs.com/
There is a CLI program npm to find, download and update packages from that registry.
If you use npm you put a file package.json in the root folder of your code project. This file provides the following information:
- It has a list of all packages that the code project depends on. You also specify the versions of the packages.
- It has a list of (build) scripts. These consist of a script name and a CLI command.
- And, less importantly, this file contains all the metadata of your project in case you want to publish the project as an npm package. As such you specify title, authors, current version etc.

When somebody new starts working on a project all they need to do is get the project code from version control (only the internal code and the package.json file is in version control - not the external dependencies) and then inside the root directory run npm install. This reads the dependency list in package.json, downloads them all and all their dependencies from the registry and puts them into a directory in the project root.

What LaTeX should learn and adopt from this

LaTeX is in a good position to easily establish its own tooling for automated dependency management. With CTAN we already have an online registry of packages. Also, LaTeX already has the means for describing the dependencies of a document: the \usepackage directives in the preamble.

My vision for a LaTeX dependency manager is a little program that reads the main LaTeX source file to install all required packages automatically. Installing can be done in any of the following ways:

If texlive is installed on the system then use the executables from that install. Download packages that are not present but required by the document directly from CTAN to ...
a. a directory in the project
b. a project-agnostic directory located in the user's home directory
Download and install TeX Live. Offer express installation, i.e. do not ask for install location etc., use sane defaults. Install no LaTeX packages except the absolut bare minimum. After that, use tlmgr to install all packages that are needed for the document. No more. There are multiple ways this can be done:

a. npm style: install in the project directory.
b. Install in the user's home directory. This will be enough for many users because oftentimes a computer is used by only one person anyway.
c. Install globally for all users. This will require fiddling with admin privileges.
Using the system package manager. I think it's going to be doable to find a way to map package names to the names of the package in the system package manager.

These options are ordered by increasing difficulty.

Build management

Revisiting NPM build scripts: For a typical fronted JavaScript project there are two very common scripts defined in package.json:
- A dev script: This typically serves the frontend by means of a locally running webserver to the browser. It also watches the files in the project for changes and automatically rebuilds the code and reloads the served page.
- A build script: This builds the project for deployment.

Scripts are run by invoking npm run <scriptname> from the command line.

'Build tool' is a common term for programs that automate the build process. I think with LaTeX many people use Makefile. The build process of a LaTeX document typically comprises bibliography processing and up to three LaTeX compilation runs. Additionally you might want to perform some actions to keep the working directory clean from compilation byproducts. Maybe you're writing the body of the document in Markdown or some other markup and need to perform a markdown-to-LaTeX-translation step. Maybe there is some data that has to be plotted during build.

TL;DR build automation is pretty much the norm. Some thought should be given to the question whether there are any benefits to be gained from including a build scripting feature in a possible LaTeX dependency manager, thereby turning it into a full-blown build tool like sbt and Maven. Maybe also just Makefile does the job well enough. However, I would love to save TeX Live users from having to find out about nonstop mode of the TeX compiler. Also, an authoring mode with automatic recompilation upon file change would be neat.