DEV Community: David Przybilla

Reproducibility with Nix : A Beginners guide

David Przybilla — Sun, 24 Sep 2023 01:47:57 +0000

Nix package manager is a powerful tool to ensure reproducibility in Software projects. Nix can help us set local dev environments, it can also unify all the dependencies in local envs, CI and container images.

In this talk I go through:

Why is reproducibility an issue in Software Engineering
How Nix addresses existing problems
The basics of the Nix package manager and the Nix Store
Derivations
The basics of the Nix language
Nix flakes
Example project unifying CI/local env/ Container building using Nix Flakes

Slides and Code samples:
https://github.com/dav009/nixtalk

Example project :
https://github.com/dav009/nixsample

Flyte and Kubeflow

David Przybilla — Wed, 11 Jan 2023 21:27:34 +0000

With this entry I want to give you insights on the following questions:

Problems with the existing orchestration frameworks
How are Kubeflow and Flyte different?
Why is Flyte a tool you might want to study?
Why is it different from existing alternatives?
Is Flyte getting traction?
How difficult is to get started ?

Context: The Machine learning toolbox

Orchestration Frameworks Reality

Defining sequence of steps that are: idempotent, resilient and cacheable, but most importantly reproducible is a huge part of the ML stack.

There are a lot of tools out there. Picking one is difficult. As always there is no silver bullet. It depends on the size of your team, your existing infrastructure and the scale of your problems.

For small ML/DS teams, this usually means taking the tradeoff between:

Option 1: Having a simple orchestration tool with the downside of low scalability, difficulty of provisioning infrastructure, difficulty gluing different services/tools together. Tools like Kedro sit here, they are good for data analysis and Dataframe manipulations. Teams can easily extend Kedro because it is python which most ML practitioners know very well.
Option 2: Having a tool that allow for better scalability, easier infrastructure provision with the overhead of A LOT of complexity. This means a team has to run and manage. a Kubernetes Cluster. On top of that the team has to understand a complex orchestration tool like Kubeflow. Kubeflow works as a Kubernetes operator. Kubernetes operators are usually written in Golang. Any extension or problem that the team faces requires a lot of K8s understanding most likely written in a not very familiar Golang stack for DS/ML engineers.

I want to point out that in both of these options ML practitioners are severely exposed to Infrastructure:

In Option1, ML practitioners will feel in a strong jacket dependent on some team to provide the needed infrastructure. They might be required to learn/manage the infrastructure themselves. I think we can agree that the ML space is already complicated. These kind of setups ask for someone to be good at setting up: security, permissions, infrastructure, manage k8s. And still be up to date as far as ML papers and models.

Another side effect is that projects that fall off the cookie cutter pattern (i.e: a recommender systems) might end up with two pipeline codebases: one for training the model and a different one for serving the model. This is a bomb in terms of bugs and consistency

In Option2. If we consider Kubeflow we start seeing some interesting phenomena. A fine line starts to divide those provisioning infrastructure and the ML practitioners building pipelines for their experiments.

This is good, as we can think of Kubeflow as a platform as a service for ML practitioners. However, here is the catch: Kubeflow pipelines are very near to Infrastructure definitions. If you have coded a Kubeflow pipeline you will be aware that it is mostly a DSL for defining Pods.

ML practitioners exposed to Kubeflow can certainly provision the infrastructure they need easier, but they are still exposed to radioactive levels of Kubernetes details.

Kubeflow issues

Kubeflow is a fantastic tool that addressed a missing Lego piece in the ecosystem when it first emerged. However let me rant with the things that reaaaally bother me about Kubeflow:

Kubeflow is hard to run locally.
1. First time I touched Kubeflow, I had to manually patch yaml files to get it running on my cloud Kubernetes cluster. I was scared to even pitched this to my team. A small team of ML Engineers
2. Kubeflow pipelines are basically “passing strings to container’s entrypoints”. In kubeflow case: these strings are paths to cloud files (i.e: s3://mydata/myfile.csv). These paths get feed into pipeline steps (containers) which are in turn container Entrypoints.

Thats precisely the problem, Kubeflow DaGs carry no type knowledge about the entrypoint and their schema. Everything is a string. Most of these steps are containers that live in another repo, a tiny refactor in how an entrypoint looks like (maybe a bash file changed) and your pipeline is suddenly broken

A Step in a Kubeflow pipeline (a container) needs to know how to transform data. Because Kubeflow just passes paths as strings. This means that your containers need to know :

How to read this file from whatever cloud (maybe local mounted volume)
How to load this file into whatever representation it is used as in the actual task. For example (see pipeline below), this means knowing how to transform parquet files to dataframes back and forth, consistently, in multiple places:

Kubeflow’s DSL is intended to define Pods. It sits too near K8s that ML practitioners need to know many K8s details when writing their tasks

Kubeflow tries to be a lot of things and have integrations with many projects, a lot of the integrations feel rather brittle. When I tried to play with Seldon and Kubeflow I felt scared of shipping something to production.
Many more details and fears regarding Kubeflow are condensed under this blog post “Is Kubeflow dead?” [3]

Flyte ( https://flyte.org/ )

So Flyte is a tool that at first sight might look very similar to kubeflow. You define pipelines, each step of the pipeline is a container/pod. However it’s scope is very different.

Flyte abstracts away the infrastructure for ML practitioners. You can still define your pipelines as an extended DSL for describing Pods. But Flyte provides and encourages different primitives that abstract k8s resources from the ML practitioners.
Typesafety. Even though flyte still runs one container per pipeline step you can get typesafety. Flyte adds types so that you get early failures if your tasks types do not compose.
Plumbing. Remember how in Kubeflow you had to transform your parquet into a Dataframe?. In Flyte lots of common types are already supported, you don’t need to specify how to load a Dataframe.

If your previous task outputs a Dataframe and your current task has as an input a Dataframe, then all the plumbing for these serialisations and deserialisations is done for you in the background.

Running a Flyte cluster locally is super easy. With a single command: flytectl sandbox start

When I tried Flyte for the first time I got a very good impression. I could easily run my pipelines locally (without a cluster), but also setting up a local cluster was super easy.

All of these being said: Flyte is a complex tool. Small teams still need to setup K8S, manage a cluster, manage security, integrate plugins..etc.

Flyte provides a lot of integrations with other tools just like Kubeflow does. That being said not as many as what Kubeflow offers (at least at the time of publishing this Blog entry). However integrations in Flyte do not feel as brittle as the Kubeflow ones.

One of the main reasons is that Flyte was an internal tool built by Flyte, and by the time it got open sourced it was already battle tested.

Personally, as I got really excited I went to take a look at flyte codebase. Flyte codebase is really interesting. The engineer in me believes that Flyte might be a point of integration of many tooling in the future, this is because the foundational layers make it easy to test integrations and get early failures.

Kubeflow is backed by Google

Indeed Kubeflow is backed by google. Big companies like canonical are offering services like “Charmed Kubeflow” (a managed kubeflow).

Flyte however does not fall behind. Lyft, Spotify [4], Gojek [5], Toyota [6]and others are using or switching to Flyte. This should give you enough confidence that the project wont just disappear.

But my team is small, this is still very complex..

I agree it is complex. The creators of Flyte gathered and started union.ai [2]. UnionAi is a startup, it is not clear to me what their product is yet, but it seems it is some sort of service on top of Flyte making the setup and maintenance much less complicated [1].

Union.ai might become an interesting player for Small teams. Why? Because this might abstract complex infrastructure gluing from ML practitioners. With “Charmed Kubeflow” practitioners are exposed to an API that looks like a DSL for defining Pods. Flyte takes that level away, and I suspect union.ai might remove the complicated parts of a Flyte.

Final thoughts

It might be worth to keep an eye on Flyte and see how its ecosystem evolves. Right now Kubeflow has many integrations and that is its strong point
I think putting Kubeflow and Flyte in the same category is a mistake.
Flyte builds new primitives that abstract infrastructure and plumbing for ML practitioners. This is key for empowering ML teams.
If you are curious about Flyte, you can have a taste in a few minutes by following the getting started section at Flyte’s website [7]

[1] UnionML Introduction video
https://www.youtube.com/watch?v=GTPKc1_QSXo

[2] https://www.union.ai/

[3] https://medium.com/mlops-community/is-kubeflow-dead-d82aadba14c0

[4] “Why We Switched Our Data Orchestration Service” https://engineering.atspotify.com/2022/03/why-we-switched-our-data-orchestration-service/

[5] “Adopting Flyte at Gojek” https://www.youtube.com/watch?v=G1ZFeLewUOA

[6] Woven Planet’s Autonomous Vehicle — Data Processing & MLOps at Scale With Flyte https://www.youtube.com/watch?v=OVLZ6-uR_so

[7] Flyte’s getting started https://docs.flyte.org/en/latest/getting_started/index.html

The Dangers of GitHub and the Importance of Code Collaboration as a Protocol: Radicle

David Przybilla — Tue, 10 Jan 2023 00:28:13 +0000

The Dangers of GitHub and the Importance of Code Collaboration as a Protocol: Radicle

In today’s ever growing hostile world, code collaboration platforms like GitHub and Gitlab are a potential hazard. In this Article I want to make the case why we need “Code collaboration as a protocol” and why it would address potential issues as well as improving Developer Experience.

By the end of this post I hope to answer the following questions :

What are the current issues with Github/Gitlab ?
What benefits would a protocol for Code collaboration give to Software developers and OSS?
What is Radicle? How does Radicle fit in this Story?

What is wrong with Github/Gitlab/etc ?

🔭 Scope

GitHub and Gitlab are important tools that help us collaborate on code. Built on top of the Git protocol they let us share code and useful metadata: project issues, community discussions, user profiles, and project history.

Git helps us share and track changes to code, while code hosting platforms provide a way for us to interact as a community, managing things like our online identity and project visibility.

The issues

🚫 Arbitrary access

In the last years we have been experiencing an increasingly hostile and politicised environment for sharing ideas and this has been extended to collaboration tools like Github/Gitlab which have to abide to local legislation.

🚫 Banning individuals from particular countries/nationalities:

GitHub confirms it has blocked developers in Iran, Syria and Crimea

As the world is taking a turn towards conflictive geopolitics , it is worth asking whether you yourself might might be an innocent casualty:

Q: What if my country of birth makes a bad decision that I can’t control, but then I face sanctions just because I was born there?

2. **❌ **Removing specific projects (on legal grounds):

GitHub takes down YouTube video download tools after an RIAA notice
Circle, GitHub Comply With Tornado Cash Sanctions

Q: What if the project I am working on suddenly becomes unpopular or controversial?

🔐 Limiting freedoms to run, copy or extend a piece of software? (GNU philosophy)

Kickstarter shut down the campaign for AI porn group Unstable Diffusion amid changing guidelines

I picked an extreme and controversial example, but you get the gist.

🆔 Identity

My GitHub profile is a combination of my identity and portfolio. It showcases my contributions and interests. It’s important to me personally and also serves as a tool when I’m searching for a new job or when potential employers are looking for me.

As we currently stand our user profiles could be deleted at any time. We deal with all the classic issues of identity in closed social media platforms: you don’t own it.

An interesting point is that not many people sign their commits on Github or Gitlab. Signing commits is as far as I understand the only way to convey identity via the Git protocol.

🏛 ️Lack of Governance for decentralised projects

Github governance goes as far as having roles per project (Admin, contributor). However there is a new class of Software projects: Decentralised projects, Crypto projects, DAOs..
At least in principle these are requiring new tooling for Governance.
Under the Github/Gitlab principles there is no space for these new emerging needs for Governance.

An Open protocol would allow anyone to implement whatever governance mechanisms they need to operate

Code collaboration as a Protocol

If we think of code collaboration as a protocol rather than a platform, anyone can both access and create data that follows that protocol. This means that repositories, code, issues, users, and comments would all be open and organized in a similar way to how we manage code repositories.

Anybody would be able to implement a custom client if need be, or even extend the protocol to include new data. Anybody would be able to provide a specific implementation of the protocol.

Radicle is such a protocol implementation

🌱 Radicle

https://radicle.xyz/

Radicle is an Open source code collaboration Protocol. It keeps git at the core, but it builds the GitHub/Gitlab parts as a protocol. Issues, project discovery, governance , project funding all of these are part of the protocol .

Please bare in mind that in this article I am trying to explain why Radicle is relevant and not the technical minutia of how it operates.

Data in Radicle is stored and distributed via P2P using a protocol that is being implemented called heartwood . You can currently use Radicle even though heartwood is in development.

What does Radicle entails?

Repositories can’t be censored
OSS protocol and clients to access protocol data
Git workflow is kept identical.
Repositories come with extra data : Issues, PRs..
OSS funding via crypto rails

🪪 Identity

Radicle comes battery ready when it comes to identity.
You can’t use Radicle without having a private key first. By default this means your git commits are signed: **Your identity is your private+public key. **Everything (Issues, commits, comments, patches ) under Radicle is signed using your identity (Keys).

Radicle also allows you to connect your identity with an ENS (Ethereum Name Service) domain. Because the data running under the protocol is open, it opens up a lot of possibilities. For example, cryptocurrency projects could use your ENS and contributions to improve your credit score or make you eligible for open-source software donations.

❌ Github/Gitlab: A story of Bad defaults for identity

Git was designed to be a decentralized protocol, which means that there is no aspect of the protocol that verifies a user’s identity against a database of users. One way to prove that you are who you claim to be is by signing commits using a private key. This means that you use your private key to add a digital signature to your commits.

On platforms like GitHub signing your commits is an option and not the default, I believe this is a bad default.

🛠️ Tooling: An Open Ecosystem

As radicle is a protocol and not a platform, anybody is able to write clients or integrations to leverage the data within the protocol.

You don’t like the current client displaying issues or diffs for PRs?
you could fork your existing client and make it your own

Github/Gitlab are closed ecosystems, we rely heavily on whatever is on their roadmap.

With an open protocol, we as a community would have the ability to move in the direction we want, rather than being dependent on corporate roadmaps. This could give rise to new companies that focus on creating the best tools, rather than having just a few giant companies monopolizing network effects and wealth.

💰 OSS Funding

I think that Radicle could also have a big impact on open-source software funding. I saved this topic for last because I was afraid it might scare off people who are not interested in cryptocurrency. However, Radicle is set up in such a way that it could revolutionize the way we fund open-source software using cryptocurrency.

Radicle sets the rails for Crypto currency in the protocol:

Having crypto at the core of OSS fundings means it would help avoid cases where Crowdfunding get censored because of Political stances.
It could introduce new mechanics, for example : issue solving incentives. You want your issue to be resolved sooner? then you can incentivise developers with a bounty.
Radicle Drip protocol is a crypto protocol being built by the Radicle team. It allows people to make donations to a project. It automatically distributes the donations to the contributors in a transparent manner.
This kind of railing might allow for tighter accountability and maybe new ways for Software developers to find jobs

🏛 Decentralised Governance models: DAOs

Currently Github repos are managed via Organizations where there are some Admin and contributor Roles. Under a protocol these mechanism could be abstracted and implemented in different ways. One alternative would be to copy Github mechanism and have equivalent Admins/Contributors role.

Another implementation could provide Governance via a DAO.
Code itself is an asset for an organisation and under a DAO choices made on it could be leveraged via “smart contracts”.

A few escenarios coming out of my head where DAOs could play a role in governance are:

Deciding The roadmap of a product
Deciding what direction to go when there is a fork on the road

Radicle itself is governed via a DAO. I know very little about DAOs but I suspect Radicle itself could be implementing a lot of the tools to let DAOs have ownership of Code artefacts.

⚡ Is Radicle a Crypto project ?

Radicle might be tagged as a crypto project but I find it confusing to do so.

Radicle has a coin which uses for the governance of its development: In other words is backed by a DAO.

The protocol implementation is totally crypto agnostic. However radicle sets the rails for crypto payments and crypto incentives for OSS . Radicle community is obviously crypto friendly and I can imagine there will be a lot of experimentation with crypto technology for OSS incentives.

The punchline here is : you can use Radicle and opt out of its crypto features if you want.

🤔 Final thoughts

You can think of Code collaboration as a protocol the same way that you think of Twitter vs Nostr/Mastodon. The Twitter migration have sparkled a lot of development and enthusiasm as Mastodon/Nostr are built on open protocols.

I did not go into any details on how Radicle works in this article, I wanted to make a case on why it is relevant as of early 2023.

My experience using Radicle’s CLI

I tried using Radicle in December 2022 and it was a really interesting experience. I definitely want to continue using it as the Radicle team works on implementing their vision. As a developer, I’m interested in having control over the identity under which I make contributions. I’m also interested in seeing how future software development projects could be funded, and potentially discovering new ways to get hired as a developer.

As of today Radicle is still in early stages and at best they have created a very promising MVP. If you were to try it at the time that this article is published you would probably find some rough edges. You can think of it as an opportunity to contribute either with your code or ideas.

Troublesome Geopolitics..

It’s a common theme in the news and on YouTube, but it really feels like our world is entering a multipolar stage. In this stage, it seems like geopolitical parties don’t trust each other and try to undermine each other. The software community has always been open to anyone based on their contributions, regardless of factors like age, location, nationality, gender, etc. If we want to continue collaborating together, it seems like we might need a code collaboration protocol. Until a few years ago, I thought that GitHub was good enough, but under the current context, I believe that relying too heavily on these tools is a disaster waiting to happen.

Finally, I wrote this article entirely from a Software Engineering perspective. I am curious about building Software. I know very little about Crypto investments and this blog post is not advising you to buy or sell any type of crypto asset.

If you want to know more about Radicle:

DEV Community: David Przybilla

Reproducibility with Nix : A Beginners guide

Flyte and Kubeflow

Context: The Machine learning toolbox

Orchestration Frameworks Reality

Kubeflow issues

Flyte ( https://flyte.org/ )

Kubeflow is backed by Google

Final thoughts

The Dangers of GitHub and the Importance of Code Collaboration as a Protocol: Radicle

The Dangers of GitHub and the Importance of Code Collaboration as a Protocol: Radicle

What is wrong with Github/Gitlab/etc ?

🔭 Scope

The issues

🚫 Arbitrary access

Q: What if my country of birth makes a bad decision that I can’t control, but then I face sanctions just because I was born there?

Q: What if the project I am working on suddenly becomes unpopular or controversial?

🆔 Identity

🏛 ️Lack of Governance for decentralised projects

Code collaboration as a Protocol

🌱 Radicle

Please bare in mind that in this article I am trying to explain why Radicle is relevant and not the technical minutia of how it operates.

🪪 Identity

❌ Github/Gitlab: A story of Bad defaults for identity

🛠️ Tooling: An Open Ecosystem

💰 OSS Funding

🏛 Decentralised Governance models: DAOs

⚡ Is Radicle a Crypto project ?

🤔 Final thoughts

My experience using Radicle’s CLI

Troublesome Geopolitics..