DEV Community: Sebastian Rath

🐙 ✅ GitHub Actions workflows: How to replace YAML with a visual editor

Sebastian Rath — Wed, 29 Nov 2023 23:36:15 +0000

Hey folks! 👋

In this blog post I would like to talk about GitHub Actions workflows and how I believe they could become much better. Over the last few years I have worked with many repos and I noted how much time I spent on creating and modifying YAML files for GitHub workflows, rather than working on the actual project.

So in true developer fashion, I am working on a solution called Actionforge. Below is a short dive into what I am aiming to solve and how.

What is Actionforge?

Using Actionforge, you can visually create and modify your workflow within a user-friendly editor in VS Code. Each node of your graph represents a GitHub Action. Together, they form what I call an 'Action Graph'. Check out an example here.

Once your action graph is ready, you can commit it to your repository.

How to execute an Action Graph as a GitHub workflow?

To execute an action graph as your workflow, you simply create a simple GitHub workflow that includes a single step with the actionforge/action. Then you point to the Action Graph file you want to run. After this setup, you can edit your graph from the Actionforge UI.

Here is a simple example to execute an Action Graph:



# Beginning of your workflow.yml

jobs:

  run-graph:

    - name: Build and deploy my app

    uses: actionforge/action@0.45.3

    with: 

      graph_file: ./github/workflows/graphs/my-graph.yml

# End

The Linear Limitation

Traditional GitHub workflows follow a linear execution order from top to bottom, which can be restrictive. In contrast, Actionforge's Action Graphs enable more complex and versatile node setups with different flow like if-conditions, for-loops, switches and more.

Executing Action Graphs directly on your GH Runners

Actionforge does not depend on external cloud services. actionforge/action serves as an interpreter, converting your visually designed graph into a sequence of GitHub Action nodes that execute as usual on your designated GitHub runners.

All repositories will be made public in December. I would love to hear your story. Share your GitHub Actions workflow challenges. Check out Actionforge.

Thanks for reading ❤️

A deep dive into file locking with NodeJS

Sebastian Rath — Mon, 07 Jun 2021 14:21:15 +0000

I developed a version control software for graphic designers and 2D/3D artists called Snowtrack in Angular and Electron. In this blog post, I will cover some technical challenges about file locks which I faced during the development of Snowtrack.

What is Snowtrack?

Snowtrack is an intuitive, easy-to-use, and super-fast version control software for graphic projects. Its purpose is to make version control accessible to graphic designers and 2D/3D artists with a non-technical workflow.

To get a better understanding of Snowtracks user interface check out the following screenshot:

What I used to build it

For the UI application I used a combination of Angular and Electron. The underlying version control engine is called SnowFS, an open-source project I developed as a fast and simple alternative to Git and Git LFS. Feel free to check it out on GitHub. A few months ago I wrote a blog post about it here on dev.to.

Technical challenge no. 1

Graphic projects can differ in size tremendously. From a single Photoshop file up to a 50 GB file set of 3D scenes, textures, and assets. These project types come with their own set of problems. In the following I want to clear up some misconceptions about the topic around file locking.

File Locking

Take a look at the code snippet below.



// Process 1
fd = fs.openSync("~/foo", "w");

// Process 2
fd = fs.openSync("~/foo", "w");

Imagine more than one process wants to open the same file at the same time. What do you think will happen?

Answer: It depends on the OS and if you're the maintainer of all processes.

When you call fs.openSync NodeJS will forward the call behind the scenes to an OS function as you can see from this C code



static ssize_t uv__fs_open(uv_fs_t* req) {
  return open(req->path, req->flags | O_CLOEXEC, req->mode);
}

The function open(..) is an OS function and available in all operating systems. But the internals of this function differ between Windows, Linux and macOS so I will cover them separately.

macOS/Linux

Technically, neither macOS nor Linux have true file-locking mechanisms. Although you can read or write-lock a file using a function called fcntl, only programs which use this function regard and respect the file lock. This means, any other process which doesn't use fcntl and directly wants to open a file can acquire a file handle and manipulate the content as long as the file permissions allow it. What a bummer.

That's why file locking on macOS and Linux is also called "advisory file locking".

Windows

Windows is more complicated in that matter. Windows offers two functions to open a file. Either through the Windows API function called CreateFile (yes, that's really the name to open files),...

...or through open(..). But the open(..) function on Windows is a POSIX extension and uses CreateFile internally as well.

As we've seen above NodeJS uses open(..), but since we know that this is just a wrapper for CreateFile, let's check out that function:



// The low-level open function of Windows.
HANDLE CreateFile(
  LPCSTR                lpFileName,
  DWORD                 dwDesiredAccess,
  DWORD                 dwShareMode,
  LPSECURITY_ATTRIBUTES lpSecurityAttributes,
  DWORD                 dwCreationDisposition,
  DWORD                 dwFlagsAndAttributes,
  HANDLE                hTemplateFile
);

CreateFile has a parameter called dwShareMode. A file that is opened with dwShareMode=0 cannot be opened again until its handle has been closed.

So if you use open(..) on a file that was already open by another process with CreateFile(…, dwShareMode=0) you receive this error message:

The process cannot access the file because it is being used by another process

On the other hand, if you use fs.openSync in NodeJS, or open(..) in C/C++, to open a file that hasn't been opened yet, you cannot prevent another application from modifying it*.

* Unless you you use file permissions as a workaround, but that’s not really a file lock.

To prove this, you will see that our fs.openSync call executes CreateFile with the read/write shared flags to comply with the POSIX standard.

This means on Windows you cannot prevent another application from opening and modifying your file if you don't use CreateFile.

What does this have to do with Snowtrack?

Imagine a user saving a big file in a graphic application and while the file is still being written to disk, the user attempts to commit the file change. How does Snowtrack deal with this?

As we learned, open(..) has no file locking and most applications don't even follow the file protocol and Snowtrack cannot control how Photoshop, Blender, and co. open and write their files.

This means the only reliable chance of detecting if a file is still being written by another process is to check prior to a commit if any process on the system has a write handle on that file.

On Windows, I solved this with a custom helper process and and the Windows API of Restart Manager which is mainly used for installers to ensure the files it is about to replace are not open anymore.
On MacOS I invoke the system process /usr/sbin/lsof (list open files) with an inclusion of the working-directory to speed up the execution of this command.

What else?

The development of Snowtrack came with countless technical challenges and I would be happy to share more insights.

File locking, Electron/Angular race conditions, I/O saturation, build server, update mechanisms, edge cases, .. with this project I touched many subjects and I would be happy to write a follow-up blog post if you are interested. Let me know in the comments below.

If you want to support SnowFS, Snowtrack or me then feel free to join me on Twitter.

Thanks for reading :-)

TLDR

Don't get me started on file-locking.

Addendum: What about the "File In Use" dialog in Windows?

If you are a Windows user you might have seen this error message before:

Windows, or rather NTFS, behaves very different compared to other file systems like HFS+, APFS, ext3, ...

There is no equivalent to inodes in NTFS and therefore no garbage collection deletes the file if the last file handle to an already deleted file is closed. The File in Use dialog only indicates, that if any process has a file handle to a given file (no matter how it got opened), it cannot be renamed, moved, or deleted. That does not imply a file lock on the file content.

SnowFS ❄️ Let's bring version control to graphic projects

Sebastian Rath — Sun, 14 Mar 2021 19:32:03 +0000

Introduction

Welcome to my first article here on Dev.to. In this article, I will be sharing an overview of an open-source project I kicked off to explore ideas and solutions on how to bring version control to graphic projects.

The open-source project is called SnowFS and it is a lightweight command-line application and library for binary files with a focus on graphic files.

But why?

I recently started developing an UI application that brings local version control and artists/designers together. Given the effort of the Git community to support larger files, I tried Git+LFS for my first prototype. It seemed, after several successful tests I've overcome the inner demons that were screaming Don't even try to use Git with such big files. Means Git with LFS is a nice solution to tackle such needs.

Unfortunately, the devil is in the detail, and some features beneficial for the software development process can be quite counterintuitive for other project types. For example, the hash integrity of Git is great to ensure the integrity of a software repository, but not really needed for a graphic project. For such project types, the requirement to delete individual files or removing entire commits without rewriting the history all the time is more important.

Basically, I needed Git, just more lightweight, support for big files, and a fraction of the functionality. And that's how SnowFS was born. It supports...

...deleting of entire commits or individual files
...instant snapshots
...instant rollback
...super-fast-detection of large modified files
...block-cloning and copy-on-write supports for APFS, ReFS, and Btrfs
...and more.

How does SnowFS perform?

Below is a comparison between Git and SnowFS on a Macbook Pro (2020) with an APFS formatted SSD to check-in, delete and restore a 4GB Photoshop File.

$ git lfs track *.psd
$ git add texture.psd   # 20164ms
$ snow add texture.psd  # 4596ms <-- Not bad
$ git rm texture.psd    # 575ms
$ snow rm texture.psd   # 111ms <-- Also ok
$ git checkout HEAD~1   # 9739ms
$ snow checkout HEAD~1  # 1ms <-- Yeah!

Where to go from here?

SnowFS is a great lightweight and flexible project to explore and experiment with new ideas for version control. If you are interested in the project, we are looking for your expertise to share and learn!

In that sense, Happy versioning!