DEV Community: Cheng Shao

Gitpod notes and thoughts

Cheng Shao — Sat, 30 Jul 2022 04:48:12 +0000

A few months ago, a colleague reposted a Gitpod blog post in my employer's slack, mentioning that if you're a member of some reputable GitHub organization, you get a plan with unlimited hours. Thank you, the past me who spent a weekend packaging an app for nixpkgs!

I knew these kinds of cloud IDEs have been around for some time, but that message inspired me to give it a bit of try, who can resist free lunch after all? Well, I'm now using it on a daily basis, so this post will record some notes and thoughts along the way. The content is not quite well structured, but I hope they can provide a bit of insight for other Gitpod users as well.

Hacking GHC

The first Gitpod use case I tried is hacking GHC. And that's quite a roadbump because GHC is hosted on their own GitLab instance!

Yes, you probably can fiddle with tokens and such, to make Gitpod work with self-hosted GitLab. It's mentioned in Gitpod docs, I haven't tried, since there's another problem ahead:

Gitpod requires you to check in .gitpod.yml into the repo you're working with. That's reasonable for most single-repo use cases, but I don't have any mental energy to upstream a Gitpod config, nor do I want to pollute my GHC working branches with this config file.

So I made a dedicated GitHub repo for hacking GHC. Gitpod allows you to specify scripts to run during workspace initialization, so I can just clone GHC to /workspace and open it later, profit.

Git authentication with SSH

All looks good except when I need to push. For GitHub repos I own, pushes to https remotes should work out of the box, but well, this doesn't work for GHC's self-hosted GitLab.

This hasn't been a problem when I used my employer's remote build machine, since I can always enable SSH agent forwarding, so I can do the pushes on that machine. Gitpod allows you to SSH into an active workspace, but honestly I don't want to open a separate terminal window just for git operations, I prefer to stay inside that browser tab all the time.

So. Given Gitpod allows specifying secret environment variables, I generated a passwordless SSH private key and encoded it as a secret. The gitpod-ghc init script will start an ssh-agent and automatically add that key, so future git operations will work out of the box.

Yes, doesn't sound like a good security story, other than permuting the key frequently and limiting the key's scope, I still need to trust Gitpod not to leak my secret, no less than they trust me not to abuse their machines to run crypto miners.

What's persisted and what's not

There are multiple ingredients in making a Gitpod workspace:

The base Dockerfile or a Docker Hub image. Use whatever you like, but given the Gitpod runtime will yolo a bunch of Linux binaries at runtime, it better be a typical glibc-based FHS distro.
The .gitpod.yml config, which allows you to specify init scripts.

Having two places to write scripts may seem redundant, but the .gitpod.yml scripts have access to the /workplace directory. Across restarts of the same Gitpod workspace, only changes in /workplace is persisted!

It's quite an annoyance since for a long-lived workspace, you often accumulate implicit state (e.g. language-specific caches) not only in your project directory, but also in the home directory.

Luckily, there's a limited workaround for that: bind mounts. In gitpod-ghc config, I bind mount a lot of home directory contents from /workplace instead, things like .config, .cache, .cabal. Even the entire Nix store!

It's a pity I can't bind mount the entire home directory though, likely because of busy resources during workplace initialization. And of course, all system package manager invocations are forgotten after workplace restart.

To think of it in a more positive way, Gitpod is enforcing the erase your darlings philosophy. It doesn't strictly prohibit state, it forces you to make your state explicit.

Prebuilds

Gitpod prebuilds is a CI service that runs on the same infra to power the workspaces. When a prebuild is triggered, the Docker image gets built, a fresh /workplace with a checkout of that commit is mounted, and the expensive (init) step of .gitpod.yml scripts get run. The updated /workplace is saved and reused upon workplace start up, so the init step can be skipped at that time.

For a large project like GHC, enabling Gitpod prebuilds is essential to ensure a sane startup time for new workspaces. Here's a non-comprehensive list of things done during gitpod-ghc prebuild time:

Install latest release versions of ghcup, ghc and cabal
Compile and install haskell-language-server
Check out GHC tree into /workspace
Go through the configure step, build hadrian, build up the .hie-bios cache, so the language server shall work nearly instantly for a fresh workspace

In-browser VSCode caveats

Oops. I've written a lot in this post and haven't even mentioned VSCode even once. Yes, Gitpod supports multiple IDEs, but the one I'm using on a daily basis is the in-browser VSCode. Some caveats to keep in mind:

Most default key bindings work. ^W closes the browser tab instead of the editor tab though, luckily there'll be a prompt about unsaved work. Don't silence that prompt.
Flaky connection is fine, but don't drop offline for more than 3 minutes.
Settings do get synced properly. A minor annoyance is you can't declare editor settings in .gitpod.yml, apart from the extension list.
The default extension store is OpenVSX instead of the Microsoft-hosted one. There do exist extensions that are severely outdated in OpenVSX (like rust-analyzer), but as a fallback measure, you can specify URLs to .vsix artifacts in .gitpod.yml.

Other things known to work (or not)

There are things that work, and things that don't work yet. I didn't really spend time to dig into Gitpod's issue tracker, so I'll be more than happy to be corrected in the comments shall the list requires some update!

sudo works. Of course you're still in a container jail, so you can't really mess with sysctl config and such.
systemd doesn't work. You need to bring your own userspace supervisor if you need such a thing.
nix works with single-user installation and sandbox = false. Multi-user mode with nix-daemon should still be possible, but not worth the effort if sandboxing won't work anyway. So don't ever push to binary caches from a Gitpod workspace.
docker works out of the box as rootless mode, and the state directory is in /workplace. podman doesn't, despite unprivileged user namespace is on.
gdb works. strace works. lldb doesn't. rr doesn't.
proot latest version works.
tailscale userspace mode works, but running the daemon as root seems quite fragile. Haven't tried other VPNs yet.
htop kinda works. The CPU/RAM stats don't reflect the workspace, but reflects the entire metal below it.

Community support

Gitpod development happens on GitHub, they allow self-hosting and the sausages are made in the open.

Apart from that, there's an official discord server. Not a super active one, but the staff are there, and questions do get answered within a day or two, regardless of whether you're a paid user or not.

Conclusion

I believe I've accumulated enough experience to say: Gitpod is nice, and it's worth a try.

Thoughts about configure scripts and feature vectors

Cheng Shao — Fri, 06 May 2022 14:17:10 +0000

For a long time, I’ve been wanting to rant about stuff like configure scripts. They indirectly contribute a lot to my worktime headaches these days. Given I’m restarting personal blogging here, let’s see if I can turn that rant into a post.

What’s configure

Suppose you need to compile some software from a source tarball. The old unix tradition goes like ./configure && make && make install. Nothing fancy about the make part, but why is configure needed in the first place?

Well, configure is just a shell script that probes the build environment and generates some C header file to be included in the source code. Each project has its unique configure script that probes for different things, headers, functions, any feature that the source code needs to check for build-time existence and provide a fallback code path when it’s absent.

Say that the source code needs to call the foo function, which exists on just some of the project’s supported platforms. If configure detects foo, it’ll write a #define HAVE_FOO 1 line to the generated header. The source code can then include the auto-generated feature header, use CPP declarations like #if defined(HAVE_FOO) to decide whether function foo exists in the build environment.

configure is typically auto-generated from a template using autoconf. In some projects it can be a hand-written python script. There are also build systems like cmake that take over configure's role completely, probes the build environment on their own and generate the feature header.

Anyway, my rants here are only related to the idea of build-time feature detection, and irrelevant to how configure is actually implemented (although that’s also annoying enough for it’s own blog post)

What’s a feature vector

How many HAVE_ macros do you have in your project?

~/ubuntu/ghc$ grep -rIF HAVE_ | wc -l
842

Wait a sec. Most of those should be mere duplications, for instance, HAVE_FOO is very likely to occur in multiple source locations. One should really check how many features (headers, functions, etc) are checked by configure.

~/ubuntu/ghc$ grep -rIF AC_CHECK_ | wc -l
171

The number above is a lower estimation, since in autoconf, a single AC_CHECK_HEADERS or AC_CHECK_FUNCS line can check multiple entities.

Now, we can introduce the concept of a “feature vector”: an N-dimentional boolean vector, where N corresponds to the number of things you’re checking at build-time. Each value of the feature vector is a point in the feature space, specifying a build-time configuration.

How large is the feature space?

Definitely not as large as 2^N. Most of the dimensions aren’t orthogonal, one can imagine clusters of things that either exist as a whole, or not exist at all.
Still, way larger than the the space where people actually test on CI to avoid bit-rotting.

My rant is the second point above.

Why the rant

In GHC, one can pass various configure arguments to enable/disable features like unreg codegen, large address space, native IO manager, etc. The default configuration passes the test suite, but once you start messing with configure config, expect failed test cases. At least those cases should be explicitly marked fragile/broken in those configurations!
In GHC, unix, probably other places I’ve hacked and forgotten: the API evolves, but people forgot to update the code in the #else -guarded parts, because it’s not tested on CI, maybe that particular CPP-checked thing is thought to exist on all platforms. Well, WASI is a rather restricted platform, so those bitrotten parts all come back to bite you when I target WASI.

There’s nothing to blame for the need to write portable code and do build-time feature detection. We all know untested code is bad, but much fewer people are aware: untested feature vectors are also bad!

Also, this isn’t just a matter of “code coverage”. It’s perfectly possible to achieve a high coverage rate by testing against just a few feature vectors, while leaving the potentially broken build-time configurations in the dark.

How to solve it

Most software written with tons of #ifdef lack the feature vector mindset, and don’t have the testing logic to:

Perform QuickCheck-style random testing in the feature space. Generate a feature vector, run the tests against it, and “shrinking” is just moving the point closer to the known-to-work base point, the default config you get when configuring on a typical platform without any custom arguments. This allows discovering failures that arise from complex & unintended interactions between different dimensions in the feature vector.
Hide certain existing auto-detected features. This allows testing for restricted/exotic platforms, while still running the tests on a common platform. Not trivial to implement, especially for things in standard libraries, but it should be possible by using poison pragma, creating cc wrappers, or even isolated sysroots.

My GHC dev environment with vscode remote & docker

Cheng Shao — Thu, 28 Apr 2022 15:10:51 +0000

For my daily work of hacking GHC, I’ve recently transitioned to a workflow of using VSCode remote + Docker containers. The main advantage is being able to reuse the same Docker images used by GHC CI, so you can have more confidence with local test results before pushing. This post briefly walks through the process to set up such a dev environment.

Building & starting the dev container

The first step is picking an image specified in the GHC CI config, using that as a base image to build our dev image. The original images aren’t suitable to be used directly, because:

The default ghc user’s uid/gid likely doesn’t match your host user’s uid/gid. This will result in file permission issues if you mount a host directory into the container and use that as the working directory.
The default PATH doesn’t contain ghc-controlled directories like ~/.cabal/bin or ~/.local/bin. This can be a minor annoyance if you build and install HLS(haskell-language-server) in those directories, the VSCode Haskell extension wouldn’t be able to find the executable automatically.

Here’s the Dockerfile recipe:

FROM registry.gitlab.haskell.org/ghc/ci-images/x86_64-linux-deb10:0849567cd9780cc8e9652118b949cb050c632ef4

ARG UID
ARG GID

RUN \
  curl -SsL https://github.com/boxboat/fixuid/releases/download/v0.5.1/fixuid-0.5.1-linux-amd64.tar.gz | sudo tar -C /usr/local/bin -xzf - && \
  sudo chown root:root /usr/local/bin/fixuid && \
  sudo chmod 4755 /usr/local/bin/fixuid && \
  sudo mkdir -p /etc/fixuid && \
  printf "user: ghc\ngroup: ghc\n" | sudo tee /etc/fixuid/config.yml

USER ${UID}:${GID}

RUN fixuid

USER ghc

RUN \
  sudo rm -r \
    /etc/fixuid \
    /usr/local/bin/fixuid \
    /var/run/fixuid.ran

ENV PATH=${PATH}:/home/ghc/.cabal/bin:/opt/ghc/9.2.2/bin

RUN \
  sudo apt update && \
  sudo apt install -y \
    bash-completion && \
  cd /tmp && \
  git clone https://github.com/haskell/haskell-language-server.git && \
  cd haskell-language-server && \
  cabal --project-file=cabal-ghc92.project update && \
  cabal --project-file=cabal-ghc92.project install && \
  rm -rf /tmp/*

Which base image to use? Check out .gitlab-ci.yml and jobs.yaml to figure out which Docker images are used by GHC CI. Here we use deb10 , although a bit outdated, it’s the most widely used one, especially for edge cases like unreg or tsan CI jobs.
We use the fixuid project for correcting the default ghc user’s uid/gid to the numbers provided. Typically, fixuid is invoked once upon first run of the container, but it’s actually possible to do this at image build time. The UID and GID variables must be specified to docker build via --build-arg UID=$(id -u) --build-arg GID=$(id -g).
We build and install HLS during image build time. There are prebuilt binaries available, but my past experience tells me HLS works properly only if built by the exact same GHC used to build your projects.

Start the container after the dev image is built. Don’t forget to mount your working directory.

The dev container is supposed to be a long-running container, and the simplest way to keep it running is using a screen or tmux session.

Optional: setting up Docker context

Skip this part if you’re using Docker on your local machine. If you’re using a remote machine, the previous steps are done within an SSH session, then you also need to:

Install the Docker CLI locally. It doesn’t need to be a full Docker installation, we don’t need to connect to the Docker daemon.
Set up the Docker context using the instructions here.

This way, the local docker commands will transparently talk to the remote Docker daemon using SSH.

Connecting to the dev container

The rest is simple: enable the VSCode remote extension pack, click on the green button on VSCode window’s bottom left corner to open the remote menu, then select “Attach to Running Container”, voila. Set up extensions you use, open your working directory, and happy hacking GHC.