DEV Community

Cover image for REMEMBER: gitlab ci runner & pipefail == ouch
Alexander Sack
Alexander Sack

Posted on

2 1

REMEMBER: gitlab ci runner & pipefail == ouch

Any odd failures that make no sense in gitlab-ci.yaml scripts? Maybe those are using pipes? Gotcha.

Gitlab has by default the pretty 'anal' pipefail bash mode fail, which you usually dont see even if you usually use "set -e" ...

So for instance we wanted to truncate certain files in CI to a certain bytesize and use this script pattern:

 - cat /path/to/largefile | head -c 131200 > /path/to/truncfile
Enter fullscreen mode Exit fullscreen mode

Now interestingly enough we always had this command failing our CI scripts if the largefile became substantially larger than the head cut off.

Turned out that cat indeed will fail as its downstream process exits before its done catting the full largefile.

Try it yourself:

$ $ dd if=/dev/random of=/tmp/largefile bs=1M count=3
3+0 records in
3+0 records out
3145728 bytes (3,1 MB, 3,0 MiB) copied, 0,0467213 s, 67,3 MB/s

# here how it works nicely, with return code 0
$ cat /tmp/largefile | head 1024 > /tmp/truncfile
$ echo $?
0

# ... now set pipefail mode it fails ...
$ set -o pipefail
$ cat /tmp/largefile | head -c 1024 > /tmp/smallfile
$ echo $?
141
Enter fullscreen mode Exit fullscreen mode

Image of Timescale

🚀 pgai Vectorizer: SQLAlchemy and LiteLLM Make Vector Search Simple

We built pgai Vectorizer to simplify embedding management for AI applications—without needing a separate database or complex infrastructure. Since launch, developers have created over 3,000 vectorizers on Timescale Cloud, with many more self-hosted.

Read full post →

Top comments (0)

Billboard image

The Next Generation Developer Platform

Coherence is the first Platform-as-a-Service you can control. Unlike "black-box" platforms that are opinionated about the infra you can deploy, Coherence is powered by CNC, the open-source IaC framework, which offers limitless customization.

Learn more