DEV Community: Gaëtan Montury

8 small data transforms I don't want to write as shell glue anymore

Gaëtan Montury — Thu, 02 Jul 2026 14:28:00 +0000

🦀🐍 Practical fimod examples for small CI/config data transforms with a Python-like taste.

In the first article,
I wrote about why I built fimod: I kept seeing tiny python3 -c scripts in CI pipelines, doing just enough JSON/YAML/CSV work to be useful, and just enough shell quoting to become annoying.

This article is more concrete.

Here are 8 small transforms that I do not want to keep rewriting as shell glue, ad-hoc Python snippets, or copied scripts between repositories.

fimod is not trying to replace jq or yq. Those are excellent tools. I see fimod more as a small data-shaping Swiss army knife: a few megabytes, Python-like expressions, common structured formats, reusable molds, and a controlled execution surface.

📚 Website/docs: https://pytgaen.github.io/fimod/

⭐ Repository: https://github.com/pytgaen/fimod

1. Extract one value for a shell script

Sometimes a CI job only needs one value from a structured file.

{"name":"demo","version":"1.2.3"}

With fimod:

fimod s -i package.json \
  -e 'data["version"]' \
  --output-format txt

Output:

1.2.3

That makes it easy to use in shell:

VERSION=$(fimod s -i package.json -e 'data["version"]' --output-format txt)

No JSON boilerplate, no quotes around the string, no temporary script.

2. Validate a config file in CI

For boolean checks, --check suppresses stdout and maps truthiness to the exit code.

fimod s -i deploy.yaml \
  -e 'all(k in data for k in ["image", "replicas", "port"])' \
  --check

For clearer error messages, use gk_assert:

fimod s -i deploy.yaml -e '
def transform(data, **_):
    gk_assert("image" in data, "missing image")
    gk_assert("replicas" in data, "missing replicas")
    gk_assert("port" in data, "missing port")
    return True
' --check

On failure, the error messages go to stderr and the process exits non-zero — the shape CI expects.

3. Read an HTTP API directly

For small API transforms, fimod can read HTTPS URLs directly:

fimod s -i https://api.github.com/repos/pytgaen/fimod \
  -e '{"name": data["name"], "stars": data["stargazers_count"]}'

Example output:

{
  "name": "fimod",
  "stars": 4
}

This is not meant to replace a full HTTP client in an application. But for CI metadata, release scripts, or quick API-to-config glue, it avoids another curl | jq | sed chain.

4. Extract named regex captures

fimod injects regex helpers into every transform. re_search returns structured data, including named captures.

echo '{"tag":"release-v2.4.1"}' \
  | fimod s \
      -e 're_search(r"(?P<major>[0-9]+)\.(?P<minor>[0-9]+)", data["tag"])["named"]'

Output:

{
  "major": "2",
  "minor": "4"
}

Under the hood, the regex helpers use Rust's fancy-regex crate, with PCRE-like features such as lookahead, lookbehind, backreferences, and named captures.

5. Flatten nested fields to CSV

APIs often return nested JSON, while the next step wants a flat CSV artifact.

[
  {"name":"Alice","email":"alice@example.com","address":{"city":"Paris"}},
  {"name":"Bob","email":"bob@example.com","address":{}}
]

fimod s -i users.json \
  -e '[{"name": u["name"], "email": u["email"], "city": dp_get(u, "address.city", "unknown")} for u in data]' \
  -o contacts.csv

Output:

name,email,city
Alice,alice@example.com,Paris
Bob,bob@example.com,unknown

dp_get avoids a small pile of defensive nested dict.get(...) calls.

6. Hash sensitive fields before exporting

For anonymized fixtures or safer artifacts, hashing is built in:

fimod s -i people.csv \
  -e '[{**row, "email": hs_sha256(row["email"])} for row in data]' \
  -o people-anon.csv

Input:

name,email
Alice,alice@example.com
Bob,bob@example.com

Output shape:

name,email
Alice,ff8d9819fc0e12bf0d24892e45987e249a28dce836a85cad60e28eaaa8c6d976
Bob,5ff860bf1190596c7188ab851db691f0f3169c453936e9e1eba2f9a47f7a0018

The point is not that hashing is hard in Python. It is not. The point is that in a small CI transform, I do not want to import and wire another script just for that.

7. Generate text/config with MiniJinja

fimod is not limited to data-to-data transforms. It can also render text from structured data using MiniJinja-powered helpers.

echo '{"app":"api","host":"localhost","port":8080}' \
  | fimod s --output-format txt \
      -e 'tpl_render_str("APP={{ app }}\nURL=http://{{ host }}:{{ port }}\n", data)'

Output:

APP=api
URL=http://localhost:8080

This opens the door to .env files, Markdown snippets, Dockerfiles, small config files, or release notes generated from structured data.

For larger templates, fimod also supports rendering template files from directory molds with tpl_render_from_mold(...).

8. Reuse a mold from a registry

One-liners are nice, but the more interesting part is sharing transforms.

A mold is a reusable transform script. A registry lets you call molds by name with @name instead of copying scripts between repositories.

For example, with the example registry configured, you can call the shared pick_fields mold by name:

fimod s -i contacts.csv \
  -m @pick_fields \
  --arg fields=name,email \
  -o contacts-public.json

This works because fimod resolves @pick_fields from a configured registry. In a fresh environment, you can configure the default registry once, or point this command at the official example registry explicitly with FIMOD_REGISTRY=https://github.com/pytgaen/fimod/tree/main/molds.

Input:

name,email,role
Alice,alice@example.com,admin
Bob,bob@example.com,user

Output:

[
  {
    "name": "Alice",
    "email": "alice@example.com"
  },
  {
    "name": "Bob",
    "email": "bob@example.com"
  }
]

In a team, that registry can be your own Git repository. That is where fimod becomes more than a one-liner tool: reviewed transforms can be reused across projects without copy-pasting another tiny Python script everywhere.

So when would I use this?

Fimod is a great fit for CI/config data plumbing: one small but powerful binary that can read many structured formats, pull input from HTTP when the data lives behind an API, and reuse molds when a transform should become shared project knowledge.

The registry and sandbox matter for that last part. They make sharing a transform less about copying a script around, and more about resolving known code from a known place, with a controlled execution surface.

I would reach for it when I need a small, explicit data transform in a pipeline:

extract one value;
validate a config;
reshape JSON/YAML/CSV/TOML;
read a small API response;
hash or mask fields;
generate a tiny text artifact;
reuse a shared transform as a mold.

That is where I want fimod to shine: boring CI/config transforms with less glue.

If you try one of these examples, I would start with a small file you already manipulate in a script today.

⭐ If the idea looks useful, a GitHub star helps the project be discovered.
💬 If you have a quick reaction, a comment here on dev.to is perfect.
🐛 If an example breaks or the docs are unclear, a short GitHub issue with the command you tried is more than enough.

Replacing fragile python3 -c scripts in CI with fimod

Gaëtan Montury — Tue, 02 Jun 2026 14:26:30 +0000

🦀🐍 A small Rust CLI for Python-like data transforms without Python installed.

I keep running into the same kind of script in CI pipelines:

python3 -c 'import json,sys; data=json.load(sys.stdin); print(json.dumps(...))'

It usually starts as a quick one-liner. Then it grows a condition. Then it needs to read YAML instead of JSON. Then someone has to escape quotes inside quotes inside YAML. Eventually it becomes either a small script committed somewhere, or another piece of shell glue that nobody really wants to maintain.

That is the problem I built fimod for. 🛠️

fimod is a small Rust CLI that transforms structured data - JSON, YAML, TOML, CSV, NDJSON, text lines - using Python-like expressions powered by Pydantic Monty, a Python runtime written in Rust.

No system Python is required. No pip install. No custom mini-language to learn if you already know basic Python. The standard binary is only a few megabytes - about 3.2 MB on my current Linux install - and still includes parsing, serialization, HTTP input, and a set of Rust-powered data helpers.

The goal is not to replace Python. The goal is to remove the boilerplate around small data transformations in CI, scripts, and config workflows.

📚 Website/docs: https://pytgaen.github.io/fimod/

⭐ Repository: https://github.com/pytgaen/fimod

In short:

🐍 write small transforms with Python-like expressions;
🦀 ship them as a small Rust binary;
📦 read/write JSON, YAML, TOML, CSV, NDJSON and text;
🔁 keep CI data plumbing explicit and shell-friendly.

🧪 A tiny example

Suppose a CI job has a JSON file like this:

[
  {"name": "api", "active": true},
  {"name": "worker", "active": false},
  {"name": "web", "active": true}
]

With a regular Python one-liner, the useful logic is often hidden inside JSON loading, stdin handling, output formatting, and shell escaping.

With fimod, the parsed input is already available as data:

fimod s -i services.json \
  -e '[svc for svc in data if svc["active"]]' \
  -o active-services.json

The expression is just the transformation. fimod handles reading, parsing, serializing, and writing.

✅ Validate config in CI

A common CI task is checking that a config file contains the fields a deployment
expects.

fimod s -i deploy.yaml \
  -e 'all(k in data for k in ["image", "replicas", "port"])' \
  --check

--check suppresses stdout and uses the truthiness of the result as the process
exit code:

exit 0 if the result is truthy;
exit 1 if the result is falsy.

That makes it easy to use in shell scripts and CI jobs:

fimod s -i deploy.yaml \
  -e 'all(k in data for k in ["image", "replicas", "port"])' \
  --check || {
    echo "Invalid deploy config" >&2
    exit 1
  }

For more explicit error messages, fimod also provides validation helpers such as
gk_assert:

fimod s -i deploy.yaml -e '
def transform(data, **_):
    gk_assert("image" in data, "missing image")
    gk_assert("replicas" in data, "missing replicas")
    gk_assert("port" in data, "missing port")
    return True
' --check

gk_assert reports each missing field on stderr and sets a non-zero exit code on failure. The final return True makes the successful case truthy for --check, while --check keeps stdout silent.

🔀 Convert formats without changing the logic

Because parsing and serialization are handled by fimod, the same expression can work across different input formats.

YAML to TOML:

fimod s -i config.yaml -e 'data' -o config.toml

CSV to JSON, with an explicit cast for age:

fimod s -i users.csv \
  -e '[{**row, "age": int(row["age"])} for row in data]' \
  -o users.json

CSV values arrive in fimod as strings, so int(row["age"]) is intentional.
Modern yq can do CSV-to-JSON very well too - for example
yq -p=csv -o=json '.' users.csv - so this is not meant as a "yq cannot do this" example. The point is that when the transformation grows, the logic stays in familiar Python-like syntax.

Extract one value for a shell variable:

VERSION=$(fimod s -i package.json \
  -e 'data["version"]' \
  --output-format txt)

The point is not that these examples are impossible with other tools. They are not. The point is that the transformation stays focused on the data, while fimod handles the repetitive I/O details.

🧰 What about jq and yq?

jq and yq are excellent tools. If your team already knows them and they solve the problem cleanly, keep using them.

fimod is useful when you prefer Python-like expressions and want the same command shape across JSON, YAML, TOML, CSV, NDJSON, and text. It is not meant as a "yq cannot do this" argument; it is a different tradeoff.

🧱 Reusable transforms: molds

One-liners are convenient, but CI pipelines eventually grow shared logic.

In fimod, a reusable transform is called a mold. It is a Python-like script with a transform(data, **_) function:

# cleanup.py
def transform(data, **_):
    for row in data:
        row["name"] = row["name"].strip().title()
        row["email"] = row["email"].strip().lower()
    return data

Run it with:

fimod s -i contacts.csv -m cleanup.py -o contacts.json

Molds can also receive parameters:

# filter_by_field.py
def transform(data, args, **_):
    field = args["field"]
    value = args["value"]
    return [row for row in data if row.get(field) == value]

fimod s -i users.json \
  -m filter_by_field.py \
  --arg field=role \
  --arg value=admin

The really practical part is that molds are not just local scripts.
They can be shared through registries and referenced by name with @.

For example, a CI job can point fimod at a local shared mold directory:

FIMOD_REGISTRY=./molds \
  fimod s -i contacts.csv -m @cleanup -o contacts.json

Or at the official fimod example registry on GitHub, and call a real published mold by name:

FIMOD_REGISTRY=https://github.com/pytgaen/fimod/tree/main/molds \
  fimod s -i contacts.csv \
    -m @pick_fields \
    --arg fields=name,email \
    -o contacts-public.json

In a real team setup, that registry can be your own Git repository. The important bit is that every project can call the same reviewed transform by name instead of copying another tiny script into every repo.

That is where molds become more than a convenience: they become shared data recipes for CI. I will cover registries more deeply in a separate article, but even the basic @name pattern is already useful.

🐍 Why not just use Python?

For many tasks, you should use Python.

But plain Python is not a unified structured-data CLI out of the box. JSON and CSV are in the standard library, but YAML usually means an extra dependency, TOML support depends on the Python version, and NDJSON, format detection, stdin/stdout plumbing, shell-friendly output, and cross-format conversion all add glue code.

That glue is fine in a real application. In a CI job, it often turns a small transformation into dependency management, quoting, and boilerplate.

fimod is useful when the script is small enough that a full Python file feels heavy, but important enough that a quoted python3 -c command is painful.

✨ What I am not covering here

fimod also has HTTP input, MiniJinja-powered templating for data-to-text outputs, and built-in helpers for regex, nested fields, grouping, hashing, validation, diagnostics, and environment substitution. The regex helpers are backed by Rust's fancy-regex crate, with PCRE-like features such as lookahead, lookbehind, backreferences, and named captures.

That means later articles can cover things like fetching an API response, extracting fields, hashing sensitive values, or rendering .env, Markdown, Dockerfile, and config files from structured data.

Those deserve separate articles. This one is intentionally focused on the first use case: replacing fragile CI one-liners and sharing transforms as molds.

⚠️ Important limitations

fimod is early-stage software.

A few things are intentionally not hidden:

fimod runs on Monty, not CPython;
it does not support arbitrary PyPI packages;
molds cannot freely access the filesystem or the network;
the project is maintained part-time;
the API and behavior may still evolve as Monty and fimod mature.

Those constraints are part of the design. fimod is not trying to be a general Python runtime. It is trying to be a practical data-shaping CLI with explicit inputs and controlled execution.

🚀 Try it

Install:

curl -fsSL https://raw.githubusercontent.com/pytgaen/fimod/main/install.sh | sh

Run a first transform:

echo '[{"name":"Alice"},{"name":"Bob"}]' | fimod s -e 'len(data)'

📚 Website/docs:

https://pytgaen.github.io/fimod/

⭐ Repository:

https://github.com/pytgaen/fimod

If you try fimod, I would start with one small, boring transform in a CI job or config workflow. 🧪

⭐ If the idea looks useful, a GitHub star helps the project be discovered.
💬 If you have a quick reaction, a comment here on dev.to is perfect.
🐛 If something feels unclear - the install, the command shape, the docs, the error message, or a small improvement idea - a short GitHub issue with the you tried is more than enough.