Build, test, publish: your Python function is now a drag-and-drop node

#python #showdev #opensource #datascience

There's a k-means clustering node in Flowfile now, and it didn't come from a release. I built it inside the app — drew the settings form, wrote one process() function, tested it against sample data, hit publish — and Flowfile opened the pull request that put it one click away for everyone else.

Flowfile is an open-source, self-hosted analytics tool built on Polars: you build data pipelines on a visual canvas — where every node is also real Python — then explore, schedule, and chart the results, all on one machine you pip install. It ships with 40+ built-in nodes, and until now that was the ceiling.

The palette grew one way: a Flowfile release added nodes. Community Nodes are the second way — someone builds a node once, and it's a card in everyone's catalog. Before, if the node you needed didn't exist, you wrote a Python Script node and kept the snippet around.

Any Python library can become a node

A node can declare its own pip dependencies and run in an isolated Docker kernel. The k-means node declares scikit-learn and gets it inside its own container — nothing is installed into your Python environment, and whoever installs the node never touches pip at all. The import happens inside process(), inside the container:

import polars as pl
from flowfile import node_designer as nd


class KmeansClustering(nd.CustomNodeBase):
    node_name: str = "kmeans clustering"
    node_category: str = "ML"
    environment: str = "kernel"
    dependencies: list[str] = ["scikit-learn"]
    # ... icon, tags, settings schema, and bundled example data trimmed

    def process(self, *inputs: pl.LazyFrame) -> pl.LazyFrame:
        from sklearn.cluster import KMeans
        from sklearn.preprocessing import StandardScaler

        cfg = self.settings_schema.clustering
        feature_cols = cfg.feature_columns.value
        k = int(cfg.n_clusters.value)  # a NumericInput is always a float
        label_col = cfg.cluster_column.value

        df = inputs[0].collect()  # sklearn needs eager data
        features = df.select(feature_cols).to_numpy()
        if cfg.standardize.value:
            features = StandardScaler().fit_transform(features)

        model = KMeans(n_clusters=k, n_init=10, random_state=42)
        labels = model.fit_predict(features)
        return df.with_columns(pl.Series(label_col, labels))

Whoever installs it drags it onto the canvas and gets clusters:

A k-means community node on the canvas, its clusters colored in the Explore Data charts.

That's the line between a snippet library and an extension system, and it generalizes to most of PyPI. A geocoding node. A forecasting node. A client for that one API your team calls from every project. The person who knows the library well builds it once; everyone after them gets a form and an output table.

Simple nodes don't need any of that. String Cleaner — trim, collapse spaces, change case — runs locally as plain Polars, one file, no installs. And the registry has room for less serious work too: there's a node that maps numbers to mood emojis. I won't defend it, but it works.

You build it inside the app

There's no SDK to install and no repo to clone. The Node Designer is part of Flowfile — a Form tab where you drag typed controls together, a Code tab with the one function your node runs, and a Test tab that executes it against sample data:

Building a custom node in Flowfile's Node Designer.

The whole node is one Python file. Here's String Cleaner, close to its entirety:

import polars as pl
from flowfile import node_designer as nd


class StringCleanerSettings(nd.NodeSettings):
    cleaning: nd.Section = nd.Section(
        title="Clean String Columns",
        columns=nd.ColumnSelector(
            label="Columns to clean",
            required=True,
            multiple=True,
            data_types=["String"],
        ),
        operations=nd.MultiSelect(
            label="Operations",
            options=[
                ("trim", "Trim leading/trailing whitespace"),
                ("collapse_spaces", "Collapse repeated whitespace"),
                ("remove_punctuation", "Remove punctuation"),
                ("lowercase", "Lowercase"),
                # ... four more in the published file
            ],
            default=["trim", "collapse_spaces"],
        ),
    )


class StringCleaner(nd.CustomNodeBase):
    node_name: str = "String Cleaner"
    node_category: str = "Text Processing"
    # ... icon, author, tags, and the example data it was tested against

    settings_schema: StringCleanerSettings = StringCleanerSettings()

    def process(self, *inputs: pl.LazyFrame) -> pl.LazyFrame:
        lf = inputs[0]
        columns = self.settings_schema.cleaning.columns.value or []
        operations = self.settings_schema.cleaning.operations.value or []
        if not columns or not operations:
            return lf

        def clean(name: str) -> pl.Expr:
            expr = pl.col(name)
            if "remove_punctuation" in operations:
                expr = expr.str.replace_all(r"[^\w\s]", "")
            if "collapse_spaces" in operations:
                expr = expr.str.replace_all(r"\s+", " ")
            if "trim" in operations:
                expr = expr.str.strip_chars()
            if "lowercase" in operations:
                expr = expr.str.to_lowercase()
            # ... digits, uppercase, title case
            return expr.alias(name)

        return lf.with_columns([clean(c) for c in columns])

The Section is the form; process() is the node. There's no third thing. The settings you draw become the form other people fill in after they install it — here's the k-means node's, four controls in a group:

The Form tab: the Clustering group and its four controls.

The Node Designer guide walks through building one step by step. If you can write a function that takes a dataframe and returns one, you can build a node.

Sharing it is the easy part

Publishing doesn't ask you to leave the app. Hit Publish, connect GitHub once, and Flowfile opens the pull request for you — no local git, no packaging. CI runs your node against the test setup you saved, a maintainer reads the code, and the merge puts it in every catalog.

Installing is the same one click in reverse: Catalog → Community Nodes, pick a card, and a consent dialog shows who wrote it, where the source lives, and what the code is allowed to do before anything lands in your palette. Every file is verified against its pinned hash. Nodes are reviewed and pinned, but not sandboxed — a local node runs with the same access your own code has, so the source being one click away is part of the design. Read it. The contributing guide has the full publishing contract if you want the details.

What's there today

Four nodes, all mine, published while proving the loop end to end — browse them here. The more interesting part is the empty shelf space: the registry's Connectors, Geospatial, Text, and Cleaning categories have nothing in them yet.

Catalog → Community Nodes — the four nodes live today. The k-means card is tagged **Kernel: it brings its own scikit-learn.

So here's the concrete version of the ask. A geocoding node that wraps whichever provider you already pay for. A forecasting node — Prophet or statsmodels behind one form. A reader for the internal API every project at your company re-implements from scratch. Each of those is a node one person builds once and everyone else installs.

Community nodes landed in Flowfile 0.13.1 — you'll want that version or later, via the desktop app, Docker, or pip install flowfile. They're not in the browser demo, which runs a smaller node set with no server. If there's a transformation you keep rebuilding in every project, the Node Designer plus one pull request is the distance between it and everyone else's palette.

Edwardvaneechoud / Flowfile

Flowfile is a visual ETL tool and Python library combining drag-and-drop workflows with Polars dataframes. Build data pipelines visually, define flows programmatically with a Polars-like API, and export to standalone Python code. Perfect for fast, intuitive data processing from development to production.

Flowfile

Visual ETL that compiles to Polars.
_{Build pipelines on a canvas, run them locally or in the browser, export them as standalone Python.
No platform lock-in. No install required to try it.}

▶ Try it in your browser →
_{No install. No signup. Polars in the browser via Pyodide.}

Docs · Releases · Discussions · Architecture deep-dive

Build pipelines on a visual canvas, run them, and export the graph as plain Python code that runs anywhere. Code and visual are two views of the same graph: drag nodes or write Python with a Polars-like API, your choice.

Beyond the canvas: a Delta-backed catalog with time-travel and virtual tables, a SQL editor with embedded viz, flow parameters, sandboxed Python kernels, and a built-in scheduler.