DEV Community: Andrew

Data Science in Banking: Finding Hard Rules with Shallow Decision Trees

Andrew — Wed, 28 Oct 2026 20:20:28 +0000

Data Science in Banking, Part 1: Finding Hard Rules with Shallow Decision Trees

A credit team reviewing an early-arrears model will eventually ask a practical question: which applications does it flag, and why? A short rule can make that discussion concrete. It identifies a segment, shows the conditions for entering it, and lets us count the cases that contradict its prediction.

The difficult part is earning that simplicity. A threshold can look convincing because it describes the training sample, because it uses information from the future, or because someone kept adjusting it after seeing the test results. A readable rule deserves the same scrutiny as a complicated model.

This first article in Data Science in Banking follows one experiment from candidate conditions to a frozen model and its mistakes. The banking question supplies the motivation. The executable experiment uses the public Wine dataset, whose chemical measurements identify cultivars. Wine contains no borrowers, and its results say nothing about lending performance.

Start with a decision that has a time

Consider an illustrative lending task: flag an application for additional review when it is assessed, before approval and disbursement. Define the eventual target as:

early_arrears = 1 if the funded loan reaches at least 30 days past due at any point during the first 90 days after disbursement; otherwise 0, once the full outcome window is observable.

This is a definition for this example, not a universal definition of credit default. Historical evaluation would use funded loans with complete servicing records and sufficient follow-up. A recent loan with no arrears yet is not automatically a negative example. Training on funded loans also means that the measured performance applies to that selected population; rejected applicants have no equivalent repayment outcome.

Potential predictors include income evidence, existing repayment obligations and a bureau snapshot available at the application decision. A later collections event, a balance updated after disbursement, or a column derived from the target cannot enter the feature table. Even an old transaction can leak information if its corrected value only became available later. Both event time and information availability matter.

A shallow tree could search these approved predictors for small segments with different observed arrears rates. Whether a segment should trigger manual review would then depend on its coverage, false alarms, missed arrears and the team's review capacity. We will not invent those banking results. Instead, we will make the mechanics inspectable on a dataset anyone can run.

Give the tree a small budget

A decision tree tries feature thresholds and chooses splits that reduce class impurity. Each path from the root to a leaf becomes a conjunction of conditions. The search is greedy: a good next split need not lead to the best possible rule set. Depth and leaf-size limits help control how much detail it can fit. Scikit-learn's tree guide describes this construction and its limitations.

A hard rule has a definite boundary: a row either satisfies its conditions or does not. That does not make its prediction certain.

The experiment retains the original published search: 24 candidate configurations, plus a constant baseline. It compares ordinary trees, ordered rule lists, and a merge_refit variant that can combine regions with similar class distributions and try splitting them again. In a rule list, the first matching rule wins, so earlier conditions affect what later rules mean.

The base trees have depth limits of 2, 3 or 4 and at most eight leaves. Each tree leaf needs at least 20 training rows, or more if its configured fraction demands it. This is total support, not a minimum number of positive cases. The extensions can produce longer expressions than a base tree's depth suggests; the final rule expressions still need inspection.

We choose the configuration by three-fold cross-validation using macro-F1, which averages the F1 scores of the two classes equally. This avoids selecting on accuracy alone when one class is larger. It is a demonstration criterion, not a cost model for lending.

The simplicity rule is fixed before evaluation: among candidates within 0.005 of the best mean CV score, prefer fewer predicate occurrences, then fewer segments. This tolerance is an engineering choice, not a confidence interval. Counting predicates across complete rules also differs from counting unique splits: one threshold appears in both complementary leaf rules.

Put the external holdout aside first

Wine has 178 rows and 13 numeric measurements. Its original label has three classes; here, original class 0 is cultivar A, and the other two classes form the negative class. The original target is never included among the predictors. Wine dataset description.

The examples use Python 3.13 and these package versions, verified on 28 October 2026:

python -m pip install "splime==0.4.13" "numpy==2.3.5" "pandas==2.2.3" 
"scikit-learn==1.8.0"

Run the Python blocks in order. This first block constructs the public demonstration table, then calls numeric_rule_partition@1 to reserve an external holdout before any feature screening, imputation or rule search:

import pandas as pd
from sklearn.datasets import load_wine
from spl import SPLClient

wine = load_wine(as_frame=True)
TARGET = "is_cultivar_a"
dataset = wine.data.copy()
dataset[TARGET] = (wine.target == 0).astype(int)

client = SPLClient.embedded(
    registry_url="https://splime.io",
    cache_dir=".sph-wine-cache",
    run_receipts=False,
)
BASE = "splime://@splime.project/case5-validation-20261011"

outer_split = client.call(
    f"{BASE}/numeric_rule_partition@1",
    kwargs={"dataset": dataset, "target_col": TARGET},
    trust=True, progress=False,
).value["result"]["default"]
training_pool = dataset.iloc[outer_split["search_input_positions"]].copy()
external_holdout = dataset.iloc[outer_split["external_holdout_positions"]].copy()
assert set(training_pool.index).isdisjoint(external_holdout.index)

Splime Public Hub (SPH) provides the versioned implementation for every search, fitting and evaluation stage below. These public calls require neither registration nor API keys. The first call downloads code from SPH and executes it locally in your Python process; this DataFrame is not sent to cloud training. Optional execution receipts are disabled. trust=True authorizes the downloaded Python code to run with your process's permissions; it is not a sandbox.

The partition stage checks a fixed input contract: real numeric predictors, finite values or missing values, and a complete classification target. Invalid input stops the call; it does not select which columns to keep. It then makes a stratified 75/25 split with seed 42 and returns row positions. This random split is an experimental convenience for Wine, not the proposed validation design for a loan portfolio.

Make the training boundaries visible

The modular route starts with rule_validate@1 and rule_split@1, using the original search settings on this numeric Wine table:

options = {
    "feature_selection_k": 128,
    "feature_selection_min_fraction": 0.5,
    "date_col": None, "group_col": None, "time_gap": 0,
    "max_candidates": 24, "cv_folds": 3,
    "metric": "f1_macro", "random_state": 42,
    "depths": [2, 3, 4], "leaf_fractions": [0.05, 0.1, 0.15],
    "min_leaf_count": 20, "max_leaf_nodes": 8, "max_rules": 3,
}

validated = client.call(
    f"{BASE}/rule_validate@1",
    kwargs={"dataset": training_pool, "target_col": TARGET, "options": options},
    trust=True, progress=False,
).value["result"]["default"]

split = client.call(
    f"{BASE}/rule_split@1", kwargs={"validated": validated},
    trust=True, progress=False,
).value["result"]["default"]

print(len(split["split"]["development"]), len(split["split"]["test"]))
# 106 27

Validation checks the input contract without fitting transformations. Splitting creates an internal test and CV folds within development:

178 Wine rows
├── 133 rows supplied to the rule search
│   ├── 106 development rows: CV selection, then final fitting
│   └──  27 internal test rows: evaluation after selection
└──  45 external holdout rows: never supplied to the search

The 128-feature cap does not remove any of Wine's 13 measurements. Screening still matters: the implementation checks training-only missingness, constants and duplicates, and ranks eligible numeric features. In each CV fold, it repeats screening and learns missing-value medians from that fold's training rows. It then transforms the validation rows using those fitted choices. The final model repeats this process on the 106 development rows.

There is no categorical encoding in this Wine experiment. For a categorical extension, the vocabulary and any encoding statistics would also have to be learned inside each training fold. Encoding the whole table first would break that separation. Scikit-learn's leakage examples explain why preprocessing and feature selection belong inside validation boundaries.

Ask what the extra conditions buy

Now run rule_search@1 as its own published stage:

searched = client.call(
    f"{BASE}/rule_search@1", kwargs={"split": split},
    trust=True, progress=False,
).value["result"]["default"]

comparison = pd.DataFrame(searched["leaderboard"])
print(comparison[[
    "candidate_id", "family", "cv_mean", "mean_predicates", "mean_segments"
]].head().to_string(index=False))
print("Selected candidate:", searched["selected"]["candidate_id"])

The selected candidate is number 13. A useful comparison is:

Candidate	Family	Mean CV macro-F1	Mean predicate occurrences	Mean segments
0	Constant baseline	0.4011	0	1
1	Ordinary tree	0.9266	4	2.67
13	`merge_refit`, selected	0.9266	2	2

These complexity counts are averages across CV fits, not the shape of the final fitted tree. Several candidates tied at the best score. The selected one matched the ordinary tree's validation score with fewer conditions. In this run, simplicity did not require sacrificing CV performance.

That is a narrower conclusion than saying that two conditions are always optimal. We tried a bounded set of configurations on a small sample. The three scores for the selected candidate were approximately 0.9042, 0.9689 and 0.9067; the mean hides that variation. Neither the tolerance nor the tie proves that the chosen threshold will be stable in another sample.

Freeze the model, then inspect its exceptions

Fit the chosen configuration on development with rule_fit@1, evaluate it on the internal test with rule_evaluate@1, and create the report with rule_finalize@1. Each stage is independently callable from SPH:

fitted = client.call(
    f"{BASE}/rule_fit@1", kwargs={"searched": searched},
    trust=True, progress=False,
).value["result"]["default"]

evaluated = client.call(
    f"{BASE}/rule_evaluate@1", kwargs={"fitted": fitted},
    trust=True, progress=False,
).value["result"]["default"]

report = client.call(
    f"{BASE}/rule_finalize@1", kwargs={"evaluated": evaluated},
    trust=True, progress=False,
).value["result"]["default"]

model = report["model"]
print(pd.DataFrame(report["rules"])[[
    "rule", "predicted_class", "development_n"
]].to_string(index=False))
print("Internal test accuracy:", report["metrics"]["test"]["accuracy"])

The fitted model contains one split:

proline <= 755  → predict other cultivar
proline >  755  → predict cultivar A

The selected configuration belongs to the merge_refit family, but this fit needed no merge. Its base tree has depth one and two leaves. The development counts make its limitations visible immediately:

Rule	Development rows	Cultivar A	Other cultivars
`proline <= 755`	67	1	66
`proline > 755`	39	34	5

Five rows in the upper segment already contradict its prediction. Another positive sits below the threshold. A hard boundary creates a simple decision, not a pure population.

The exact saved expression uses float32(fill_missing('proline', 660)). The median comes from development; it is part of the model, not recomputed for each prediction batch. Wine has no missing measurements, so this run does not establish whether that missing-value policy is useful in practice.

The upper segment's estimated class-1 probability is about 0.858, rather than the raw fraction 34/39. The implementation smooths counts toward the development class proportions. This is not a separately calibrated probability, and certainly not a probability of loan default.

The internal test gives 25 correct predictions out of 27, or 92.6% accuracy. Those 27 rows were not used to select the configuration or fit the final model. They are not subsequently added to training.

Open the external holdout without moving the threshold

Prediction with rule_predict@1 needs only the saved model and measurements. Evaluation with rule_monitor@1 can then add the known answers:

scored = client.call(
    f"{BASE}/rule_predict@1",
    kwargs={"dataset": external_holdout.drop(columns=[TARGET]), "model": model},
    trust=True, progress=False,
).value["result"]["default"]

monitoring = client.call(
    f"{BASE}/rule_monitor@1",
    kwargs={"dataset": external_holdout, "model": model, "target_col": TARGET},
    trust=True, progress=False,
).value["result"]["default"]

predictions = pd.DataFrame(scored["predictions"])
assert predictions["row_position"].tolist() == list(range(len(external_holdout)))
errors = external_holdout.reset_index(drop=True).copy()
errors["prediction"] = predictions["prediction"].to_numpy()
print(errors.loc[errors[TARGET] != errors["prediction"],
                 ["proline", TARGET, "prediction"]].to_string(index=False))
print("External accuracy:", monitoring["quality"]["accuracy"])
print("Confusion matrix:", monitoring["quality"]["confusion_matrix"])

row_position refers to positions in the submitted batch, not the original pandas index. The saved schema retains all 13 measurements, so prediction receives the full feature table even though this fitted tree splits on only one.

The external result is 41 correct predictions out of 45, or 91.1% accuracy. A majority-class prediction would get 30 out of 45 correct. The errors are more informative than that comparison alone:

Known class	Predicted other	Predicted A
Other cultivar	26	4
Cultivar A	0	15

All 15 cultivar A samples were found, but four other samples also crossed the threshold. There were 19 positive predictions, of which 15 were correct: precision was 78.9%, with recall of 100% on these 15 positives. That small denominator matters.

Each point represents an external holdout row. Vertical jitter separates overlapping points; the vertical categories show the known class. Orange crosses mark the four mistakes.

One mistake has proline = 780, just above 755. Moving the threshold to fix it is tempting. It would also use the final holdout to make a modelling decision. We leave the rule unchanged. A revised rule would need a fresh evaluation sample.

For the banking question, the corresponding discussion would be about borrowers incorrectly referred for review and early-arrears cases missed by the segment. The acceptable balance must come from the intended action and its costs; Wine cannot answer it.

Run the whole experiment in one call

We have now seen what each stage contributes. The published numeric_rule_experiment@1 pipeline repeats the entire procedure: contract validation, outer partitioning, the six training stages, external prediction, evaluation and error reporting.

experiment = client.call(
    f"{BASE}/numeric_rule_experiment@1",
    kwargs={"dataset": dataset, "target_col": TARGET},
    trust=True, progress=False,
).value["result"]["default"]

# The one-call and step-by-step experiments must agree.
assert experiment["outer_split"] == outer_split
for key in ("model", "rules", "metrics", "segment_metrics", "split", "options"):
    assert experiment["report"][key] == report[key], key
assert experiment["external_holdout"]["monitoring"] == monitoring

holdout_result = experiment["external_holdout"]
print(holdout_result["n"] - len(holdout_result["errors"]), "/", holdout_result["n"])
# 41 / 45

This is a published workflow, not a locally assembled export. Its partition helper is the same implementation as the independently callable partition stage; its training, prediction and monitoring calls pin the public versions shown above. Both routes produced the same model, rules, selections and metrics in the verification run; candidate execution times naturally differ.

The external holdout is inside the orchestration, but outside model fitting and selection. It reaches the fitted model only for prediction and evaluation. The pipeline returns errors for inspection and makes no attempt to fix them. It downloads its implementation and dependencies as needed and runs them locally, with receipts disabled.

What must change before using this on loans

The modular experiment checks statistical separation for these inputs. It does not certify a banking dataset as free of leakage.

The original find_rules@4 release reproduced these same Wine results, but has a concrete implementation limitation: before its internal split, it checks numeric usability across the whole supplied table. In a verification probe, replacing a proline value only in an internal-test row with text caused the whole feature to be dropped and changed the fitted model. The clean Wine run is unaffected, but this behaviour prevents a general claim that held-out inputs cannot influence preprocessing. The modular validator route preserved training and raised an error when that malformed value reached evaluation. The complete experiment for this article uses a fixed numeric input contract: invalid types and ranges stop execution instead of changing the feature set. Statistical screening then runs only within the training partitions.

There is a second limit in the historical release: find_rules@4 recognizes optional user_id and application_date_utc columns, but invalid metadata can be excluded and the splitting strategy can fall back. Its one-block time gap is a gap between distinct timestamps, not a 90-day outcome-maturity rule. Inspecting split.strategy is necessary, but insufficient for banking validation. The new experiment deliberately declares an IID random-split contract; it does not claim to solve temporal or borrower-aware validation.

For the lending task defined here, the experiment design needs the following explicit controls:

Later, mature evaluation cohorts. Reserve the external holdout chronologically. At every internal validation boundary, allow a training loan only if its full 90-day outcome window and reporting delay have elapsed before that boundary. The evaluation cohort must itself have complete follow-up when scored retrospectively. Sorting application dates alone does not establish either condition.
A policy for repeat borrowers. For an evaluation intended to assess new borrowers, remove borrowers appearing in validation or test from the corresponding training partition, including each CV fold. Record the rows lost to that purge. A repeat-borrower use case needs its own chronological design and past-only histories; group-disjoint evaluation answers a different question.
An approved feature snapshot. Exclude the target, target derivatives, future servicing fields and post-decision updates before modelling. IDs and dates used for splitting are metadata, not predictors. A column name cannot prove when its value was available.
Training-only transformations and choices. Fit imputation, encoding, screening and models inside each training fold. Use development validation to select rules and any operating threshold. Freeze those decisions before opening the final holdout. Do not encode all categories or choose a convenient arrears cut-off after seeing holdout performance.

A bank would also need to examine segment event counts, missingness, changes in customer mix, stability over later periods and the consequences of the resulting action. A short rule is easy to inspect, but may still be unstable, unfair in its effects or operationally unhelpful. Neither predictive association nor interpretability establishes causality.

The useful result of this experiment is a compact, testable decision: one threshold, two segments and four visible external mistakes. Extra conditions did not improve the best cross-validation score here. That gives us a reason to keep the simpler model for this demonstration. In banking, a hard rule earns its place only when the same scrutiny survives the timing of the decision, the formation of the target and evaluation on later loans. If short rules are too weak under those constraints, that is a useful finding too.

When One Tech Stack Isn’t Enough: Orchestrating a Multi-Language Pipeline with Local FaaS

Andrew — Fri, 11 Apr 2025 14:29:25 +0000

Hi, my name is Andrew! I oversee a large-scale online service where our queues (Kafka or similar) process millions of events each day. We have to spot anomalies quickly, track key metrics, and notify the team about critical issues—all in real time. Because data streams in from various systems, it’s never as simple as using one language or tool. Some engineers favor Go for performance and concurrency, others prefer Python for machine learning, and Node.js often wins for handling external integrations (Slack, webhooks, etc.). On top of that, our DevOps folks rely heavily on classic CLI tools like tar and gzip for archiving logs.

At first, everything was scattered across different scripts, containers, and services. It worked, but it got messy—too many handoffs, too many config files, and a constant risk of something breaking when one team decided to update their part of the puzzle. So we decided to unify the entire process in one computational graph. The solution we came up with is a local-first FaaS approach where each “node” can be written in whatever language or tool we want, but everything is orchestrated under a single workflow. Below is how we designed each node in our real-time pipeline, why we chose each language or tool, and how data flows between them via simple JSON adapters.

Node 1 (Rust): High-Performance Ingestion

Why Rust?

We needed maximum throughput and memory safety. Our ingestion service handles hundreds of thousands of events per minute from sources like IoT devices and microservices in our cluster. Rust’s ownership model helps avoid memory leaks and race conditions, which is crucial when your service runs 24/7.

Example Rust code (node_rust_ingest.rs):

use std::fs;
use serde::{Deserialize, Serialize};

#[derive(Deserialize)]
struct IngestConfig {
    pub source_path: String,
}

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Read config from JSON
    let config_data = fs::read_to_string("config.json")?;
    let config: IngestConfig = serde_json::from_str(&config_data)?;

    // Imagine reading from a high-volume stream here:
    let raw_events = vec![
        "temperature:24.5 sensor_id=alpha",
        "temperature:25.1 sensor_id=beta",
    ];

    // Convert the raw events into JSON and save
    let json_str = serde_json::to_string_pretty(&raw_events)?;
    fs::write("raw_events.json", json_str)?;

    Ok(())
}

Input: A JSON file named config.json (handled by an adapter, e.g., adapter_input_json("config.json")).
Output: raw_events.json with the raw events.

In practice, this Rust code might pull real messages from Kafka or a custom TCP socket, but the idea is the same: read something from a stream, then write it out as JSON.

Node 2 (Go): Filtering and Normalization

Why Go?

It’s great for concurrency (goroutines and channels) and compiles into a single binary we can deploy anywhere. This node focuses on quick filtering of raw data, dropping empty or invalid records, and transforming some fields before sending them off for deeper analysis.

Example Go code (node_go_filter.go):

package main

import (
    "encoding/json"
    "fmt"
    "io/ioutil"
    "os"
    "strings"
)

type FilteredEvent struct {
    Valid bool   `json:\"valid\"`
    Data  string `json:\"data\"`
}

func main() {
    // Read raw events JSON from file (adapter does the path resolution)
    rawData, err := ioutil.ReadFile(\"raw_events.json\")
    if err != nil {
        fmt.Println(\"Error reading file:\", err)
        os.Exit(1)
    }

    var events []string
    if err := json.Unmarshal(rawData, &events); err != nil {
        fmt.Println(\"Error unmarshalling JSON:\", err)
        os.Exit(1)
    }

    // Filter out any malformed or empty events
    var filtered []FilteredEvent
    for _, e := range events {
        if strings.TrimSpace(e) == \"\" {
            continue
        }
        filtered = append(filtered, FilteredEvent{
            Valid: true,
            Data:  e,
        })
    }

    // Write filtered events out to JSON
    output, err := json.MarshalIndent(filtered, \"\", \"  \")
    if err != nil {
        fmt.Println(\"Error marshalling JSON:\", err)
        os.Exit(1)
    }

    if err := ioutil.WriteFile(\"filtered_events.json\", output, 0644); err != nil {
        fmt.Println(\"Error writing file:\", err)
        os.Exit(1)
    }
}

Input: raw_events.json.
Output: filtered_events.json with an array of cleaned-up records.

Node 3 (Python): ML Classification

Why Python?

Most data science and ML teams rely on Python’s ecosystem (NumPy, scikit-learn, PyTorch, etc.). It’s easy to iterate quickly and load pretrained models.

Example Python code (node_python_ml.py):

import json

def main():
    # Load filtered events from JSON
    with open('filtered_events.json', 'r') as f:
        filtered_data = json.load(f)

    # Imagine we have a pre-trained model loaded here
    # For illustration, let's just mock predictions
    predicted_events = []
    for item in filtered_data:
        data_str = item['data']
        # Some ML logic here...
        prediction = 'normal'
        if 'sensor_id=beta' in data_str:
            prediction = 'warning'  # as an example

        predicted_events.append({
            'data': data_str,
            'prediction': prediction
        })

    # Save predictions to JSON
    with open('predicted_events.json', 'w') as f:
        json.dump(predicted_events, f, indent=2)

if __name__ == '__main__':
    main()

Input: filtered_events.json.
Output: predicted_events.json with a prediction field for each record.

In a real scenario, you might load a Torch or TensorFlow model and do actual inference. The concept stays the same: read JSON, process, write JSON.

Node 4 (Node.js): External Service Notifications

Why Node.js?

We use it for real-time hooks (Slack notifications, webhooks, etc.), thanks to its event-driven nature and wide range of packages on npm.

Example JavaScript code (node_js_notify.js):

const fs = require('fs');
const axios = require('axios'); // For sending data to a webhook, Slack, etc.

async function main() {
  // Read the predicted events JSON
  const rawData = fs.readFileSync('predicted_events.json', 'utf8');
  const predictedEvents = JSON.parse(rawData);

  // For each event, if it's a warning, send an alert
  for (const event of predictedEvents) {
    if (event.prediction === 'warning') {
      await sendAlert(event.data);
    }
  }
}

async function sendAlert(data) {
  // Example: post to Slack (replace with your actual webhook)
  const slackWebhook = 'https://hooks.slack.com/services/your/webhook/url';
  await axios.post(slackWebhook, {
    text: `Alert! Check sensor data: ${data}`
  });
  console.log('Alert sent to Slack for:', data);
}

main().catch(err => {
  console.error('Error sending alerts:', err);
  process.exit(1);
});

Input: predicted_events.json.
Output: Not necessarily a JSON output (though you could log to a file). Typically, it just triggers external actions.

Node 5 (Shell Script): Archiving Logs

Why Shell/CLI?

Classic tools like tar and gzip are reliable workhorses. There’s no need to rewrite them in some other language. They’ve been proven in production for decades.

Example Shell script (node_shell_archive.sh):

#!/usr/bin/env bash

# We assume raw_events.json, filtered_events.json, predicted_events.json
# are present in the current directory. We'll archive them together.

ARCHIVE_NAME="events_archive.tar.gz"

tar -czf "$ARCHIVE_NAME" raw_events.json filtered_events.json predicted_events.json

echo "Archive created: $ARCHIVE_NAME"

Input: The JSON files we produced in the previous steps.
Output: events_archive.tar.gz, ready to be stored or uploaded to a long-term storage solution.

The Role of Data Adapters

In our setup, each node declares what type of data it expects to read and what it produces. But instead of tying every node to a single format like JSON, we rely on adapters to handle the actual loading and saving of data in the target language. An adapter is just a small function or library call that knows how to read or write a specific data type (files, streams, binary formats, you name it) within a particular runtime.

Example:
- adapter_input_json("filtered_events.json") might parse a JSON file into native objects in Python or Go.
- adapter_output_csv("results.csv") could serialize an array of events into a CSV string and save it.

The beauty of this approach is that each node can seamlessly work with different data representations, as long as there’s an adapter for them. Whether you prefer JSON, CSV, binary blobs, or even specialized formats (like Parquet or Avro), adapters keep the underlying implementation details out of the node’s logic. That way, Rust stays focused on ingestion performance, Python remains laser-focused on ML tasks, and Node.js is free to handle whatever output format suits your external APIs—without forcing the entire pipeline to stick to one rigid standard.

Final Thoughts

By unifying everything into a single computational graph and adopting a local FaaS mindset, we drastically reduced the friction between teams. The Rust folks can keep optimizing throughput without breaking anything downstream, Go developers can refine their filtering logic, Python ML engineers can retrain or tweak models, and our Node.js layer can easily tie into Slack. Even our DevOps guys are happy to keep using battle-tested CLI tools for archives.

Before we set this up, we spent way too much time managing glue code and random integration points. Now, each team can just focus on their own node, with minimal dependencies on everyone else. If you’re dealing with a multi-language environment—or foresee that one tool won’t suffice for your entire pipeline—this sort of local-first FaaS approach might be exactly what you need.

SPL – Multi-Language Pipelines and Personal Mini-FaaS on One Machine

Andrew — Fri, 04 Apr 2025 14:45:37 +0000

Hi, Developers! My First post here, don’t roast me too hard 😅

I’d like to share a pet project my teammate and I have been working on. The core idea is to build a multi-language computational graph that also lets you quickly deploy a mini-FaaS (Function as a Service) platform on your local machine. In other words, you can easily mix and match code from various sources (and even different third-party tools) using a local framework and server. Right now, we’re calling this project SPL (Smart Pipe Lime).

How did this idea come about?

While working on a complex model, we realized we needed to combine code and utilities written in different programming languages, pulled from earlier projects. We had separate database queries, several fundamentally different methods of preprocessing large datasets, a two-stage training process, plus final evaluation and validation of the resulting model.

We considered well-known tools like Airflow, Dagster, and Prefect. However, they felt a bit heavy for some simpler scenarios, and they weren’t ideal for rapid prototyping. Besides, part of our dataset required lower-level processing with C++ rather than standard Python. That’s how the idea for a pet project arose—something that would let us seamlessly bring together code that otherwise wouldn’t play nicely, and also allow us to share our work within the team. Essentially, we had a few key goals:

Build a connected computational graph made up of functions or utilities, regardless of their language or dependencies.
Support both local and remote execution of these graphs (so teams can share their work).
Make it possible to run only part of a graph, keeping the state and results of previous steps, to simplify testing new approaches.

Some implementation details

A computational graph is a directed, connected graph with nodes (functions or utilities) and links between them (inputs and outputs). Each node takes input parameters, performs a specific task, and sends the result along to the next node.

We use a few key terms:

Node: A function that has input and output ports.
Port: A named argument (input port) or a return value (output port).
Artifact: The result of a node (a specific port), which gets cached and passed on.

Our approach involves a framework for a specific language — if we’re running code — plus a server that handles the FaaS side, orchestrates nodes in the graph, and takes care of passing artifacts around correctly.

We chose to build the first version of the SPL framework for Python, since that’s the language we use most. The end result will be a library that lets you intuitively create computational graphs right from a Python notebook.

To manage the graph itself — adding or removing nodes, saving results, and running only certain parts — we decided on a mechanic similar to PyTorch. In PyTorch, you build a model by sequentially adding layers to nn.Sequential(). In SPL, you likewise assemble a graph by adding function-nodes in a simple, easy-to-follow way:

# Example of creating a simple graph
from spl import Graph, step

graph = Graph()

graph.add(step(load_data))
graph.add(step(process_data, params={'method': 'mean'}))
graph.add(step(train_model, params={'epochs': 10}))
graph.add(step(save_results))

Running the graph or individual nodes is also straightforward:

# Run the entire graph
graph.run()

# Run the graph starting from the third step
graph.run(from_step=2)

# Run a specific step
graph.run(step=process_data)

A key feature is the ability to “freeze” the execution environment (including Python and library versions) with a single command:

# Freeze the environment in one go
graph.freeze_environment()

A pocket-sized FaaS

One of the coolest features, in our opinion, is that you can quickly spin up your own mini-FaaS right on your machine. If your computer has internet access, your functions and graphs become instantly available to other users.

Right now, the SPL server supports:

An HTTP API for remotely executing functions.
Import and export of graphs in JSON format.
Task coordination and distributed result caching.
A simple web interface for viewing and editing graphs.

Possible use cases for SPL

Local development: Build graphs and functions you can reuse across different projects without constantly copying code.
Production usage: Keep business logic and infrastructure separate, with easy zero-downtime updates.
Personal FaaS (including a function marketplace): Potentially publish your work for others (including monetization), delivering only results instead of the entire codebase.
Visualizing business processes: The server supports graph rendering and displays input and output ports, which can be handy for high-level project management.

Why am I writing this post?

We’d really love to hear what you think:

Have you faced similar challenges?
Would such a tool be useful for you?
What features would you like to see in a project like this?

We’re excited to discuss these ideas in the comments here or in Telegram https://t.me/SPLime_io!