DEV Community: Wisaroot Lertthaweedech

Building a Resilient API Gateway

Wisaroot Lertthaweedech — Wed, 16 Sep 2026 01:54:44 +0000

Originally published on wisl.dev.

Why Did Peak Traffic Become the Worst Enemy?

Every developer knows the sinking feeling of watching their system buckle under unexpected load. Our API services were experiencing frequent outages during peak traffic, cascading failures that would bring down entire service chains, and frustrated users abandoning our platform. With peak loads reaching 40 requests per second, our traditional monolithic approach was failing spectacularly.

The wake-up call came during a particularly bad incident where a single service failure brought down our entire platform for 6 hours. We knew we had to rethink our architecture fundamentally.

One slow downstream service was enough: requests piled up, timeouts cascaded upstream, and six hours later we were still recovering. The failure of a single component must never take the whole platform with it.

What Does the Queue-Driven Architecture Look Like?

After analyzing our failure patterns, we designed a resilient API gateway system built around three core principles:

Asynchronous processing to handle traffic spikes
Intelligent caching to reduce downstream pressure
Graceful degradation to prevent cascading failures

Architecture Overview

Our solution centers around an API gateway that acts as a traffic coordinator rather than a simple proxy. Here's how the components work together:

Request path: the gateway enqueues into Cloud Tasks, the Firestore cache absorbs repeat queries, and Elasticsearch never sees more than 10 req/s

Component Breakdown

API Gateway: The central orchestrator that receives all client requests and makes intelligent routing decisions based on system health and load patterns.

Queue System (/enqueue & /dequeue): Instead of processing requests synchronously, we enqueue them for asynchronous processing. This creates a natural buffer that absorbs traffic spikes without overwhelming downstream services.

Cloud Tasks Queue: Provides reliable, scalable task processing with automatic retry mechanisms and dead letter queues for failed tasks.

Firestore Cache: Stores Elasticsearch query results for 24 hours, dramatically reducing the load on Elasticsearch by serving frequently accessed data from cache.

Elasticsearch: Our search and analytics engine, protected by rate limiting to ensure it never receives more than 10 req/s, maintaining stability even during traffic spikes.

Which Design Decisions Made the Difference?

1. Embracing Asynchronous Processing

The biggest architectural shift was moving from synchronous request-response to asynchronous processing. When a client makes a request:

The API gateway immediately returns an acknowledgment with a task ID
The actual processing happens asynchronously in the backend
Clients can poll for results or receive webhooks when processing completes

This approach eliminated the direct coupling between client requests and backend processing time, making our system inherently more resilient to load spikes.

2. Smart Caching with Firestore

We implemented a simple but effective caching strategy using Firestore to cache Elasticsearch results for 24 hours. This dramatically reduces the workload on Elasticsearch by serving frequently requested data from cache rather than executing expensive search queries repeatedly.

The cache hit rate improvement alone reduced our Elasticsearch load by approximately 30-40% during peak hours.

3. Rate Limiting to Prevent Cascading Failures

The key insight was that cascading failures occurred when traffic spikes overwhelmed Elasticsearch, causing it to fail and affecting all clients. Our solution was elegant: implement rate limiting at the queue level to cap requests to Elasticsearch at 10 req/s.

This approach provides several benefits:

Elasticsearch stability: By never exceeding 10 req/s, Elasticsearch remains healthy and responsive
Isolated failures: All client requests are queued, ensuring Elasticsearch never gets overwhelmed. In the original approach, traffic surges would cause Elasticsearch to fail completely, bringing down all clients. Now, all clients continue to work (albeit potentially slower due to queuing) rather than experiencing complete failure
System-wide protection: No single client can bring down the shared Elasticsearch instance

What Were the Results?

The impact was immediate and measurable:

Downtime dropped by 70% (from 6 hours to roughly 2 hours of recovery for affected clients), and the platform stayed up through every peak-traffic event since.

70% reduction in downtime through graceful degradation and fault isolation (reduced from 6 hours to 2 hours recovery time for affected clients)
Consistent performance even during peak loads of 40+ req/s
Zero cascading failures through intelligent rate limiting that protects Elasticsearch
Graceful degradation: all clients continue working (potentially slower) instead of complete failure
Improved user experience with faster response times due to intelligent caching

What Would We Do Differently?

The Good

Queue-based processing was a game-changer for handling unpredictable traffic
Rate limiting at the queue level provided excellent protection against cascading failures
Simple but effective caching with 24-hour TTL dramatically reduced Elasticsearch load

The Challenges

Complexity: Managing asynchronous workflows requires careful error handling and monitoring
Debugging: Tracing requests through multiple services became more complex
Consistency: Ensuring data consistency across cache layers required thoughtful design

Building for the Real World

Creating resilient systems isn't about perfect code: it's about understanding that failure is inevitable and designing accordingly. Our API gateway system proves that with thoughtful architecture, you can build systems that not only survive peak loads but actually perform better under pressure.

The 70% downtime reduction wasn't just a number; it translated to happier users, fewer 3 AM emergency calls, and a development team that could focus on features instead of firefighting.

Remember: the best system design is one that gracefully handles the unexpected. Plan for failure, embrace asynchronous patterns, and always have a fallback plan.

Smarter Python Configs: an OOP Framework

Wisaroot Lertthaweedech — Mon, 14 Sep 2026 03:17:49 +0000

Originally published on wisl.dev.

Update: this framework grew into bakefile, a task runner: the same base, service, and instance inheritance, rearchitected and published as a library. The story behind it: Developer Workflows Need an Abstraction.

Managing configurations across multiple environments and services is one of those things that should be simple, but rarely is.

If you've worked with .env files, you know the drill: every environment (e.g., sit, uat, canary, prod) has its own set of config files, and every service (be it Cloud Run, Dataflow, or others) duplicates a large portion of the same values. Add CI/CD pipelines into the mix, and the friction becomes obvious. Slight changes in config require regenerating .env files, updating deployment scripts, and often maintaining separate CI/CD logic per service and per environment.

This setup creates redundancy and slows down delivery.

So I built an OOP-based configuration framework in Python, built on top of Pydantic, to simplify and centralize configuration management. Here's how it works.

Why do configs get so redundant?

In most systems, we deal with multiple environments, such as sit (system integration test), uat (user acceptance test), canary, and prod (production).
Some teams might use different naming conventions like dev, qa, stage, or preprod, but the core challenge remains the same: Each environment requires its own config setup, often with small differences.

Additionally:

Each service (e.g., Dataflow job, Cloud Run service) needs its own config.
A large portion of the config values are shared (Docker registry URLs, GCP credentials, etc).
CI/CD pipelines often need to be duplicated to reflect these separate configurations.

That's a lot of repetitive work for something that's logically hierarchical and composable.

The tell-tale smell: a config change that touches three environments means three .env edits, three deployment-script checks, and three chances to get it wrong. Hierarchy collapses all of that into one edit at the right layer.

What does a layered config look like?

Layered composition: the base layer holds shared values, service and instance mixins add specifics, and the environment mixin flips the switches

Instead of treating configs as raw key-value pairs in .env, I defined them as Python classes, layered to reflect how configs are structured in reality.

The framework is built on Pydantic's BaseModel for type-safe config validation, and Object-Oriented Programming (OOP) to encourage composition and reuse.

Let's walk through the layers.

Base Layer: `BaseConfigs`

Contains shared, global settings like:

env: environment identifier (e.g., sit, prod)
is_localhost: true/false for local development
logging_level_per_module
Local GCP credentials path

Also provides template methods such as:

setup_environment()
deploy()
assert_deploy()

Each environment-specific mixin (e.g., ProdConfigsMixin) will override env = "prod" and anything else environment-specific.

Service Layer: e.g. `DataflowConfigsMixin`, `CloudRunConfigsMixin`

Contains service-specific variables:

For example, in DataflowConfigsMixin:

dataflow_job_name
dataflow_gcp_project
dataflow_subnetwork

Also contains service-specific behavior like:

deploy() method for triggering Dataflow job
assert_deploy() logic tailored to Dataflow logs or job status

For a Cloud Run-based service, you'd swap in CloudRunConfigsMixin.

Instance Layer: e.g. `DataflowJobAConfigsMixin`

Overrides any instance-specific config:

dataflow_job_name = "dataflow_job_a"
assert_deploy() might validate output in BigQuery or check an API result

Each job or service instance gets its own mixin, where you specialize the behavior.

How do the pieces come together?

Here's how all the pieces assemble in configs.py:

sit_configs = SitCommonConfigs()
uat_configs = UatCommonConfigs()
canary_configs = CanaryCommonConfigs()
prod_configs = ProdCommonConfigs()

configs = get_configs(
    prod_configs=prod_configs,
    canary_configs=canary_configs,
    uat_configs=uat_configs,
    sit_configs=sit_configs,
)

get_configs() returns the config object for the current environment, determined by a single ENV variable in .env.

Example of an environment-specific class:

class ProdCommonConfigs(
    DataflowJobAConfigsMixin,
    ProdDataflowConfigsMixin,
    DataflowConfigsMixin,
    ProdConfigsMixin,
    BaseConfigs,
):
    pass

What does the CLI give us?

We provide a small CLI tool (ac, short for Abacus Configs) with a few essential commands:

ac echo_configs

Outputs all resolved configs for the current environment:

> ac echo_configs
export ENV=sit
export IS_LOCALHOST=True
export DATAFLOW_GCP_PROJECT=production-project
...

You can use this in shell sessions or pipelines with:

> eval $(ac echo_configs)

This means you can dynamically generate environment variables from your Python config at runtime: no more handcrafting .env files for CI/CD.

eval $(ac echo_configs) inside a pipeline step means the pipeline itself carries zero environment-specific values. The same YAML deploys sit, uat, and prod: only the ENV variable changes.

ac deploy

Triggers the appropriate deploy() method for the current config. Could be deploying a Dataflow job or pushing to Cloud Run, depending on the config object's class structure.

ac assert_deploy

Runs validations defined in the config class:

Check for logs
Confirm data landed in a target system
Make an API call to confirm behavior

This fine-grained control is often needed to test service correctness post-deployment.

Why does this work?

Because all config objects implement the same interface (deploy, assert_deploy, etc.), CI/CD pipelines no longer need to be duplicated per service or environment.

Here's what changes:

Environment-specific behavior is encoded in the config object
CI/CD pipelines can simply run ac deploy and ac assert_deploy in the right environment
Adding a new service or environment requires only a new config class, not a whole new pipeline

On top of that:

Zero duplication: shared logic and values are defined once, and reused through inheritance
Type-safe: configs are validated early using Pydantic
Flexible overrides: environment, service, and instance layers allow for precise customizations
Testable & extensible: logic like deploy() can easily be unit tested or extended
Unified CI/CD: one pipeline logic to rule them all

Where to take it next

This configuration system has significantly reduced overhead for our dev teams. It's made configs declarative, validated, and reusable, and has unified deployment across environments and services.

If you're finding yourself copying .env files, duplicating CI pipelines, or fighting config drift, this might be worth exploring in your Python projects.

A Robust CI/CD Pipeline: Dev to Prod

Wisaroot Lertthaweedech — Sun, 13 Sep 2026 11:06:33 +0000

Originally published on wisl.dev.

In modern software development, CI/CD is more than a luxury: it's a necessity. It ensures consistent quality, security, and speed. Over the past year, I led the initiative to implement a full CI/CD pipeline in our team. We focused on enforcing best practices, automating everything we could, and ensuring developer experience wasn't sacrificed along the way.

Let me walk you through the components of our CI/CD pipeline: how each stage fits into the development lifecycle, the tools we chose (and why), and some hard-earned lessons we picked up along the way.

CI/CD Platform: We use GitHub Actions to orchestrate our CI/CD workflows. Its tight integration with GitHub, support for reusable workflows, and built-in secret management made it a natural choice for our team.

CI: Continuous Integration

CI runs every time a pull request (PR) is opened. It's our safety net, catching problems early before they hit production.

1. Secret Scanning with Gitleaks or GitGuardian

First, we run Gitleaks and GitGuardian to scan for secrets accidentally committed to the repo: API keys, tokens, credentials, etc. This has saved us multiple times. Mistakes happen, but automation makes sure they don't make it to main.

A leaked credential in a public repo gets scraped by bots within minutes, long before a human reviewer would notice it. Secret scanning is the one CI stage that should block a merge outright, every time.

2. Security & License Scanning with Trivy

We use Trivy to scan for vulnerabilities in:

Our dependencies (including transitive ones),
Docker images,
IaC misconfigurations,
And even open-source licenses (to avoid legal issues).

It's comprehensive, fast, and runs directly in our CI.

3. `pre-commit` Checks

Even though we require developers to run pre-commit locally, CI acts as the last line of defense. We rerun it to catch cases where it might not have been installed or configured properly. Our hooks include:

Code formatting,
Trailing whitespace cleanup,
Linting,
And other team-specific checks.

4. Unit Tests

We run unit tests using pytest, verifying each function does what it's supposed to. Fast feedback here ensures contributors don't break small pieces of logic.

5. Integration Tests

Integration tests combine multiple components together: database + business logic, service A + service B, etc. We run these in isolated environments to simulate real-world conditions.

6. Pre-Release Versioning

Before merging to main, we generate a pre-release version of the service or package. This helps us test in staging environments without affecting the production release stream.

We follow a structured format for the version:

1.2.3-rc.4.dev.5+branch.name.timestamp.commit.hash

This includes:

The base version it's targeting (e.g., 1.2.3)
rc (release candidate) or dev indicator
A unique suffix with the branch name, timestamp, and commit hash for traceability

These versions are automatically bumped and tagged in CI, then deployed to non-prod environments or published to a test PyPI index (like an internal Google Artifact Registry). This allows QA and downstream services to test changes before the official semantic release happens in CD.

7. E2E & Pre-Release Testing

This is where things get real.

We deploy the service to a dedicated non-production environment that connects to actual downstream and upstream services. It's the closest simulation to production, giving us confidence that everything works end to end: APIs, data flow, infrastructure, and third-party integrations.

For Python libraries or internal packages, we generate and publish pre-release versions.

These pre-releases are deployed or installed in staging environments for end-to-end testing, without polluting the main release channel.

It's our last gate before merging to main and triggering the real release.

8. Code Quality with SonarCloud

We analyze every PR with SonarCloud, which gives insights into:

Maintainability,
Security hotspots,
Code duplication,
Reliability,
And test coverage.

It's a great way to spot tech debt before it grows.

9. Semantic Pull Requests

We enforce Semantic PR titles like:

feat: add new user registration API
fix: resolve login timeout issue

This enables semantic release (more on that below) to automate versioning and changelog generation.

CD: Continuous Deployment

CD kicks in after a PR is merged into the main branch. This is where the code goes live.

1. Semantic Release

We use semantic release to automatically bump version numbers based on the PR title:

feat: → minor bump
fix: or others → patch bump
feat!: or BREAKING CHANGE: in footer → major bump

This removes the need to manually edit version files or changelogs.

Once released, the service or package gets a clean, production-ready version like 1.2.3. This version is free of any -rc, .dev, or build metadata, ideal for production deployments or publishing to a package registry.

2. Production & Non-Production Deployments

Once versioned, we:

Deploy the service to production and non-production environments.
Or publish the package with a clean version tag like 1.2.3.

Everything is traceable and reproducible.

3. Gitleaks & Trivy (Again)

We rerun Gitleaks and Trivy post-merge to double-check for secrets or new vulnerabilities. It's a belt-and-suspenders approach, and it's worth it.

Continuous Reruns & Dependency Updates

Security isn't a one-time thing. Even "safe" dependencies can become vulnerable tomorrow.

Trivy's database updates daily, so we schedule it to rerun periodically to catch new issues in our existing dependencies.
Tools like Renovate or Dependabot help us auto-update stale dependencies. Older packages are prime targets for exploits, so we keep our stack fresh.

Final Thoughts

CI/CD is never truly "done." It evolves with your team and your product. But the foundation we've built (security-first, automated, and feedback-rich) gives us confidence with every change we ship.

If you're just getting started, start small. Automate one thing at a time. But don't wait too long: your future self (and your team) will thank you.

If you adopt only three stages, make them secret scanning, unit tests, and semantic PR titles. That trio alone buys you most of the safety, and the semantic titles unlock automated releases later without reworking anything.

Let me know how your CI/CD journey is going. I'd love to hear what's worked for you.

A Scalable ML Framework with Monadic Design

Wisaroot Lertthaweedech — Sun, 13 Sep 2026 08:13:03 +0000

Originally published on wisl.dev.

In the world of machine learning, going from research to production is often a painful, time-consuming process. As a developer and ML practitioner, I've personally felt this friction: juggling multiple libraries, inconsistent data formats, fragile pipelines, and the perpetual anxiety of things breaking in production.

To solve this, I built a highly scalable, Python-based machine learning framework that streamlines the entire ML lifecycle, from exploration to deployment, using monadic design principles to bring structure, composability, and reliability to the process.

Here's how it worked and what I learned.

The Problem: ML Pipelines Are a Jungle

A typical ML project might involve:

Scikit-learn for preprocessing and models
TensorFlow or PyTorch for deep learning
Optuna for hyperparameter tuning
Imbalanced-learn for dealing with skewed data
XGBoost for gradient boosting
Seaborn / Matplotlib for visualization

Each tool is great on its own, but stitching them together into a consistent, maintainable workflow? Not so much.

Worse, when it's time to deploy, you often end up rewriting large chunks of code, manually fixing bugs due to unexpected inputs, or patching over pipeline inconsistencies with brittle logic.

The Vision: Composability + Reproducibility + Resilience

I set out to build a research-to-production machine learning framework with three goals in mind:

Composable Components: Each ML step should be a plug-and-play module.
Reproducible Pipelines: From notebooks to deployed APIs with zero drift.
Production-Grade Safety: No critical crashes from schema mismatches or malformed data.

To achieve this, I drew inspiration from functional programming: specifically, monads.

Monads in ML: Chaining State with Context

In functional programming, a monad is a design pattern that wraps values with context (like logging, errors, or side effects) and allows transformations to be chained without losing that context.

In this ML framework, I designed a custom monadic pattern that revolves around two key entities:

1. DataPod: The State Carrier

The DataPod object acts as the context holder: it contains:

All relevant datasets (e.g., main, support_df, etc.)
Intermediate results and derived features
Any research-time variables (e.g., train/test splits, configuration flags)
Metadata used across the ML lifecycle

As the pipeline evolves, the DataPod flows from one transformer to the next, getting updated with new data or attributes while keeping the full research state intact.

2. Transformer: The Behavior Capsule

Each Transformer is a composable, stateful function that:

Transforms the DataPod (e.g., scaling, encoding, feature engineering)
Stores any trained variables inside itself (e.g., mean, std for scalers; trained models)
Can later be serialized and used for production inference or pipeline reproduction

This separation of data (in DataPod) and behavior (in Transformer) allows clean chaining of transformations, while also making the pipeline reproducible and deployable.

The flow looks like this: a DataPod (data + state) passes through a series of transformers, each of which learns something during fit and leaves its footprint behind. After the chain completes, the accumulated footprints list is what gets replayed in production.

How It Looks in Code: A Step-by-Step Breakdown

Let's walk through how the monadic pattern works in this ML framework using simplified Python code.

Step 1: Define your data container

This is the core monadic context. DataPod holds all the data and shared state passed from one transformer to the next.

class DataPod:
    def __init__(self, dfs):
        self.dfs = dfs  # Dictionary of dataframes (main, support, etc.)
        self.metadata = {}  # Optional: Store any global metadata or pipeline state
        self.footprints = []  # Track the sequence of transformers used

    def fit_transform(self, transformer):
        # Call transformer's fit_transform method and pass self (the DataPod)
        transformer = transformer.fit_transform(self)
        self.footprints.append(transformer)  # Record the transformer
        return self

Step 2: Define a base Transformer interface

Each transformer is a self-contained unit that holds any trained variables (e.g., mean, model) and knows how to transform a DataPod.

class TransformerA:
    def fit_transform(self, dp: DataPod):
        # Learn from the data
        self.mean_val = dp.dfs["main"]["feature1"].mean()

        # Optionally store the learned state for deployment
        dp.metadata["mean_val"] = self.mean_val

        return self  # Important: Return self to store in footprints

    def transform(self, dp: DataPod):
        # Apply transformation using learned state
        dp.dfs["main"]["feature1_scaled"] = dp.dfs["main"]["feature1"] / self.mean_val
        return dp

Step 3: Compose the pipeline

You create a DataPod with your raw data and apply transformations in a chainable, declarative way:

import pandas as pd

# Example input data
main_df = pd.DataFrame({"feature1": [100, 200, 300], "target": [1, 0, 1]})
support_df = pd.DataFrame({...})  # Optional

# Initialize the data container
dfs = {"main": main_df, "support_df": support_df}
dp = DataPod(dfs=dfs)

# Compose the pipeline with transformers
dp = (
    dp.fit_transform(TransformerA())
    .fit_transform(TransformerB())
    .fit_transform(TransformerC())
)

# Access outputs
print(dp.dfs["main"].head())
print(dp.metadata)

One of the powerful aspects of this design is the ability to easily compose and reuse sequences of transformations during the research phase. The Serializer class is essentially a convenient way to chain multiple transformers together into a single reusable pipeline, enabling you to apply all transformations in order without repeating code.

Here's the Serializer class that applies a list of transformers sequentially:

class Serializer:
    def __init__(self, transformers):
        self.transformers = transformers

    def transform(self, dp: DataPod):
        for transformer in self.transformers:
            dp = transformer.transform(dp)
        return dp


pipeline = Serializer(
    transformers=[
        TransformerA(),
        TransformerB(),
        TransformerC(),
    ]
)

dp = dp.fit_transform(pipeline)

Step 4: Reproduce and Deploy the Pipeline with Trained Transformers

One of the key benefits of this monadic design is that each Transformer stores its trained parameters internally (e.g., learned model weights, scaling factors). This means the entire pipeline can be reproduced exactly for deployment, ensuring consistency between research and production environments.

One detail worth calling out: the footprints list is the single artifact you ship. It holds the fitted transformers in application order, so research and production run byte-identical logic with no export/import step in between.

After training, your DataPod keeps a record of all applied transformers in dp.footprints. This list acts as a serialized artifact capturing the entire pipeline's state.

To deploy the pipeline on new production data, you simply:

Initialize a fresh DataPod with the production dataset.
Apply the saved transformers (the pipeline footprint) to the new data, using their stored trained parameters.

Here's how it looks in code:

# Assume dp is the trained DataPod from research with footprints saved
pipeline = dp.footprints  # List of trained Transformer instances

# Initialize DataPod with new production data
dp_prod = DataPod(dfs=data_prod)

# Sequentially apply each trained transformer (using stored trained vars)
dp_prod = dp_prod.transform(pipeline)

# Now dp_prod contains transformed production data ready for inference or downstream tasks

Features at a Glance

The framework grew to support a wide range of ML tasks out of the box:

Train-Test Splitting, Imputation, Encoding
Upsampling, Resampling, Cross-validation
Supervised/Unsupervised Learning, Deep Learning
Neural Networks, Recommendation Systems
Natural Language Processing (NLP)

It also included built-in error handling to catch and adapt to common production-time issues, like incompatible data types, schema mismatches, or missing fields, without halting execution.

Results and Benefits

Across our internal projects, the framework cut research-to-deployment time roughly in half and eliminated the "works in the notebook, breaks in production" class of incidents entirely.

50% faster development time for research-to-deployment pipelines
Reproducible pipelines across dev, QA, and production environments
Zero critical incidents in production due to robust pipeline design
Scalable architecture that allows team members to add or swap transformers easily

Final Thoughts

What started as a developer's frustration turned into a powerful internal ML framework, unifying machine learning best practices with composable software design.

Using monads might seem abstract at first, but they offer real, pragmatic value in ML engineering: allowing you to build predictable, traceable, and extensible pipelines that scale from experiment to production without rework.

If you're tired of rebuilding pipelines for every use case or firefighting deployment issues, this architecture may be the shift you need.

DEV Community: Wisaroot Lertthaweedech

Building a Resilient API Gateway

Why Did Peak Traffic Become the Worst Enemy?

What Does the Queue-Driven Architecture Look Like?

Architecture Overview

Component Breakdown

Which Design Decisions Made the Difference?

1. Embracing Asynchronous Processing

2. Smart Caching with Firestore

3. Rate Limiting to Prevent Cascading Failures

What Were the Results?

What Would We Do Differently?

The Good

The Challenges

Building for the Real World

Smarter Python Configs: an OOP Framework

Why do configs get so redundant?

What does a layered config look like?

Base Layer: BaseConfigs

Service Layer: e.g. DataflowConfigsMixin, CloudRunConfigsMixin

Instance Layer: e.g. DataflowJobAConfigsMixin

How do the pieces come together?

What does the CLI give us?

Why does this work?

Where to take it next

A Robust CI/CD Pipeline: Dev to Prod

CI: Continuous Integration

1. Secret Scanning with Gitleaks or GitGuardian

2. Security & License Scanning with Trivy

3. pre-commit Checks

4. Unit Tests

5. Integration Tests

6. Pre-Release Versioning

7. E2E & Pre-Release Testing

8. Code Quality with SonarCloud

9. Semantic Pull Requests

CD: Continuous Deployment

1. Semantic Release

2. Production & Non-Production Deployments

3. Gitleaks & Trivy (Again)

Continuous Reruns & Dependency Updates

Final Thoughts

A Scalable ML Framework with Monadic Design

The Problem: ML Pipelines Are a Jungle

The Vision: Composability + Reproducibility + Resilience

Monads in ML: Chaining State with Context

How It Looks in Code: A Step-by-Step Breakdown

Step 1: Define your data container

Step 2: Define a base Transformer interface

Step 3: Compose the pipeline

Step 4: Reproduce and Deploy the Pipeline with Trained Transformers

Features at a Glance

Results and Benefits

Final Thoughts

Base Layer: `BaseConfigs`

Service Layer: e.g. `DataflowConfigsMixin`, `CloudRunConfigsMixin`

Instance Layer: e.g. `DataflowJobAConfigsMixin`

3. `pre-commit` Checks