Polyglot persistence in one pipeline: MySQL, Mongo, SQLite & Cassandra.

Modern data stacks are polyglot. Your pipeline shouldn't look like an unmaintainable monster just because it bridges SQL, Mongo, and SQLite. wpipe-steps provides unified primitives for all of them.

Here is how you use MySQL, Mongo, SQLite & Cassandra Steps in a production pipeline with wpipe-steps:

from wpipe import Pipeline
from wpipe_steps.database import MySQLQueryStep, MongoInsertStep, SQLiteAuditStep

pipeline = Pipeline(pipeline_name="polyglot_sync")

pipeline.set_steps([
    # 1. Query legacy transactional relational DB
    MySQLQueryStep.as_step(
        name="read_orders",
        query="SELECT id, customer, amount FROM orders WHERE synced = 0",
        response_key="pending_orders"
    ),
    # 2. Insert into document store
    MongoInsertStep.as_step(
        name="sink_to_mongo",
        database="ecommerce", collection="orders",
        data_key="pending_orders"
    ),
    # 3. Write local SQLite audit record
    SQLiteAuditStep.as_step(
        name="audit_run",
        db_path="./audit.db",
        action="SYNC_ORDERS"
    )
])

pipeline.run({})

Why developers love wpipe-steps:

• 196 cataloged steps covering Redis, ClickHouse, MySQL, WAF, S3, Docker, and local HuggingFace AI.
• Lazy-loading imports for instant sub-100ms startup times.
• Clean .as_step() factory interface.

Check out the full repository on GitHub!

Comments

[William Rodriguez]

Polyglot persistence often introduces significant serialization overhead across boundaries. How do you handle schema synchronizations when chaining SQL and NoSQL engines in high-throughput pipelines? What patterns have proven most reliable in your production setups?