DEV Community: Mee Mee Alainmar

Simple Queue Can Save Your Pipeline: DuckDB + Python

Mee Mee Alainmar — Wed, 08 Apr 2026 15:20:43 +0000

If you’ve ever tried to shove millions of rows into a database, you know the pain, slow inserts, blocked threads, and that sinking feeling when your pipeline just can’t keep up.

I recently worked on a minimal pipeline design that tackles this problem head-on, and I think it’s worth sharing, especially if you’re building a bronze layer for data lake.

For small to medium scale workloads (think tens of millions of rows), this pipeline is enough. It’s simple, easy to maintain, and doesn’t require spinning up distributed infrastructure. DuckDB is surprisingly capable here.

You can access the repo here: https://github.com/meemeealm/Multithreaded-Ingestion-Pipeline.git

The Idea
The pipeline is built around a producer-consumer model. Instead of one big monolithic process, we split responsibilities:

Producer (Shredder Thread): Reads Parquet rows, batches them, and pushes them into a queue.
Queue (Thread-safe buffer): Acts as the middleman. It smooths out the flow and prevents the producer from overwhelming the consumers.
Consumers (Workers): Multiple threads pull from the queue, cast types, batch inserts, and push data into DuckDB.
DuckDB: Landing zone. Everything fans in here, into a raw_data table.

So for a nice fan-out → fan-in pattern: one producer, many workers (3 workers in example), one database.

Queue Pattern

The critical part of this pipeline is queue. It also provides backpressure control, meaning, if workers are slower, the queue absorbs the imbalance.

The producer fills the queue with batches of rows, but it’s the workers that show the queue pattern in action — they keep pulling jobs out, processing them, and marking them done. This is the classic producer–consumer setup.

def db_ingestor(worker_id, job_queue, shared_con, batch_size):
    con = None
    try:
        # Each worker gets its own cursor from the shared connection
        con = shared_con.cursor()
        print(f"[Worker - {worker_id}] Ready and waiting for data...")

        batch = []
        while True:
            row = job_queue.get()  # Pull next job from the queue
            try:
                if row is None:  # Poison pill → stop signal
                    if batch:
                        con.executemany("INSERT INTO raw_data VALUES (?, ?, ?)", batch)
                    break

                # Cast row values into proper types
                clean_row = (
                    int(row['id']), 
                    float(row['value']), 
                    str(row['category'])
                )
                batch.append(clean_row)

                # Insert batch when it reaches the limit
                if len(batch) >= batch_size:
                    con.executemany("INSERT INTO raw_data VALUES (?, ?, ?)", batch)
                    batch = []

            finally:
                # Mark the job as done so the queue knows it’s processed
                job_queue.task_done()

Why This Works for Bronze Layer

The bronze layer is all about raw ingestion. Here, fancy transformations is not required yet — the most critical job is to just get the data in reliably.

This design is good for -
Parallelism: Multiple workers keep inserts flowing without bottlenecks.
Decoupling: Producer and consumers don’t block each other. If one side slows down, the queue absorbs the shock.
Batching: Both reads and writes are batched, which makes inserts way faster.

Limitations

Python GIL: If your workers are CPU-heavy, threading won’t scale in long run. Multiprocessing or native extensions might be needed.

Single-node DB: DuckDB is designed to run on a single machine, so if your workload grows beyond what one node can handle, you’ll eventually need to switch to a distributed system

Error handling: In bronze layer, usually it doesn’t demand perfection, but you’ll still want retries and logging (which is not in this pipeline).

From Data to Deployment: My Journey in the Google Cloud Gen AI Academy APAC 🚀

Mee Mee Alainmar — Tue, 17 Mar 2026 13:01:52 +0000

I recently joined the Gen AI Academy APAC, powered by Google Cloud and Hack2Skill, and it has been a fruitful experience for me.
So if you’re in the APAC region and looking to move beyond Jupyter Notebooks into production-grade AI in Google Cloud Environment, here’s why this program should be on your radar.

It’s a zero-cost, challenge-based program designed to upskill developers across the Asia-Pacific region. Unlike a standard video course, this is a "Build-First" environment. We aren't just learning theory; we are using Vertex AI, Cloud Run and AI-ready databases to solve real-world problems.

For practicing codelab session, the program provides the credits to practice the labs smoothly.

There are three tracks participants can pick: one or all.

Track 1- Build and deploy AI agents using Gemini, ADK, and Cloud Run
Track 2 - Connect AI agents to real-world data and tools using Model Context Protocol (MCP)
Track 3 - Build AI-powered applications using AI-ready databases like AlloyDB

FAQs about this program: Link

From My Kitchen to Yours: I Built a Food Swap Board for My Neighborhood using streamlit and sqlite3

Mee Mee Alainmar — Sun, 01 Mar 2026 22:23:31 +0000

This is a submission for the DEV Weekend Challenge: Community

💁🏽‍♀️ The Community

In my part of the world (Southeast Asia), we have a beautiful culture of "အိမ်နီးနားချင်း", "Gotong Royong" or "Bayanihan." It’s that unwritten rule that if your mango tree is overflowing, you don’t let the fruit rot, you pass it over the fence to your neighbor.

This culture is still alive in some parts of rural areas. However, in our modern, fast-paced neighborhoods, we’ve lost that "over the fence" connection. I built this app to bring that village-style sharing into the digital age.

🥔 What I Built

NextDoor is a lightweight, real-time bulletin board for food sharing. It’s a "zero-waste" tool that lets neighbors post surplus garden harvests, pantry extras, or home-cooked meals.

The Goal is to make sure no wasted food happens in my home or in our neighborhood. And there is also direct-to-WhatsApp integration that skips the "official" sign-up friction and gets neighbors talking instantly.

🏘️ Community-First Features

Regional Pantry Sharing Specifically designed for the South East Asian kitchen. Whether it's extra Kangkong from the garden, or fresh Lime, the board is built to showcase the flavors of our local community.
Reliable "Always-On" Board Unlike temporary chat groups where messages get buried, this digital board keeps a permanent record of what’s available. The post stays visible until the item is gone, ensuring nothing in the neighborhood goes to waste.
Instant WhatsApp Coordination One tap on the green message button opens a direct chat with the owner, complete with a pre-written message asking if the item is still available. It’s the fastest way to say "I'm coming over!"
Neighbor-Wide "Claim" System When a neighbor clicks "Claim," the item is instantly marked as Reserved for everyone viewing the board. This prevents three different people from showing up at your door for the same bag of mangoes.
Smart Pantry Discovery Looking for something specific like Eggs or Basil? The search and category filters let you skip the scroll and find exactly what you need in seconds, whether it’s a cooked meal or garden herbs.
Clean, Color-Coded Interface The board is designed for clarity. Available items are bright and clear, while reserved items are neatly crossed out. The color-coded buttons make it obvious how to reach out or how to grab a spare portion of food.
Effortless Posting A simple sidebar form allows anyone to list an item in seconds. Just type what you have, how much is available, and your contact info—the board handles the rest, keeping the neighborhood fed and connected.

🌍 Community Impact: Why this Matters

Building a neighborhood food swap isn't just about sharing, it's about creating a sustainable ecosystem right at your doorstep. Here is how NextDoor transforms our local community:

Zero-Waste Neighborhoods Every bunch of Kangkong or bowl of Pancit shared is one less item ending up in a landfill. By connecting neighbors with surplus food to those who need it, we turn potential waste into shared meals.
Food Security at the Hyper-Local Level Inflation and rising food costs affect us all. NextDoor provides a safety net where neighbors can support one another, ensuring that fresh produce and home-cooked meals remain accessible within the community.
Strengthening Social Bonds In a world of digital walls, this platform encourages real-world interaction. Meeting a neighbor at the gate to hand over fresh Cauliflower builds trust and fosters a sense of belonging that traditional social media often misses.
Supporting Local Home-Cooks It provides a stage for home-cooks to share their passion. Whether it’s a surplus of Ube Halaya or a large pot of Curry, the app helps neighbors discover the culinary talents living just a few doors away.
Instant Response for Perishables Because the app is designed for speed (with one-tap WhatsApp links), perishable items like overripe mangoes or freshly cooked rice find a home within minutes, rather than hours.

🎲 Demo

The interface is responsive, making it easy for an Uncle at the wet market or a Tita in her kitchen to post an item in seconds.

Link to streamlit:

nextdoor-food-swap.streamlit.app

💻 Code

meemeealm / next-door-food-swap-app

A community-driven web app built with Streamlit to share surplus food and reduce local waste.

🍎 NextDoor: Neighborhood Food Swapping App

A localized, community-driven web application designed to reduce food waste and strengthen neighborhood ties. This app allows residents to post extra garden harvests, pantry items, or home-cooked meals for their neighbors to claim.

📸 App Preview

| | |

✨ Features

Localized Community Board: A neighborhood-first platform tailored for South East Asian staples—sharing everything from Adobo and Kangkong to Ube and Calamansi.

Persistent SQLite Backend: Robust data management using SQL, moving beyond temporary files to ensure community postings are safely stored and easily retrievable.

Intelligent Search & Categorization: High-performance filtering that allows neighbors to instantly sort through local produce and cooked meals by keyword or regional category.

Seamless WhatsApp Coordination: Direct-to-chat integration via custom-styled API links, featuring pre-filled messages to eliminate friction between neighbors.

Real-time Reservation Logic: A synchronized "Claim" system that updates the database instantly, marking items as "Reserved" across all user sessions…

View on GitHub

🧱 How I Built

I wanted this to be fast, free to host, and easy to use.

Most neighborhood apps feel cluttered. I wanted to build a high-performance, localized "Food Swap" platform using Python, Streamlit, and SQLite.

*🗄️ From Volatile to Persistent: SQLite Integration *
By implementing a relational database (SQLite), the app now handles concurrent data more reliably. I built a dedicated Data Access Layer (database.py) to decouple the SQL logic from the UI, making the app modular and easier to scale.

⚡ Real-Time State Management
Instead of relying solely on Streamlit's Session State, I implemented a Database-Driven State for item claiming. When a user clicks "Claim," it triggers an atomic update in the DB. This ensures the "Reserved" status persists across different browsers and server restarts—no more "phantom" items.

📱 Regional UX: WhatsApp API Integration
In South East Asia, WhatsApp is the "Operating System" for the community. I integrated a custom URL generator that deep-links to WhatsApp with pre-filled URI-encoded strings.

User Flow: Click Button >>> Open WhatsApp >>> Automatic Message: "Hi [Owner], is the [Item] still available?"

🔍 Pandas-Powered Querying
To keep the search "snappy," I used Pandas to handle the filtering layer. By reading SQL queries directly into DataFrames, I can perform complex string matching and category filtering on the fly with minimal latency.

🎨 CSS-in-Python (Scoped Styling)
Streamlit can be visually rigid, so I injected a custom CSS injection layer using unsafe_allow_html. This allowed me to override the standard component styling to create a color-coded action system:
Brand Success: WhatsApp Green (#25D366)
Primary Action: Claim Blue (#007BFF)
Secondary Action: Undo Grey (#6C757D)

Interactive Neighborhood Map

This is a dynamic, color-coded map powered by Pydeck and Mapbox/CartoDB. Instead of scrolling through an endless list, users can instantly visualize the "Supply and Demand" of their specific street.

Technical Highlights:

Color-Coded Layers: Items are categorized by type (Vegetables, Cooked Meals, Fruits) with distinct RGB signatures, allowing for instant visual filtering.

Smart Clustering: Uses a custom-calculated ViewState to automatically center the map based on the geographic mean of all active listings.

Interactive Tooltips: Custom HTML/CSS tooltips provide a "Quick View" of item details and quantities without leaving the map interface.

Faker-Powered Geospatial Data: For development, a custom data_generator.py simulates realistic neighborhood distribution using GPS coordinate randomization within a 3km radius.

Thanks for reading!

What are Snowflake Views? A Simple Guide for Beginners

Mee Mee Alainmar — Tue, 24 Feb 2026 04:06:03 +0000

If you’ve been learning databases and modern data warehouses like Snowflake, you’ve probably heard the word "View".It sounds technical, but it’s actually one of the coolest and easiest tools.

Think of a View as a "Saved Search" or a "Virtual Window" into your data existed in database. It looks like a table, acts like a table, but it’s actually just a saved recipe for a query.

So what exactly is a View?
Imagine you have a giant, messy spreadsheet (a Table) with 100 columns and millions of rows.

In most day, you only need to see three columns: Customer Name, Item Purchased, and Price.
Instead of writing a long code to filter that data every single morning, you create a View.
The View is just a "shortcut" that shows you exactly what you want to see, instantly. And the Table is where the actual data lives.

Benefits of "Views"

It’s a Time-Saver. You write the complex logic in SQL Worksheet once, name it (e.g., Daily_Sales_View), and then you just call that name whenever you need it.
It’s compact. You can hide the "messy" parts of your data and only show the clean, useful bits.
It’s Secure: You can let a co-worker or Business Analyst see a View of the data without giving them access to the other data in the main table (sensitive information like credit card numbers or addresses).

Let's talk about "View" in Snowflake

Snowflake has a few different types, but here are the three you'll hear about most.

Standard Views
The most common. They don't take up any extra space. They just run your query the moment you ask for it.

Secure Views
These are like standard views but with a "privacy shield." They stop people from seeing the secret code inside the view. You can learn more about Secure Views on Snowflake Documentation.

Materialized Views
These are the "pro" version of views. They actually save a copy of the results to make things super fast, but they cost a little extra money because they use storage and compute.

In summary.

Limitations

Snowflake views have limitations regarding performance, data manipulation operations, and structure. Standard views are not pre-computed (can be slow), while Materialized Views have strict limitations, including single-table restriction, no joins, no window functions, and no non-deterministic functions, often leading to higher storage and compute costs.

Key limitations of Snowflake views include-

Standard Views

Cannot be updated via ALTER VIEW , the entire table must be replaced.
They do not cache data, meaning they compute results on every query.

Materialized Views

Cannot join multiple tables, cannot query other views, cannot use ORDER BY, LIMIT, HAVING, or window functions.
Require background compute for maintenance, making them unsuitable for frequently updated data.
You cannot run INSERT, UPDATE, or DELETE directly on materialized views.
Non-secure views may not be directly shared; Secure Views are required for sharing, which can limit query performance optimization.

Therefore, for complex, high-latency queries, Dynamic Tables are often recommended over Materialized Views to overcome these constraints. In professional environments, to overcome these trade-offs, dbt is widely used along with Snowflake. In another words, dbt can manually managed Snowflake views by treating them as "models".

How to Create VIEW

It is simple and straightforward to create Standard "View". For this, you use CREATE VIEW or CREATE OR REPLACE VIEW.

CREATE OR REPLACE VIEW My_First_View AS
SELECT name, email
FROM customer_table
WHERE city = 'Manila';

To make it secure, you just add the word SECURE right before VIEW.

CREATE OR REPLACE SECURE VIEW My_First_Secure_View AS
SELECT name, email
FROM customer_table
WHERE city = 'Manila';

To make it materialized, you add the word MATERIALIZED.

CREATE OR REPLACE MATERIALIZED VIEW My_First_Materialized_View AS
SELECT name, email
FROM customer_table
WHERE city = 'Manila';

Ref: Snowflake Docs