DEV Community: Sebastian Zakłada 🧛

Avoiding the SQS Event Mapping Trap 💥 in Lambda

Sebastian Zakłada 🧛 — Thu, 12 Dec 2024 11:39:23 +0000

tl;dr

Triggering Lambda invocations directly from SQS by the way of event source mapping may lead to messages silently dropped from the queue when event filtering is used

This is something that I kept forgetting so I thought I would put it in writing for my future self to finally remember. You guys can reap all the benefits!

When building event-driven architectures using Lambda and SQS, event filtering can seem like an elegant solution for message routing. If message matches the criteria trigger the lambda, otherwise ignore. Nice! What can go wrong?

When message is sent to an SQS queue, matching messages (decided by filtering) are processed by Lambda. If the processing fails, it's retried and upon reaching maximum retry count it' sent to a Dead Letter Queue (DLQ).

One might think such approach would give the best of all worlds without too much complexity

Resilient - resilience guaranteed by SQS and reliability with DLQ preventing data loss
Elegant - filtering rules provide a clean way to route messages to dedicated lambdas for different message types
Minimalistic - simple infrastructure requirements

You need to be careful though

There's a critical behavior that can lead to unexpected data loss if not properly understood.

Many developers will assume that when a message doesn't match the event mapping filter criteria, it will be put back in the queue for other consumers or will not be picked up from the queue by Lambda at all.

This is not the case

Once a message batch is picked up by Lambda's polling mechanism, messages that don't match the filter criteria are deleted from the queue without being processed by any function.

Yes, you read it right. Messages are PERMANENTLY DROPPED.

Real-World Impact

Consider this scenario:

// Message in SQS queue
{
    "type": "foo",
    "action": "login",
    "userId": "12345"
}

// Filter criteria - Lambda A 
{
    "type": ["foo"],
    "action": ["signup"]
}

// Filter criteria - Lambda B
{
    "type": ["foo"],
    "action": ["login"]
}

If Lambda A polls this message first:

The message is picked up by the Lambda A trigger and made invisible to other consumers
It doesn't match filter criteria
Message is dropped and never put back in the queue
Lambda B never gets a chance to process it, even though it matches its criteria

This behavior can manifest gradually - during testing with low message volumes, messages might get processed correctly by chance. As production traffic increases, concurrent Lambda polling increases and the likelihood of messages being dropped grows with scale and there's no built-in monitoring or alerting for filtered-out messages.

This behavior creates a well-concealed pitfall in your event-driven architecture. While event filtering appears to offer a clean solution for message routing, it is not the case with SQS.

The issue becomes particularly treacherous in systems with multiple consumers where the order of message polling can determine whether a message gets processed or discarded. Without careful consideration, you might end up losing important data without any indication of failure or warning. Even more deceptively, this issue might not surface during initial testing, only to emerge as a serious problem later when message volumes and concurrent processing increase.

Why this is happening

It's important to understand the difference between SQS and other messaging services in AWS

Characteristic	SQS	SNS	EventBridge	DynamoDB Streams Kinesis Data Streams
Type	One-to-one queue	Pure fanout	Event bus	Streaming
Message Processing	Exactly one Lambda processes each message with visibility timeout	Each subscribed Lambda gets its own message copy	Each rule's target gets its own event copy	Each Lambda reads independently from its checkpoint
Filtering Behavior	Filtered messages permanently deleted from queue	Filtered messages silently dropped but still reach other subscribers	Filtered events dropped for that target only	Filtered records stay in stream
Error Handling	Failed messages return to queue until `maxReceiveCount`, then go to DLQ if configured	Failed Lambda invocations go to DLQ if configured, not affecting other subscribers	Failed deliveries retry based on target policy then go to target's DLQ if configured	Failed processing retries until record expires (24h DDB, configurable in Kinesis)

Knowing that, the 🎲 dropped messages behavior becomes clearer when we understand how SQS handles message visibility and processing.

Source: Amazon Simple Queue Service documentation

When a message enters an SQS queue, its lifecycle follows a pattern:

Publisher sends the message to the queue where it becomes available for processing. When a consumer (in this case, Lambda) retrieves the message, it triggers a visibility timeout period during which the message is invisible to other consumers.
During this period, Lambda applies its filter criteria, and if the message doesn't match, it's deleted from the queue - just as if it had been successfully processed.

The key insight is that the message deletion occurs at the SQS level, before your Lambda function even sees the message. This means:

Once a Lambda polls a message batch, the filtering happens during the visibility timeout
Messages failing the filter criteria are deleted immediately
There's no opportunity for other consumers to process these messages

This is particularly important because once the deletion occurs, there's no mechanism to recover or reprocess the message, even if other consumers might have been interested in it.

In other words - the visibility timeout, which normally serves as a safety mechanism to prevent duplicate processing, becomes the window during which messages can be permanently lost if filter criteria don't match.

Unnecessary ambiguity

While not explicitly documented, I've consistently observed this behavior across multiple implementations and test scenarios. Each attempt to work around this limitation led to the same conclusion.

When Lambda receives messages from SQS with an event filter configured, any message that doesn't match the filter criteria is permanently dropped.

Call to Action for AWS

Please, pretty please with sugar on top - make it less confusing. The current behavior should be more explicit in the documentation. While the docs state that non-matching records are "discarded," they should clearly emphasize that messages are permanently deleted from the queue and cannot be processed by other consumers.

Moreover, the current documentation suggests using event filtering to "reduce unnecessary Lambda invocations" by filtering messages with certain data parameters. This guidance is problematic because:

It actively encourages an architectural pattern that can lead to data loss
The suggested use case could be better served by SNS fanout, EventBridge rules, or separate SQS queues

AWS should consider improving the implementation of event filtering for SQS to Lambda integrations to prevent silent data loss while maintaining simplicity. Here are two potential high-level solutions:

1. Make the current filtering behavior configurable through a simple parameter in the event source mapping:

{
  "FilterCriteria": { ... },
  "FilteredMessageBehavior": "DELETE | RETURN_TO_QUEUE"
}

This would allow teams to choose whether non-matching messages should be deleted or remain available for other consumers.

2. Or event better, add support for automatic SQS queue creation when setting up EventBridge rules or SNS topic subscriptions. Instead of manually creating queues for each filter pattern, allow developers to define their routing logic and have AWS automatically manage the underlying queues:

# SNS example
  OrdersTopic:
    Type: AWS::SNS::Topic
    Properties:
      Subscriptions:
        - Protocol: sqs-managed  # New protocol type
          FilterPolicy: 
            type: ["order_created"]
          EndpointProperties:    # Queue will be created automatically
            FunctionArn: !GetAtt OrderProcessor.Arn
            DLQEnabled: true

This would automatically create the necessary SQS queues behind the scenes, while presenting a simple interface to developers.

Currently, developers must choose between using Lambda's problematic event filtering or managing complex queue infrastructure. By automating queue creation while preserving control over event routing, AWS could provide a better developer experience without sacrificing reliability or flexibility.

Until these improvements are made, users should be strongly cautioned against using event filtering with SQS to Lambda integrations.

I am struggling to understand the use case for SQS event filtering

With the current behavior of silent message dropping, SQS event filtering appears to be a solution in search of a problem. The most obvious use cases where filtering might seem appealing actually turn out to be anti-patterns that could be better handled through other AWS services.

For example, trying to route different message types to specific Lambda functions would be better handled by SNS fanout or EventBridge rules, as these services are designed for message routing. Using SQS filtering for this purpose introduces the risk of silent message loss due to the polling order of Lambda functions. Improved resilience in this pattern comes at a cost of having to maintain dedicated SQS queues for each of the filters, which is exactly what an SQS filter misleadingly promises to avoid.

Optimization through reduced Lambda invocations might seem like another potential use case, but this comes at the dangerous trade-off of potentially losing business-critical messages. The minimal cost savings from fewer Lambda invocations rarely justifies the operational risk of missing important events. This would be better handled either through proper capacity planning or if using EB/SNS is not an option, by implementing filtering logic within the Lambda function itself.

Another apparent use case might be implementing priority queues by filtering for high-priority messages first. However, this approach is fundamentally flawed due to the message deletion behavior - lower priority messages could be permanently lost if they're picked up by the high-priority consumer first. Better solutions exist for this scenario.

The current implementation seems to fall into an awkward place - it creates a significant risk of data loss while not providing unique benefits. What's more the event filtering implementation breaks SQS core guarantee of reliable message delivery. It introduces silent failure modes that contradict the typical reliability patterns developers expect.

What's the alternative?

Consider these approaches. Just remember, the best solution will always depend on what you are solving for.

Separate Queues

Instead of using filter criteria, create dedicated queues for different message types. This ensures reliable message delivery at the cost of additional infrastructure.

Message Pre-filtering

Implement filtering logic before messages enter SQS, such as using SNS with multiple SQS subscriptions or EventBridge with dedicated pipes or rules for message routing.

While adding infrastructure complexity, it provides important benefits:

Explicit message routing with reliable delivery
Clear visibility into message flow
Independent scaling for each consumer
No risk of accidental message loss

The required SQS infrastructure provisioning part can be simplified by templating and scripting, creating reusable patterns that make queue management more maintainable.

In-Function Filtering

Process all messages in a single Lambda function and implement filtering logic within your code. While this may result in more Lambda invocations, it guarantees reliability and is a dead simple solution that's easy to understand even for junior developers.

Don't use Queues

For simple workflows triggering Lambda directly from SNS can work well. This approach is suitable when message ordering isn't critical, and event volume is predictable and within Lambda's concurrency limits (no support for buffering, batching and less flexibility in handling traffic spikes).

Wrapping up

After digging into SQS event filtering, it's clear there's more to this feature than meets the eye.

Here's what we've learned:

The Unexpected Behavior - messages that don't match your filters are silently deleted without notification - a concerning end result
Alternative Approaches - several more reliable options exist:
- Separate queues for different message types
- SNS or EventBridge for message routing
- Filtering logic within Lambda functions
- Direct SNS-to-Lambda for simpler workflows

The reality is that sometimes seemingly elegant solutions come with hidden complexities. Understanding these nuances helps us make better architectural decisions, even if it means taking a slightly longer path to get there.

I'm still searching for compelling use cases where SQS event filtering provides clear advantages over traditional approaches. If you've successfully implemented this feature in production or found specific scenarios where it works well, I'd be interested in hearing about your experience in the comments.

Disclaimer

This article is an independent developer guide. All views, experiences and recommendations are my own. Your mileage may vary.
Always refer to the latest AWS documentation for the most current information about service behaviors and limitations. Confirm any assumptions by the way of test implementations.

Help! I Think I Broke DynamoDB – A Tale of Three Wishes 🧞‍♂️

Sebastian Zakłada 🧛 — Thu, 28 Nov 2024 21:12:21 +0000

In my previous article we discussed DynamoDB's dual nature - simultaneously the best and worst database imaginable (talk about split personality!). We also explored some pretty unconventional topics:

🩸 Why DynamoDB is terrible at grocery shopping 🛒
🩸 Medical risks related to working with DynamoDB 👀
🩸 What item count and F1 racetrack 🏎️ have in common

I found that theory alone rarely convinces people about DynamoDB limitations. On the other hand, specific examples sometimes also fail to resonate. That's when I realized what we really need are stories. I must tell tales in which you could see your own mistakes reflecting in other people's experiences.

It's story time!

When collecting article ideas, I came up with a handful of DynamoDB disaster stories that, despite being works of fiction, are almost certainly happening somewhere right now. These are situations where teams pushed DynamoDB too hard to learn that sometimes it's better to stop and think rather than act "smart" and "clever" only to end up feeling defeated and confused.

I took those ideas for a spin and created a short tale that hopefully shines a light on real teams, real projects, and real problems. A tale of what happens when you try to trick DynamoDB into granting your wishes instead of working with its design.

And so it begins...

Three DynamodDB Wishes

All events and characters appearing in this work are fictitious. If you find anything like this actually happening in real life, please contact me immediately for help!

Might I also suggest some appropriate background music...?

We Don't Need Another Database champions

The application design was prominently displayed as a neat diagram on a whiteboard. The team gathered in a room following a meeting with product owners, discussing the usual - new enhancements and quality of life improvements in the app.

It all started as a perfect plan - a hot new startup building a SaaS platform for content creators. Just basic CRUD operations for profiles, projects, and payments. The engineering team had read all the AWS blog posts and designed the most elegant single-table DynamoDB schema you'd ever seen. A beauty to behold. Six months later, it had evolved into something that looked more like a Jackson Pollock painting, but it was still coping - a calculated technical debt. All in all, everything was going pretty well.

The new feature request that they just finished discussing - "What if you could look up freelancers based on their past project history?" - was a tough nut to crack.

– Why can't we just use PostgreSQL? We know SQL! – asked one developer.
– We don't need another database! – declared the Lead – DynamoDB can do everything!
– Look how clean this access pattern is! – the team celebrated – Zero latency, infinite scale, and we barely pay anything. We can't let it be ruined!

That night, as the team huddled around their screens desperately searching for answers, their frustration echoing through Stack Overflow posts, AWS wikis, and GitHub issues, they were still at a dead end. After all, this was their first serverless project since moving away from hosted solutions they'd been using for years until they started using AWS a year ago. In their darkest moment of desperation, someone stumbled upon a hidden AWS forum thread that sparked their attention.

It spoke of a legend - a peculiar AI assistant that could be summoned by asking Amazon Q the right question at exactly midnight (UTC). Not your typical chatbot, this mystical creature was said to have been granting database wishes since the beginning of Epoch time.

The cryptic post described a digital genie 🧞‍♂️ and claimed that as long as you phrased your wishes in valid JSON, all was possible. The message offered surprisingly detailed instructions for summoning it.

They were desperate and getting nowhere despite putting in extra hours. In perfect unison, everyone in the room looked at the clock. It was showing exactly 2024-11-28T23:59:00.000Z.

– This is ridiculous – muttered the Lead, staring at the screen – but I've seen crazier things.
– Have you? This must be the weirdest post I've ever seen. Someone must be having a good laugh. – said one of the developers.
– What's the worst that could happen? – shrugged another, already typing the suggested prompt – Better than explaining to Product why we need two more sprints only to work on some spikes.
– Wait, let me record this – grinned the junior dev, pulling out their phone – You guys are nuts. This thing, you summoning that legendary creature... it will totally blow up on social media!

The words died in their throats as the screens flickered in unison, terminal windows filling with swirling blue text that coalesced into a shimmering form.

Initializing...
> Checking midnight timestamp alignment ...........OK
> Deploying wish-granting instance ................COMPLETE

⚡️ You have summoned DynamoDB Genie v1.0.0-wish (build 1970-01-01T00:00:00.000Z)

Hey guys, what's happening? What is going on?
You can have three wishes
If you don't take too long...

(type wish -h for a list of options)

genie@dynamodb-realm:~$ ▋

The cursor pulsed with a green glow, each blink seeming to last an eternity, as if time had slowed down. A sudden commotion filled the room as everyone rushed to their keyboards, each desperate to be the first to type their wish command.

What they did not know was that, like all genies through history, this one also had a peculiar way of interpreting wishes - and an unusual fondness for eventually inconsistent results.

Wish #1: I Wish I Could Count Items

The first team wished for a simple counter.

"Just COUNT the items, that's what we need,
But we don't know what it its."

The genie smiled

;) Consider it done!
You have a timeline that you don't want to miss

> Granting... (1/3) OK! 

genie@dynamodb-realm:~$ ▋

And just like that, the wish was granted as the genie flicked its fingers to deploy the whole solution stack to production in an elegant single CLI call.

"It's but a simple counter."
Lead said with a hint in his voice
"Something that should be available
in any database of choice."

Little did they know they were about to learn why implementing a simple counter in DynamoDB could turn into an exercise in advanced distributed systems engineering... and creative AWS budget explanations.

Wish #2: I Wish I Could Search

The second team wished for flexible search.

"Let us comb quickly through all our store,
Filter, then find, and then do some more."

Then genie promised

I will make it clean,
Searching through the items no-one has yet seen

> Granting... (2/3) OK!

genie@dynamodb-realm:~$ ▋

Once again, with a mysterious grin, genie performed an elaborate dance with his fingers and with a barely audible crack, a new stack materialized in their account.

Nobody suspected it to be another trap that the genie had set for them. Why would they? The solution worked perfectly. How could they have known that it would completely fall apart the moment they got featured in TechCrunch, resulting in the company holding the record for the most expensive AWS bill accrued in a single day in history?

Wish #3: I Wish I Knew the Past

The last team wished for analytics.

"Give us the power to slice and to dice,
All of our data, make insights precise!"

The genie's eyes sparkled

Oh, that sounds nice,
But do remember - you cannot wish twice.

Granting... (3/3) OK! 

genie@dynamodb-realm:~$ ▋

For the final time genie performed his deployment magic. His fingers moved with practiced precision, though anyone watching closely might have noticed a slight hint of anticipation in his movements. The CloudFormation stack materialized with a sound like distant thunder.

Process your data, aggregate more,
Calculate trends from your DynamoDB store.
Real-time insights at your command,
What could possibly get out of hand?

genie@dynamodb-realm:~$ ▋

The unexpected genie's words hung in the air like a fog, thick with unspoken warnings. His smile - or perhaps a smirk - held secrets that would only become clear in production.

And then it said:

I'm sorry
That's the way it goes
It's time for me to go
Bye

⚠️ Warning: Wish capacity exceeded 

genie@dynamodb-realm:~$ ▋

Session terminated...

~$ █

The Smirk in Genie's Wish-Granting

Fast forward one month and the startup was like a battlefield. AWS bill had increased tenfold, the pages were taking ages to load as the DynamoDB read capacity was constantly maxed out. Every search operation triggered a full table scan, reading (and paying for) every single item. The elaborate event-driven aggregates were constantly off, making "eventually consistent" look like a precise science in comparison. A new term "eventually inconsistent" was even coined.

It was a total disaster...

They tried summoning the genie again but to no avail. It's as if he was mocking them...

Initializing...
> Checking midnight timestamp alignment ...........OK
> Deploying wish-granting instance ................COMPLETE

⚡️ You have summoned DynamoDB Genie v1.0.0-wish (build 1970-01-01T00:00:00.000Z)

⚠️ Request capacity exceeded
Bye
Session terminated...

~$ █

Last time I checked, they had a dedicated team just for maintaining their DynamoDB access patterns and were spending enough AWS credits to buy a small island.

The Three Lessons of the DynamoDB Genie

Lesson #1: Count Your Wishes Carefully

DynamoDB has gaps, significant limitations that can't be "wished away". We are not talking about small imperfections - it's huge chasms in functionality that can swallow entire development teams.

Working with DynamoDB isn't about personal preference – it's like building a space station. When you first launch it, everything is perfectly engineered for your mission. The systems are optimized, performance is there, and the cost-to-value ratio makes perfect sense.

But missions change. What starts as a simple research station will eventually grow and evolve. Each modification requires a spacewalk 🧑‍🚀 and careful planning - there's no room for error when you're in orbit. Sure, you can add new docking ports (GSIs) to allow different types of ships to connect - these are your backup plans, different ways to access your station when you need to. But they're just entry points to your existing structure; they can't change your core architecture. In the end, major changes mean rebuilding entire modules in space. The design that was perfect for your original mission has become a constraint on your future ones.

Apollo 13 mission was a lesson in resourcefulness and resilience. It taught us that sometimes we need to build a carbon dioxide scrubber from incompatible parts just to stay alive. Just like those astronauts had to craft a literal filter from nothing to prevent suffocation, DynamoDB developers often find themselves engineering elaborate workarounds just to perform basic filtering operations - all while watching their CPU credits running out like oxygen leaking out of a damaged spacecraft. The problem is it's not done out of necessity, after exhausting all other options. On the contrary - workarounds are often wielded like weapons.

This is exactly like DynamoDB – your initial data model might be perfect for your current requirements, but when business needs evolve, even small changes can require complete data restructuring. You can by read about this in more detail here, but in short:

🩸 Everything has to be carefully planned
🩸 There is no ad-hoc queries - everything has to be carefully planned
🩸 Filtering as part of data retrieval has to be carefully planned
🩸 Did I tell you that everything has to be carefully planned?
🩸 SELECT COUNT(*) requires a team of Noble-prize scientists to implement
🩸 You can't join tables for querying related data
🩸 Last but not least, forget about even the simplest reporting or analytics

DynamoDB is THE perfect core datastore. It excels for CRUD, scaling to infinity with unmatched performance while remaining very cost effective.

But it struggles with everything else.

Lesson #2: Be Careful What You Wish For

Know Your Tool

Don't try to force DynamoDB beyond its boundaries or you will find setting yourself up for failure.

The real problem isn't DynamoDB's limitations - it's not considering complementary tools that would make these limitations irrelevant. After all, you wouldn't blame a hammer for being bad at cutting wood; you'd reach for a saw. But what if there were no saws available?

The Tooling Gap

Here's something that continues to blow my mind. 🤯 There's a surprising gap in the tools and services ecosystem, both in and out of AWS. While everyone seems eager to integrate with everything else, DynamoDB often gets treated as a second-class citizen.

The lack of native, seamless DynamoDB integrations is staggering and I just don't get it. Think about it:

📈 AWS has a dominant position on the market - check!
❤️ DynamoDB has been widely adopted - check!
💰 Those who adopted it happen to both have money and no problem spending it

It's a huge piece of the market that has been up for grabs for quite a while now, yet there is still minimal support for native DynamoDB integrations.

This is particularly ironic given that AWS provides its own native solutions that eliminate the need for third-party streaming alternatives (DDB Streams, Kinesis). Yet pick a random third-party product and you will be most likely advised to use Kafka for streaming DynamoDB data in and out of it. For us, serverless AWS software engineers, "industry standard" solutions like Kafka are unnecessary middleware layers that only increases operational burden and cost.

This landscape is slowly changing

The Rockset demise drama caught the market off-guard with countless Rockset refugees running around and screaming like headless chicken. I don't blame them - there was literally no other solution on the market that provided their seamless DynamoDB integration experience while delivering exceptional added value. If I were to bet, I would bet all my money on Rockset being on a fast-track for an eye-watering figure AWS acquisition. Imagine my surprise when OpenAI acqui-hired all of its staff and scraped the service practically overnight.

That event highlighted a critical weakness in the DynamoDB ecosystem - when a single third-party tool disappears, teams can be left with no viable alternatives. While new solutions have emerged to fill the void, the fundamental problem of limited choice persists. The demand from customers willing to redirect their past Rockset budgets has driven some market changes, third-party support remains surprisingly limited. This persistent gap creates unnecessary complexity for teams that have deliberately chosen DynamoDB for its operational simplicity and scalability promises.

Lesson #3: Let Not What's in Front of You Blind You to Alternatives

Look Beyond DynamoDB

Being good at DynamoDB doesn't mean using it for everything. On the contrary, if you're good at it, you'll know exactly when to stop. And when you do, the next step is finding just the right tool that will complement DynamoDB by being its perfect extension, integrating so smoothly that you won't even notice.

What makes a solution truly "fit" with DynamoDB? It comes down to three crucial factors:

🩸 Seamless Integration - it should work with DynamoDB native features (especially DDB Streams) without requiring complex middleware or custom code
🩸 Complementary Strengths - it should fill DynamoDB gaps while letting DynamoDB continue doing what it does best
🩸 Operational Simplicity - it should maintain promise of low operational overhead and should not introduce scaling or reliability challenges

Always be on the lookout for tools that have the potentially of being better than your first choices. Invest in spending time to carefully consider your options so that you don't have to pay premium for making the wrong choices in the future.

I have major grievances and complaints about AWS

My complaint is that AWS makes everyone think that DynamoDB is more than it actually is. Take this blog post for example. Publishing an official guide about counting in DynamoDB could and frankly, does mislead developers into thinking this is a reasonable use case.

Think about it in terms of "productive laziness" vs "harmful laziness" - forcing DynamoDB to do counting is a perfect example of expending enormous effort to make the wrong tool do something it wasn't designed for. And people still fall for it.

By not giving developers access to tools that they need in order to be successful in their work, AWS is forcing them into doing something that should never even be considered. Even the best of us are limited in what they can do with business constraints - whether it's money, pre-approved supplied lists, legal or simple reluctance of your superiors for introducing new tools and different solutions.

At the end of the day, AWS is well aware of the fact that once they pull someone into their ecosystem, many companies won't look outside of it for solution assuming, rightfully so, that all of their use cases will be covered by the tools and services provided by AWS.

What's Next

While I will keep preaching that sometimes the best demonstration of DynamoDB expertise is knowing when not to use it, there's more to the story than just avoiding DynamoDB for unsuitable use cases. The real key is being aware of the areas in which alternative tools can seamlessly complement DynamoDB's strengths while addressing its limitations and being open to utilizing such tools.

That's why in my next articles I'll explore such areas and tools that work alongside DynamoDB to create truly robust, scalable architectures. We'll look at solutions that don't just work around DynamoDB's limitations but turn them into opportunities for building better applications.

Stay tuned to learn about the solutions that every DynamoDB developer should be aware of. After all, even the most powerful database needs a few good friends 🤝 and I am not talking about stabbing-in-the-back genies 🧞‍♂️

Disclaimer

This article is an independent developer guide. All views, wishes, and mystical encounters are entirely my own.

Side effects of reading this article may include: spontaneous JSON formatting, midnight AWS console checking, and an irrational fear of making wishes to database genies.

DynamoDB and the Art of Knowing Your Limits 💥When Database Bites Back 🧛‍♂️

Sebastian Zakłada 🧛 — Fri, 22 Nov 2024 10:45:11 +0000

Sometimes, the best demonstration of DynamoDB expertise is knowing when not to use it

Time and again, developers reach for familiar but ill-suited technologies, driven by everything from career-building tech choices to corporate mandates to simple comfort with the known.

No service exemplifies this better than DynamoDB - while powerful in the right hands, it's no silver bullet, but unlike some 🧛‍♂️s it won't suck the life out of your application... if you use it wisely! Which means you not falling into the traps of:

Ad-Hoc Querying
Filtering
Joining Tables
Counting Items
Analytics
SQL

just to name a few...

⚠️ Spoiler alert ⚠️

Let me set the record straight about what I think about DynamoDB

It is a terrifyingly terrific tool.
You simply cannot go wrong with it as your primary data storage and main CRUD data source.
While DynamoDB is undoubtedly the unspoken 🐐 of the serverless world, it can be pretty terrible at tasks you might consider basic - like counting or filtering, just to name a few.
When someone tells you DynamoDB is good at everything, politely excuse yourself and come talk to me instead. I'll explain why that's far from true.

I love ❤️ DynamoDB

I've been using it for years now and cannot emphasize enough how much it's transformed the way I approach data in my service design. Pair it up with Lambda, wrap nicely with API Gateway and AppSync, and sprinkle some EventBridge magic dust – and you have a powerful, scalable, and cost-effective serverless system that just works.

It's also been a tough love. I've quickly realized that what makes DynamoDB so powerful is also why it can be super frustrating to work with when designing web or mobile applications.

If you've read my previous writeup (thank you!) you may remember how I said that Redshift was a freight train. Following that analogy, DynamoDB is like a Formula 1 🏎️ car - or better yet, a Formula E bolid, those are way WAY cooler! 😎 - it's super fast but also very limited in what it can do.

It moves at an incredible speed on a pre-planned circuit (known access patters)
Focuses on performance with minimal latency (consistently 🏎️💨💨💨)
Built for specific purpose while absolutely nailing it (highly focused)

BUT...

Not unlike F1 car, it's not-so-great at taking off-track excursions (struggles with unplanned query patterns)
Has no passenger seats and zero trunk cargo space (unable to JOIN with other data - no trips to the mall 🛒!)
One-way speed - want to run a simple "count all cars?" query? You have to drive the entire circuit (full table scan), count manually while paying fortune for fuel (read capacity units) while suddenly crawling at truck speeds

Eventually, trying to force DynamoDB beyond its comfort zone feels like pushing that race car through pit lane. No amount of tuning – tires, engines, or aerodynamics – can change its fundamental nature. Beyond basic CRUD operations, you will eventually outgrow DynamoDB intended use cases.

Which is perfectly fine 😄

The real challenges emerge when developers, unaware of system limitations, find themselves fighting against these constraints. This struggle leads to problematic implementations – and from there, little monsters 👾 are born in the code...

The classic case of database-as-swiss-army-knife immediately comes to mind. A developer thinks, "I have my data and SQL, so why not..." and proceeds to write a nightmare of a query. Convoluted and difficult to explain, unoptimized, slow, expensive and draining resources. Even worse, these solutions often pass initial testing (if tested at all... 😶‍🌫️) and only much later when faced with real-world scale, they crumble leading to significant production issues.

Day of the Tentacle, anyone...?

Not thinking ahead about 'boring' but crucial aspects of system design - scaling, high availability, cost efficiency - is a common sin in the developer community. Too many of us put our blinders on and do just what the Jira ticket tells us to do. Out of acceptance criteria, out of mind, let's bring next one in.

One of the reasons why tools like Lambda or DynamoDB are THE perfect building blocks and became so wildly popular is because they take care of many of those 'boring' aspects for you without having to even lift a finger. Both scale pretty much infinitely with your traffic, offer built-in resilience, and remain cost-effective to run.

With tools of the past it was super easy to mess things up - whether by applying the wrong pattern, using something for unintended purposes, or skipping entire architectural layers simply because business requirements didn't explicitly ask for them (yes, this happens more often than I'd like to admit). Can you mess things up with Lambda or DynamoDB? Sure you can! Long-running functions, badly designed keys, larger-than-life lambdaliths - these can still bite 🧛‍♂️ you. But even with these potential pitfalls, there is less room for error, leading to growth-ready apps even if they were created by less experienced teams.

Which I think is great!

DynamoDB takes an interesting approach to database design by intentionally making certain operations costly or cumbersome to steer developers away from anti-patterns. There is method to this apparent madness - by making inefficient patterns painful to implement, DynamoDB nudges teams toward thinking carefully about their access patterns upfront.

It's a clever bit of behavioral design that saves many applications from scalability problems down the road

Rather than giving developers enough rope to hang themselves with complex queries and full table scans, it guides them toward sustainable NoSQL patterns from day one.

The problem with people, and especially software engineers, is that we are a lazy bunch. And don't get me wrong - lazy can be good. It propels progress when approached with the right mindset. But it can also be a huge wall that you build around yourself, preventing from succeeding at anything.

The best in our field of work have mastered the art of productive laziness - it's practically an unofficial part of the job description for senior software engineers. This instinct to find the most efficient solution has given us everything from automations to reusable libraries, from informative CLI prompts to safe data repopulation scripts.

BUT (again)...

There's a crucial difference between the laziness that drives innovation ("I'll spend three hours automating a five-minute task") and the laziness that holds us back ("I'll just copy-paste this solution I don't fully understand"). The former pushes us to create better tools and elegant solutions, while the latter builds technical debt and limits our growth while introducing a real risk of an app imploding in production. Being lazy because you don't feel like learning is a very dangerous state of mind to be in, and with the rise of AI-assisted programming tools, I feel like many of us are slowly taking the "engineer" from the "software" and becoming a mindless copy-paste machine.

The real skill lies in recognizing which type of laziness we're embracing at any given moment

Are we cutting corners to avoid getting out of our comfort zone, or are we streamlining processes to focus on what truly matters?

Choosing the right tool for the job remains one of software engineering's fundamental challenges. Yet time and again, developers reach for familiar but ill-suited technologies, driven by everything from career-building tech choices to corporate mandates to simple comfort with the known. These misaligned choices eventually extract their toll - in maintenance nightmares, performance bottlenecks, inability to handle traffic surges and ultimately frustration stemming from countless hours spent wrestling with a tool that was never right to begin with.

It's the technological equivalent of using a sledgehammer to hang a picture frame - sure, it might work, but at what cost?

Don't be one of these guys...

Enough with the rant... 😄 Let's get back to the star of the show!

DynamoDB and Where it Falls Short

When implementing modern, data-driven applications, certain DynamoDB design choices can and will cause friction. Understanding these limitations is crucial for architects and developers alike making informed implementation decisions.

Problem 1: No Ad-Hoc Queries

DynamoDB is very stubborn and set in its ways 😄 taking a strict, opinionated approach to data access. While it's amazing at delivering blazing-fast queries, it comes with a significant constraint - you need to know exactly how you'll query your data before you even start. As it requires predefined access patterns through its keys and indexes, you typically cannot perform efficient ad-hoc queries outside these established paths.

For example, consider this SQL query (shown for illustration, as DynamoDB doesn't use SQL):

SELECT id, mpn, name FROM Projects WHERE mpn = '00638FOO'

If your table only has an id as the primary key and no secondary indexes on mpn, this query will require a full table scan — an operation that is both costly and inefficient for large datasets. To query efficiently, you would need to either redesign your primary key or create a secondary index or change your access patterns.

That sounds like a lot of work, doesn't it?

Problem 2: No Filtering (almost)

This problem stems from the previous one. While DynamoDB has support for filter parameters, these filters are applied after the data is retrieved from the table, not during the initial query. This means you still pay for the read capacity of all the data fetched before filtering!

For example:

SELECT * 
FROM Products 
WHERE category = 'Electronics' 
  AND price < 500 
  AND rating > 4

Unless you have an index specifically designed to support this access pattern, implementing such ad-hoc request in DynamoDB would result in:

retrieving all items
applying the filters
returning the filtered results

It's effectively the same as performing a full table scan and applying an in-memory filter on the entire result set

const filteredResult = items.filter((i) => 
  i.category === 'Electronics' 
  && i.price < 500 
  && i.rating > 4
)

This approach can become inefficient especially with large datasets or complex filtering requirements.

Problem 3: No Table Joins

DynamoDB doesn't support table joins like traditional relational databases. Each query can only access a single table at a time. For example, this type of SQL join operation is not supported.

SELECT P.name, O.quantity 
FROM Products P
INNER JOIN Orders O ON O.productId = P.id

Instead, you would need to either denormalize your data by embedding related information in a single JSON document or perform multiple separate queries in your application code to fetch and combine the data.

Again, this was a conscious design choice that AWS made to maintain consistent performance at scale, but it also means you'll need to carefully plan your data modeling to avoid the need for joins, which can significantly limit your flexibility, productivity and ultimately - time to market.

Problem 4: No analytics

Like nothing. Nada. Zero. None.

DynamoDB's analytical capabilities are severely restricted compared to traditional relational databases. It lacks fundamental analytical functions that are standard in SQL databases:

No COUNT
No aggregations or window functions
No GROUP BY
No subqueries, CTEs
No math functions
Frankly not much beyond simple CRUD operations is available

Even the simplest of use cases such as SELECT COUNT(*) becomes super complex and expensive in DynamoDB

Consider this basic SQL example

SELECT COUNT(*) 
FROM Products

While it's trivial in relational databases, it becomes an utter nightmare to develop and maintain if you want to support such scenario in DynamoDB. Take my word for it, there is countless of different examples available online discussing different ways to approach this requirement which in the DynamoDB world becomes surprisingly complex and resource intensive.

Problem 5: No SQL

There's something special about SQL that we've all grown to love over decades. It's not just about familiarity - it's about expressiveness, power, and simplicity that comes with it.

With DynamoDB, you're forced to learn and use a completely different query language and SDKs. Instead of a simple, readable SQL statement, you're now deep in the world of expressions, attribute values, and other verbose SDK constructs. Even with PartiQL (AWS's attempt at SQL "compatibility"), you're still heavily restricted - no JOINs, no COUNT, nothing that makes SQL... well, SQL.

The lack of SQL support isn't just about syntax - it's about losing decades of tooling, knowledge, and ecosystem that make data work easier and more productive.

Compare this simple query:

SELECT * 
FROM Users 
WHERE status = 'active' 
  AND lastLogin > '2024-01-01'

with its DynamoDB potential equivalent:

const params = {
  TableName: 'Users',
  KeyConditionExpression: '#status = :status',
  FilterExpression: '#lastLogin > :date',
  ExpressionAttributeNames: {
    '#status': 'status',
    '#lastLogin': 'lastLogin'
  },
  ExpressionAttributeValues: {
    ':status': 'active',
    ':date': '2024-01-01'
  }
};

Not exactly what you'd call developer-friendly, is it?

And this is just a simple query - imagine dealing with more complex needs. Sure, there are tools trying to make this better like dynamodb-toolbox by providing a more friendly API. While they are great at what they do - they make code look better and easier to work with - but at the same time they can't change DynamoDB fundamental capabilities. They're band-aids that make development less painful, but don't address the fundamental limitations.

Combined: Even more problems

Add all of the above together

const moreProblems = simpleQuery
  .createGlobalSecondaryIndex()
  .addCompositeKey()
  .denormalizeEverything()
  .sacrificeGoatToNoSQLGods()
  .pleaseJustWorkThisTime();

and you will quickly realize that any moderately complex data access needs turn into a massive engineering effort. Each of these limitations alone is manageable, but combined they create a perfect storm of complexity that can seriously slow down development. You'll find yourself writing tons of application code, managing complex data access patterns, and building intricate workarounds just to get basic functionality that comes out of the box with SQL databases.

Working with DynamoDB can be like planning a train network for a city. Sure, it's great for the routes you planned, but what happens when your needs evolve? Rigid foundation doesn't play well with the needs of modern software development, which prioritizes agility and rapid iteration cycles.

Mini Metro is such a terrific and cleverly designed game!

Any change may turn into major engineering effort, like trying to retrofit a new line through downtown just because you didn't predict today's traffic patterns years ago.

Good luck explaining that to the affected homeowners... 😄

Why Not Just Use a SQL Database Then?

Fair question! Thanks for asking!

For many use cases, you absolutely should. But as with many things in this world - you gain something and lose something with each choice you make. While traditional relational databases such as PostgreSQL give you the power of querying flexibility, they also have their own limitations and best use-case applications.

DynamoDB shines in specific scenarios - when you need consistent single-digit millisecond performance at any scale, with minimal operational overhead. It's particularly great for known access patterns like user profiles, session management, or real-time game states.

The issue arises when your application grows and you need both: the rock-solid performance of DynamoDB for your core operations AND the flexibility of SQL for analytics, complex queries and rapid feature development.

The DynamoDB Tunnel Vision

Consider yourself spending months or years working with DynamoDB and suddenly seeing everything through NoSQL-tinted glasses. Every new project becomes a nail for your perfectly-tuned DynamoDB hammer. "Of course we'll use DynamoDB - we always use DynamoDB!" becomes your unofficial motto.

This tunnel vision is particularly dangerous because it often comes from a place of genuine expertise and success. You've mastered the intricacies of partition keys, you can design access patterns in your sleep, and you've successfully launched multiple DynamoDB-powered applications. Your comfort zone has become a fortress, and stepping outside feels unnecessary, even risky.

The symptoms are subtle but telling

You start force-fitting complex analytical queries into DynamoDB, creating elaborate workarounds that would be otherwise trivial
Your application code grows increasingly complex as you build client-side solutions for basic database operations
You find yourself defending DynamoDB's limitations as "features" that force better design (while spending hours implementing basic counting functionality)
The phrase "we'll just denormalize the data" becomes your default solution to every data modeling challenge

The dangerous part is that this kind of thinking can feel like you're just being really good at your job. "I know DynamoDB's limitations!" you might say, "and I can work around them!"

But there's a fine line between working around limitations and fighting against the fundamental nature of your tools

Remember our Formula 1 car analogy? This is exactly like insisting on using your racing car for grocery shopping, daily commutes, and moving furniture, then spending countless hours engineering elaborate solutions to make it work. Do you really think strapping a cargo container to your F1 car is a good idea? 😅

The real danger isn't just the technical debt you're accumulating - it's the cost of not exploring better solutions. While you're busy implementing complex DynamoDB workarounds, you could be leveraging purpose-built tools that solve your problems out of the box. That analytics query that took you three days to optimize in DynamoDB? It could have been a simple SQL statement in Tinybird.

Breaking free from this mindset requires humility and a willingness to admit that sometimes, just sometimes, your favorite tool isn't the right one for the job. It means acknowledging that expertise in one technology shouldn't limit your architectural choices. Most importantly, it means remembering that our job isn't to use DynamoDB

Our job is to solve problems effectively

The next time you find yourself reflexively reaching for DynamoDB, pause and ask:

Am I choosing this because it's the best tool for the job, or because it's the tool I know best?
What would this solution look like with a different database?
Does this solution need a database at all (S3, anyone?)
Am I building workarounds for functionality that comes standard in other databases?
Is my comfort with DynamoDB blinding me to better alternatives?

Sometimes, the best demonstration of DynamoDB expertise is knowing when not to use it.

(to be continued...)

Have you faced similar challenges with DynamoDB? How did you solve them? Share your experiences in the comments!

Next Up

The landscape of data engineering has evolved. Tools like OpenSearch (or Algolia), Redshift (or Databricks), Tinybird (or Rockset... oh, wait) have emerged to bridge the gap between purpose-built databases.

In the upcoming article I will explore how we can leverage these specialized tools alongside DynamoDB to build robust, scalable systems without breaking the bank and spending months implementing features.

Time-to-Market is king! 👑

Disclaimer

This article is an independent developer guide. All views, rants and recommendations expressed are my own.

No databases were harmed in the making of this article. Just some egos and a few AWS accounts over the years of me using DynamoDB.

I took this article as an excuse to dive into my beloved World of Words 😄

Gosh, I never realized how much I missed writing...

Improving Tinybird DevEx 🚀 Data Re-Population CLI Tool

Sebastian Zakłada 🧛 — Wed, 20 Nov 2024 10:49:35 +0000

Quote of the day

The key isn't blindly following "best practices" - it's about understanding your tools well enough to know exactly when to use them.

We continue exploring ways to improve Tinybird developer experience in the command line. In my last article, we explored enhancing the Tinybird CLI with Oh My Posh, creating a custom prompt that displays critical workspace and environment information. That improvement was aimed at preventing accidental deployments and keeping aware of the current terminal session context.

Today, we're diving deeper into another crucial aspect of Tinybird development - managing data in your downstream data sources efficiently.

The Data Management Dilemma

Ever found yourself neck-deep in Tinybird (or any other data-centric) development, making sweeping changes to your processing pipelines and materialization logic, only to realize you need a fresh start? Any early stage feature development is prone to changing requirements and that makes data structures a moving target.

Or perhaps you've encountered that heart-stopping moment when production data becomes corrupted, and you need a reliable and fast way to recover?

Can you guess my age from this image...?

I've been there, and today I'll share how we've tackled these challenges at my eSUB mothership.

What Can Trigger a Full Refresh

Analytics platforms such as Tinybird, where you ingest and transform data to produce materialized views tuned for best performance and cost efficiency, they all come with their own sets of challenges. During rapid development cycles, you might find yourself:

Making changes to pipelines
Implementing major shifts in business logic
Recovering from unexpected data corruption in an environment
Periodically refreshing development environments

All of that may demand a clean slate as it's no longer possible to support new data models or changed requirements when there is breaking changes involved.

The Pitfalls of Destructive Full Refreshes

Before we get into the weeds and I can explain the tool that I've built, let me get one thing straight - full data re-builds are super bad, and YOU SHOULD AVOID DOING THAT AT ALL COST.

I just threw a wrench into my whole "the-best-ever-o-matic new CLI tool coming up!" sales pitch, didn't I? 😄

Jokes aside, and I am a jolly person, it has to be said out loud and clear - when faced with the need to refresh data, performing a destructive full refresh in production environments carries significant risks that should make you pause and re-consider your approach.

Service Interruption - during a full refresh, your analytical queries and dashboards may become unavailable or return incomplete results, potentially impacting business operations and decision-making.
Resource Intensive - full refreshes typically consume substantial computational resources, which lead to cost increase, performance hit of other running processes and even potentially exceeding service quotas or limits
Data Consistency Challenges - full refresh throws your whole system into partially inconsistent state, if it receives new real-time events during that time, that data may be lost or you risk having inconsistent states between different parts of your system as a result of downstream pipes and even other systems receiving incomplete or incorrect data
Recovery Complexity - if something goes wrong during the refresh you may not have an easy way to roll back and the original state could be permanently lost

That's just to name a few.

Instead of destructive full refreshes, consider these alternatives:

Implementing incremental updates where possible. Sometimes all it takes to avoid a breaking change from happening is a little bit of thinking ahead and preparation. It's possible more often than you think.
Versioning your pipes and datasets. Can't emphasize this enough. If it hits production, then it's your contract and unless you really know what you are doing (like, really REALLY) making any breaking changes to an already published contract is a big no-no. Think of your Tinybird data carrying structures (pipes, data sources) as APIs. In the end, most often than not you are going to publish results of your user-facing pipes as APIs. And what do you do when you have to change a contract in the existing API? You version it. The same principle applies in Tinybird or really to any part of the system you are building.

With proper versioning you can maintain parallel data structures during transitions, empowering you with the capability to do many cool things such as

A/B testing and canaries (NOT the birds!)
super easy rollback to proven-and-tested previous versions
making sure that your new contracts are stable before fully committing

Going Against the Flow

You must be pretty confused at this point, asking yourself the question... why building the tool?! After all I just said that triggering full data rebuild is the last thing you should do!

Look... while I just spent a good chunk of this post telling you why full rebuilds are generally a terrible idea (and they really are!), sometimes you just need the right tool for the job. You know what they say about rules being made to be broken? Well, not exactly broken, but more like... carefully bent when you really know what you're doing! 😉

Sometimes you DO need that nuclear option - maybe you're in development, testing some wild new ideas, or dealing with a situation where a full rebuild is actually the safest bet. The key isn't blindly following "best practices" - it's about understanding your tools well enough to know exactly when to use them.

And hey, that's exactly why I built this tool - for those specific situations where a full rebuild is exactly what the doctor has prescribed!

The Manual Console-driven Method

For simple use-case scenarios where all you need is trashing data in a single data source it's perfectly fine to trigger rebuilds from the Tinybird Console as all it takes is just a few clicks to get data flowing again.

But let's be real - when you're dealing with anything but a sample app, you will be dealing with dozens if not hundreds of data sources and a complex web of dependencies. Clicking through the UI becomes a pain real quick. You also need to consider orchestrating the order of operations as you definitely don't want to refresh that downstream pipe before its dependencies are ready, right?

With more than just a handful of pipes and data sources present in a project these scenarios can quickly turn into time-consuming ordeals. Lack of proper tooling can turn into a significant operational burden and before you know it, you're spending half your day just clicking through the UI or trying to remember which pipe needs to be refreshed first. Been there, done that, got the t-shirt!

There's an old engineering wisdom that states: if you find yourself doing something manually more than twice, it's time to automate it. This principle became crystal clear when I was recently working on Tinybird implementation, where I found myself repeatedly resetting data sources during rapid prototyping stages. The first time, it was a novelty. The second time, it felt like déjà vu. By the third time, it was clear

I Needed Automation

That's when the data re-population script was born - forged from the necessity of automation, refined and polished through repeated use.

One tool to bring them all and in the darkness bind them 😉😅

Utilizing the native Tinybird CLI capability to manage pipes and data sources from the command line, I crafted a specialized script that handled truncating and repopulating Tinybird data sources in a carefully orchestrated pattern.

My preciousss... 💍

Let's Dive In

After acknowledging all the risks and considerations we discussed earlier, I am giving you a tool that transforms what could be a hazardous operation into a controlled, deliberate process. Think of it as your "break glass in case of emergency" tool - but one that actually knows what it's doing!

The tool provides two core capabilities, each wrapped in multiple layers of safety controls. First, it manages your data sources through selective truncation based on configurable rules, with built-in protection for critical sources through prefix-based exclusions. Second, it handles orchestrated repopulation using a three-phase strategy that respects data dependencies: starting with core reference data, moving through standard operational pipes, and finishing with dependent calculations (that part you should adjust to match your use-case).

Getting started is straightforward:

# Clone the repository
git clone https://github.com/sebekz/tinybird-devex-plus.git

# Navigate to script location
cd tinybird-devex-plus/tinybird/scripts

# Make it executable (Unix/Linux/macOS)
chmod +x repopulateAllDataSources.sh

You can also pull only the script from the GitHub repo both approaches are fine.

One must not forget about all the required dependencies. You'll need the following tools installed:

Tinybird CLI - obviously...
jq - for all the JSON parsing that the script is doing

Once it's done you are ready to run it 💥!

# Usage:       . scripts/repopulateAllDataSources.sh [COMMAND]
#
# Commands:
#              dryrun     - Performs a dry run without mutating any data sources
#              repopulate - DESTRUCTIVE OPERATION! Truncates and populates 
#                           matching data sources
#
# Example:     . scripts/repopulateAllDataSources.sh dryrun
#              . scripts/repopulateAllDataSources.sh repopulate
#
# Note:        You must be in /tinybird folder for this script to be able to 
#              use tb authentication details

But wait... didn't we forget about something?

Safety First: The Art of Not Breaking Things

Remember how we talked about the risks of full rebuilds? Remember that scene in Jurassic Park where Samuel L. Jackson's character says "Hold onto your butts" before rebooting the park's systems? Well, data repopulation isn't quite as dramatic, but it deserves the same level of respect.

I added a bunch of extra safety measures ensuring, that anytime you run the script, you must turn all the required safety keys first.

By default, this script protects your critical data sources - landing sources (with source_ prefix), operational data (ops_ prefix), and snapshots (snapshot_ prefix) remain untouched. These particular prefixes resulted from the use-case scenarios that I was working on, yours may and most likely will be different so feel free to adjust that part to your liking!

Protection goes beyond just prefixes. The tool implements what I like to call "progressive do-not-break-everything safety" - you need to explicitly confirm your intentions at several checkpoints before any action is taken.

The script enforces several safety measures:

🔑 Mandatory workspace verification before execution
🔑 Configuration review prompts that demand explicit confirmation
🔑 Dry run capability for risk-free testing
🔑 Final Warning and confirmation
🔑 Built-in exclusions for critical data sources

Each step requires explicit "y" or "Y" confirmation. Any other response (including empty) aborts the operation.

You can see that it's not just a simple reset button - it provides a safe and configurable way for dropping nukes on your data. Think of it as a "reset button with guardrails". Safety is enforced through:

Usage

Run script without any parameters to see all the available commands.

YES. I have OCD when it comes to nice looking tools. Can't help myself with all the colors, formatting, etc. Some call it an utter waste of time, but I find that good looking tool or good-looking code is just better - as you put more thought to how it looks you end up putting more thought into how it WORKS.

I guess that's a good topic for a separate article 😊 and I digressed again. That's also something I am good at 😊

When you run the tool, it first walks you through a configuration review. You'll see exactly what data sources would be affected and what the execution plan looks like. It verifies your current workspace context and makes sure you're operating in the environment you intend to.

As mentioned earlier - do not rush to press Y, take your time and triple check the settings.

Always start with dryrun. It's not just a suggestion - it's best practice. This mode lets you validate every step of the process without touching your data. Only when you're absolutely certain about the changes should you proceed with the actual repopulation.

During execution, the tool orchestrates all the required steps. It first truncates the selected data sources, then executes priority pipes to establish your core data foundation. From there, it processes standard operational pipes, and finally handles any dependent calculations. This ordered approach ensures data consistency throughout the rebuild process.

Here is how it looks like in action, starting with the truncation

after which data sources are re-populated from their respective pipes

The result? A tool that respects the destructive nature of data rebuilds while providing the automation we need for those rare but necessary occasions. It's not about making rebuilds easy - it's about making them safe when they're unavoidable.

Room for improvement

Like any tool born from practical needs, this one has its own wishlist for the future. Two main items are keeping me up at night (along with my usual vampire schedule 🧛):

Moving configuration to script parameters - because hardcoding configuration is so last century
Migrating the code to Python - as much as I enjoy the old-school Bash routine, Python would make this tool much more maintainable and extensible
Removing repetitions, shortcuts and unnecessary code - a result of iterative approach and YAGNI as soon as I got what I needed back when I needed it

I'll probably not make the time to tackle any of these improvements, so if you are feeling adventurous - go for it and share! I would love to see what you were able to come up with! But for now, the current version gets the job done while keeping your data safe from accidental nuclear launches.

Where to Next?

In the upcoming articles, I will explore:

Tinybird Console Pro Tips - unlocking the full-width view and most importantly, the dark mode - vampires like me 🧛 fear the light!
IaC Blueprint - structuring your Tinybird projects like a pro
Multi-tenancy Guide - implementing secure and scalable multi-tenant analytics with Tinybird and AWS
Production Checklist - everything you need for truly production-ready analytics
DynamoDB with Tinybird - tips and tricks learned when working on a real-world implementation
Node.js SDK - Tinybird SDK for Node.js that you never knew you needed!
and many more!

Stay tuned!

Enjoying this content? 🧛

If you found this article helpful, consider following me for more similarly engaging posts!

Note

I'd love to hear your thoughts, suggestions, or even gentle 😄 corrections - don't hesitate to drop a comment.

Your feedback, whether it's a nod of approval or pointing out areas for improvement, will help me craft better content in the future!

Disclaimer

This article is an independent developer guide. I am not affiliated with, sponsored by, or officially connected to Tinybird in any way. All views and recommendations expressed are my own.

While these customizations are generally safe, if your coding adventures somehow result in your workstation spontaneously combusting, I must respectfully decline any responsibility for damage to you or your property. However, as a courtesy to your friendly Engineering Vampire 🧛 please do give advance notice of any potential... mishaps. It would be terribly wasteful to let good blood 🩸 go to waste. Just saying!

Improving Tinybird DevEx 🚀 Creating an Informative CLI Prompt with Oh My Posh

Sebastian Zakłada 🧛 — Tue, 19 Nov 2024 13:34:28 +0000

If you are working with Tinybird regularly, you know how important it is to stay aware of your current workspace and environment. Instead of running multiple commands to check these, wouldn't it be great to have this information right in your prompt?

In this guide, I'll help you to enhance your CLI experience by creating a custom Oh My Posh prompt that displays Tinybird-specific information.

Here is how your customized prompt will look once configured:

Shortcut: If Tinybird and Oh My Posh are not news to you, feel free to skip directly to here for information on how to add Tinybird to your existing prompt.

Tinybird? Never heard of it before!

Let me fix that for you right away! 😄

Tinybird is a real-time data platform that lets developers build SQL-based APIs for working with massive datasets. It builds on ClickHouse, which is widely recognized as one of the fastest analytical databases for real-time data processing workloads.

While traditional data warehouses like Redshift excel at batch analytics and BI workloads - think of them as freight trains hauling massive amounts of data - Tinybird is more like a high-speed maglev train, optimized for real-time, user-facing analytics that require instant responses.

When building data products with Tinybird, you'll typically work across multiple contexts:

Different workspaces for various projects
Multiple environments (development, staging, production)
Distinct configurations and data pipelines per context

Managing these different contexts efficiently is crucial for a smooth development workflow. An informative CLI prompt becomes invaluable here - it helps you maintain awareness of your current workspace and environment without repeatedly running status-checking commands.

Why Customize Your Prompt?

As developers, we spend significant time in the terminal. When working with Tinybird, you need to:

Check which workspace you're in
Verify your current environment
Ensure you're using the right AWS account

How often have you run a tb command only to realize you were in the wrong workspace? Or deployed to production (I know, I know... it was "someone else" on the team... right? 😉) while your AWS credentials were pointing to a different account?

Having this information directly in your prompt eliminates these "oops" moments and helps prevent mistakes like accidentally pushing to the wrong workspace.

We'll set up Oh My Posh to display Tinybird information in your prompt
You'll see your current workspace, environment, and other helpful information at a glance
This works with bash, zsh, and PowerShell
The configuration is also fully customizable to your needs

Prerequisites

You'll need the following tools installed:

Tinybird CLI - configured and ready to use
Oh My Posh - for prompt customization
A terminal that supports Unicode characters like the one bundled with Visual Studio Code
Nerd Font - required for icons

Setting Up Oh My Posh

Before diving into Tinybird-specific customizations, let's set up Oh My Posh correctly. The installation process varies depending on your operating system.

Windows Installation

If you're using Windows, you have several installation options:

# Option 1: Using winget (Recommended)
winget install JanDeDobbeleer.OhMyPosh

# Option 2: Using Scoop
scoop install https://github.com/JanDeDobbeleer/oh-my-posh/releases/latest/download/oh-my-posh.json

# Option 3: Using Chocolatey
choco install oh-my-posh

macOS Installation

For macOS users:

# Option 1: Using Homebrew (Recommended)
brew install jandedobbeleer/oh-my-posh/oh-my-posh

# Option 2: Using MacPorts
sudo port selfupdate
sudo port install oh-my-posh

Linux Installation

For Linux systems:

# Option 1: Using curl (Universal)
curl -s https://ohmyposh.dev/install.sh | bash -s

# Option 2: Using homebrew
brew install jandedobbeleer/oh-my-posh/oh-my-posh

For detailed installation instructions tailored to your specific setup and additional configuration options, consult the official documentation. The docs cover all platforms and special cases like WSL environments.

Understanding Nerd Fonts

Oh My Posh needs a Nerd Font to display icons correctly.

Nerd Fonts are modified versions of popular programming fonts that add a vast collection of icons (glyphs) to the original font. Think of them as regular programming fonts supercharged with extra symbols - like the Git branch icon, folder symbols 📂, or platform-specific icons.

Installation

Oh My Posh has a CLI to help you select and install a Nerd Font:

# To install the recommended font
oh-my-posh font install meslo

# For manual font selection
oh-my-posh font install

After installing a Nerd Font, you'll need to configure your terminal to use it. The configuration process varies depending on your terminal emulator.

Remember that each terminal maintains its own font settings. Configuring VS Code integrated terminal will not affect your standalone terminal emulator, and vice versa. For detailed configuration steps for each terminal type, refer to the official Oh My Posh documentation.

Here's how to set it up in VS Code:

// settings.json
{
  "terminal.integrated.fontFamily": "MesloLGM Nerd Font",
  // Optional: enable font ligatures
  "terminal.integrated.fontLigatures": true
}

Creating the Tinybird Prompt Segment

To display Tinybird workspace information in your prompt, we'll create a custom segment that integrates with Oh My Posh. This involves two steps:

Creating a script to fetch Tinybird information
Adding the segment to your Oh My Posh configuration

Step 1: Create the Tinybird Status Script

Create a new file called tb-prompt.sh in your .aws directory. On different systems, this will be:

Windows: C:\Users\YOUR_USERNAME\.aws\tb-prompt.sh
Mac/Linux: ~/.aws/tb-prompt.sh

Add the following content to the script:

if [ -e ".tinyb" ]; then        
    branch_name=`grep '"name":' .tinyb | cut -d : -f 2 | cut -d '"' -f 2`
    region=`grep '"host":' .tinyb | cut -d / -f 3 | cut -d . -f 2 | cut -d : -f 1`
    if [ "$region" = "tinybird" ]; then
        region=`grep '"host":' .tinyb | cut -d / -f 3 | cut -d . -f 1`
    fi    

    echo "tb:${branch_name}"
fi

Make the script executable (Mac/Linux only):

chmod +x ~/.aws/tb-prompt.sh

This script:

Checks for a .tinyb configuration file in the current directory
Extracts the branch name and region information
Outputs the branch name in the format tb:branch_name

Step 2: Configure Oh My Posh

Add the Tinybird segment to your Oh My Posh configuration by including this segment configuration:

{
  "type": "command",
  "style": "powerline",
  "powerline_symbol": "\ue0b0",
  "foreground": "#00b7ff",
  "background": "#193549",
  "properties": {
    "shell": "bash",
    "script": "~/.aws/tb-prompt.sh"
  },
  "template": " \uf1c0 {{ .Output }} "
}

This configuration:

Runs the tb-prompt.sh script to get Tinybird information
Uses a database icon (\uf1c0) from Nerd Fonts
Displays the output in a clean format with distinctive colors
Integrates with the powerline style of the prompt

Note: The segment will only show Tinybird information when you're in a directory containing a .tinyb configuration file.

Let's look at a complete Oh My Posh configuration that combines some of the tools a developer may need:

AWS Profile & Region - shows your current AWS credentials context
Environment Stage - displays the current stage (like dev/prod)
Node.js Version - shows Node version and package manager
Current Path - displays your working directory
Git Status - shows branch, changes, stash count, and more
Tinybird Workspace - displays current Tinybird workspace (from our script)
NET Version - shows the .NET SDK version if applicable
Root Indicator - shows when you're running as administrator/root
Execution Time - displays how long the last command took
Exit Code - shows success/failure of the last command

Save this as ~/.ohmyposh.json (or your preferred config location) and customize the colors, symbols, and segments to match your preferences:

{
  "$schema": "https://raw.githubusercontent.com/JanDeDobbeleer/oh-my-posh/main/themes/schema.json",
  "blocks": [
    {
      "alignment": "left",
      "segments": [
        {
          "background": "#FFA400",
          "foreground": "#193549",
          "leading_diamond": "\ue0b6",
          "powerline_symbol": "\ue0b0",
          "style": "diamond",
          "template": " {{ .Profile }}{{ if .Region }}@{{ .Region }}{{ end }} ",
          "trailing_diamond": "\ue0b0",
          "type": "aws"
        },
        {
          "type": "text",
          "style": "powerline",
          "powerline_symbol": "\uE0B0",
          "foreground": "#ffffff",
          "background": "#f792c1",
          "template": "{{ if eq \"devtest\" .Env.STAGE }} {{ else }} \uf635 {{ .Env.STAGE }} {{ end }}"
        },
        {
          "type": "node",
          "style": "powerline",
          "powerline_symbol": "\uE0B0",
          "foreground": "#ffffff",
          "background": "#6CA35E",
          "template": "{{ if .PackageManagerIcon }}{{ .PackageManagerIcon }} {{ end }}{{ .Full }} ",
          "properties": { "fetch_package_manager": true }
        },
        {
          "background": "#ff479c",
          "foreground": "#ffffff",
          "powerline_symbol": "\ue0b0",
          "properties": {
            "style": "folder"
          },
          "style": "powerline",
          "template": " \ue5ff {{ .Path }} ",
          "type": "path"
        },
        {
          "background": "#fffb38",
          "foreground": "#193549",
          "powerline_symbol": "\ue0b0",
          "properties": {
            "fetch_stash_count": true,
            "fetch_status": true,
            "fetch_upstream_icon": true
          },
          "style": "powerline",
          "template": " {{ .HEAD }}{{if .BranchStatus }} {{ .BranchStatus }}{{ end }}{{ if .Working.Changed }} \uf044 {{ .Working.String }}{{ end }}{{ if and (.Staging.Changed) (.Working.Changed) }} |{{ end }}{{ if .Staging.Changed }} \uf046 {{ .Staging.String }}{{ end }}{{ if gt .StashCount 0}} \uf692 {{ .StashCount }}{{ end }}{{ if gt .WorktreeCount 0}} \uf1bb {{ .WorktreeCount }}{{ end }} ",
          "type": "git"
        },
        {
          "type": "command",
          "style": "powerline",
          "powerline_symbol": "\ue0b0",
          "foreground": "#00b7ff",
          "background": "#193549",
          "properties": {
            "shell": "bash",
            "script": "~/.aws/tb-prompt.sh"
          },
          "template": " \uf1c0 {{ .Output }} "
        },
        {
          "background": "#6CA35E",
          "foreground": "#ffffff",
          "powerline_symbol": "\ue0b0",
          "properties": {
            "fetch_version": true
          },
          "style": "powerline",
          "template": " \ue70c {{ if .Unsupported }}\uf071{{ else }}{{ .Full }}{{ end }} ",
          "type": "dotnet"
        },
        {
          "background": "#ffff66",
          "foreground": "#ffffff",
          "powerline_symbol": "\ue0b0",
          "style": "powerline",
          "template": " \uf0e7 ",
          "type": "root"
        },
        {
          "background": "#8800dd",
          "foreground": "#ffffff",
          "powerline_symbol": "\ue0b0",
          "properties": {
            "style": "austin",
            "threshold": 500
          },
          "style": "powerline",
          "template": " <#fefefe>\ufbab</> {{ .FormattedMs }} ",
          "type": "executiontime"
        },
        {
          "background": "#2e9599",
          "background_templates": ["{{ if gt .Code 0 }}#f1184c{{ end }}"],
          "foreground": "#ffffff",
          "powerline_symbol": "\ue0b0",
          "properties": {
            "always_enabled": true
          },
          "style": "powerline",
          "template": " {{ if gt .Code 0 }}\uf00d{{ else }}\uf42e{{ end }} ",
          "trailing_diamond": "\ue0b4",
          "type": "exit"
        }
      ],
      "type": "prompt"
    }
  ],
  "final_space": true,
  "version": 2
}

The Impact of a Better Prompt

Configuring your command prompt goes beyond aesthetics - it's about developer productivity and safety. Having instant visibility into your workspace and environment status helps prevent common mistakes and speeds up your workflow.

It's about making your daily work smoother and safer.

Think of it as your development cockpit - just as pilots rely on their (super cool!) instruments, developers need clear visibility of their working context. This becomes especially crucial when working with multiple Tinybird workspaces and AWS environments.

With instant visibility into your workspace and environment status, you can confidently avoid those costly mistakes that happen when context isn't clear.

Where to Next?

This is just the beginning of improving your Tinybird development experience. In the upcoming articles, I will explore:

Tinybird Console Pro Tips - unlocking the full-width view and most importantly, the dark mode - vampires like me 🧛 fear the light!
IaC Blueprint - structuring your Tinybird projects like a pro
Multi-tenancy Guide - implementing secure and scalable multi-tenant analytics with Tinybird and AWS
Production Checklist - everything you need for truly production-ready analytics
Node.js SDK - Tinybird SDK for Node.js that you never knew you needed!
Data Re-population CLI Tool - for efficiently managing and reloading your datasets
and many more!

Stay tuned!

Note

This is my first technical writeup in quite a while (I usually write about craft beer, and in my native language at that!). I'd love to hear your thoughts, suggestions, or even gentle 😄 corrections - don't hesitate to drop a comment.

Your feedback, whether it's a nod of approval or pointing out areas for improvement, will help me craft better content in the future!

Disclaimer

This article is an independent developer guide. I am not affiliated with, sponsored by, or officially connected to Tinybird in any way. All views and recommendations expressed are my own.