DEV Community

loading...
Cover image for DynamoDB Streams

DynamoDB Streams

Royal Bhati
{. . . ❤ } !hate
Updated on ・2 min read

While working on a backend service, I came across this problem where I had to retrieve a set of data of a specific type, but that type wasn’t having any index and the data was also updating frequently.

Since the data was constantly changing (add/update), first thing that came to my mind was to have a trigger but then as I looked at the documentation of how to add triggers to Dynamo in AWS, I pursued a different solution :p

So, there were three straightforward solutions that were present before me without thinking much -

Complete scan of the table and filter the results
This solution was screaming “slow” right from the moment it entered my mind.

Add an Index for that type

There were a couple of reasons I didn’t go along with this idea

  • That wasn't a very frequently queried data
  • Already had so many indexes so didn't want to add one more just yet before trying other solutions

Run a cron-job
A cronjob that will scan the table and store the result in a cache or in another table.

A Great man once said, If you can't think of anything else, run a cron-job.


Back in my mind, I wasn’t quite convinced with any of the above solutions and so I gave Dynamo db triggers try. I was aware of Dynamo db streams but never tried them.

DynamoDB Streams are basically triggers like that we have in Relational DBs but the only difference is that it generates a stream of events when there is a data change and you can easily pipe that stream into Kinesis or a Lambda (In my case it was lambda).

Architecture

AWS made the things look difficult but the process was not complex at all.

All I had to do was the following -

  • Setup a stream on the dynamodb table which was just a one click action.
  • Create a new lambda function and attach it to the dynamodb stream which was also fairly easy
  • Rest was basic stuff of listening the data then processing it and finally saving it to the new table

DynamoDB streams are much more than what I have covered in my use case. You can read more about it here : -https://aws.amazon.com/blogs/database/dynamodb-streams-use-cases-and-design-patterns/

Discussion (0)