Table Of Contents
Introduction
DynamoDB provides an excellent way to store data in a flexible and scalable way. Add it to Lambda and API Gateway, and you have a powerful group of tools, which I have written about. It was for these reasons that I chose to use it for a side project I am building to familiarize myself with React. It was in these features that I struggled with an issue in how boto3, the SDK for talking with AWS in Python, implements their libraries.
The Problem
I have a personal rule; if it takes a certain amount of time to debug an issue, then there is something about the underlying technology that I do not understand. As it happens, I spent some time trying to get the APIs to work correctly in my side app recently. The problem boiled down to how my return data from the API was displayed in the application. It confounded me for some time. So I stepped back and challenged my assumptions about the tech. I followed the rabbit hole back to the API I wrote, where I found the problem.
It turned out the problem was in using the Client class in boto3. From their docs, Amazon calls the Client class a “low-level client representing Amazon DynamoDB.” So this made sense when I started seeing how the API would interact with my app.
When making a call to the API, the JSON returned would have all these extra tags on them. Below is an example.
It seems that DynamoDB was tagging the data types, as string types would have the type “S” for string values and “N” for number values. It made little sense to me. There are possible solutions available in the library that will help unmarshal the data and clean it up a little, but these tools are buried in the source code for boto3 and don’t seem to have official documentation. There’s a yet unresolved GitHub issue about it. So I had to look deeper for a solution.
The Answer
After a while of searching through the AWS docs and trying various solutions, I was able to come up with a fix. I rewrote my API to grab all the records using the Table class rather than the Client class.
It turns out that the Client class works precisely like a low-level abstraction should, returning all the stuff that’s stashed into DynamoDB data tags and all. Not very reader-friendly.
Here you can see what the code looks like using the Client class.
And here is the updated code for the Table resource.
Very little in the logic changed. We are still creating a connection to AWS using a Lambda function. We are still returning the response to that function.
There are, however, a couple of differences. We call the Table resource a little differently from the client, and I added some pagination to the table code. As boto3 will only return everything up to 1 MB, you need to put in pagination to get all your results.
Making that change makes the API returns a cleaner response.
No data tags and no messy nesting in our API data anymore.
Conclusion
I like challenging my assumptions. Being outside my comfort zone is how I learn best. I’ve been making small scripts similar to that using the Client class for a while now, so it was good to understand the limitations in the tool and the tradeoffs using others. How this discomfort shakes out in the React app remains to be seen, but I am sure I’ll figure those problems out too. I hope this helps someone.
-George
Top comments (1)
Hey @georgeoffley no sure if it's a cache thing but both images in the post have almost identical code and I'm not seeing any calls to the dynamo table object. Also the last link that points to the conclusion has the text "The Answer".
Hope you can fix the image so I can see the difference in the calls 😊