DEV Community: peetss

Postgres locally and in CI.

peetss — Mon, 18 Dec 2023 20:40:55 +0000

It all started with a little nodejs library called pg-mem. It promised an in-memory postgres database that would be a great lightweight option for testing in CI. It was advertised as experimental but figured it would be worth it just for its sheer convenience.

Getting it up and running was easy but I did run into two small issues, certainly not deal breakers.

Some queries return a slightly incorrect result (primary key of joined data a bit mangled when using GROUP BY clauses).
The entire suite of functions available in Postgres are not implemented, so depending on your queries you may have to implement them on your own (ie: jsonb_build_object(...)).

A quick pop onto their Github page revealed at least a few pages of unresolved issues. In any case, tests were running smoothly both locally and in CI so I was happy.

Sadly, as queries became more complex, pg-mem became less able to cope and over time it became clear that we had... Outgrown each other. To be clear, this isn't a knock on pg-mem, not even a bit. In fact, the opposite, it's a great library I even attempted to fix the bugs I encountered. Finding enough time to work for free is just the reality of open source development. I have no doubt that with more resources it could truly realize its potential. The allure of an in-memory Postgres database for testing remains strong but I needed proper query execution and so along came Docker.

Docker is an easy technology to work with. It's well-documented, ubiquitous, and there are libraries for it in almost every programming language you can imagine. The only real stumbling block was coming to the realization that a volume needs to be mounted at the absolute path on a Windows system. Once ChatGPT helped me with that, we had one running database in a Docker container. I refactored the tests that relied on weird pg-mem results and we were successfully back to square one. Integrating the nodejs library dockerode to start and stop the container using Jest's global setup/teardown functions further automated the solution.

One weird issue worth mentioning is that sometimes when Jest would exit it would leave a nasty error in my terminal and the Docker container wouldn't spin down. After some research, the issue turned out to be a result of not releasing all the client connections to the database pool. Ensuring client.release() occurs during the finally block of a try in client.query(...) resolved the issue completely.

At this point I was fairly impressed by just how well this worked, thinking little about how CI would dash all my hopes and dreams. After naively thinking dockerode would just work in Gitlab CI, I had to go completely back to the drawing board. After doing some research, Postgres turned out to be pretty simple to setup within the gitlab-ci.yaml file. Unfortunately, it didn't support seeding the database with a schema in any capacity. Apparently, I'm not the only person to bemoan this lack of functionality. My local solution mounted my schema file at dockerinit.d. After some thought I decided to refactor the logic in the globalSetup Jest function to simply start the container, using the pg library directly to insert the schema in the beforeAll method of my test suite. Importantly, the code to start the container would only run if the CI env var injected by Gitlab was not present.

After separating container logic from SQL inserts, we finally had our golden calf. Tests ran both locally and in CI. It wasn't particularly easy to accomplish and documentation on this entire effort was oddly lacking. Hopefully, this post will make it easier for others to accomplish similar outcomes on the future.

Cheers 🍻!

AWS Lambda Custom Authorizer

peetss — Tue, 20 Jun 2023 03:42:40 +0000

If you are like me you probably struggled to get custom authorizers working in Lambda. It probably isn't because you are too dumb to understand, it is because of magic. Magic is great if everything works as expected. When it doesn't, though, it makes debugging an absolute nightmare.

There are four things you absolutely need to get right when working with custom authorizers:

Return the correct policy document.

This one was probably the biggest oversight on my part. In order for your custom authorizer to essentially forward the request to your Lambda it needs to return a valid policy document. In short, the policy document returned by the authorizer function is evaluated by API Gateway to determine whether the client is authorized to access the requested resource.

The example in the AWS docs works well and you can simplify it as needed.
Callbacks vs async/await

Many of the custom authorizer examples (even more recent ones) were written with callbacks in mind. For example, you'll see most custom authorizer functions defined with the callback argument and then a response starting like callback(null...), ironically indicating success. Ok. Searching about how to convert this to using async/await was oddly time-consuming, especially considering I thought this would be a problem many people would've run into.

Eventually I found what I was looking for, courtesy of stackoverflow. The answer was comically simple, just return the policy document object or throw an error.

Great, no more callbacks.
throw new Error("Unauthorized")

Think you can just return whatever error you want based on your specific domain model? So naïve. AWS is very specific in the errors you can return but to their credit it is documented. Again, this was another area where I spent more time than I care to admit agonizing about why my Lambda wasn't returning the error code I expected.
CORS

Of course, a "gotchas" list like this wouldn't be complete without an entire section on CORS. Once I solved 1, 2, and 3, I assumed I'd be off to the races only to be presented with a CORS error (using the one and only Swagger UI to interact with my API like a true developer).

Traditional Lambdas allow you to arbitrarily return payloads (including headers) but with custom authorizers this is not possible. Using Serverless (an incredible tool, btw), I learned you need to configure your API Gateway to respond with the proper CORS headers in certain situations, specifically 4xx and 5xx responses from a custom authorizer. This guide was pivotal to me breaking through the final barrier to a working solution.

Now, each of these problems on their own don't sound so significant but when you encounter all of them at once and you don't know what you are doing to start with, it can feel like climbing a mountain. Granted, I probably could've read the documentation more closely but there were a lot of disjointed examples out there that only covered one or two of the four potential pitfalls I ran into.

After all is said and done I am now the proud owner of a robust authorizer middleware that I can easily add to any of my existing Lambda functions and I was able to share my experiences with everyone here.

That joyous moment when you get your first 200 OK building a new app 😌

peetss — Tue, 13 Jun 2023 04:32:39 +0000

To VPC or not to VPC...

peetss — Thu, 03 Nov 2022 00:22:34 +0000

If you are building on AWS you've probably run into the situation where you have a Lambda that needs to talk to a database. Easy right... well, it actually requires some non-trivial knowledge of concepts like VPCs, subnets, and security groups. Probably more than what you initially bargained for.

Of course, you probably made it through this (there are a plethora of tutorials solving this exact thing) and also realized the importance of restricting access to your database by placing it within the confines of a VPC.

Then your use case changed and now you need to make a network request from a public source to your Lambda. Problem is, the Lambda is inside a VPC which is not accessible to the public internet.

After days spent scouring the internet for solutions to this problem, here is what I found.

Get rid of the VPC, who needs 'em anyway? Seriously though... don't do this. The most valuable asset to any application is its data. If you make your database public by taking it out of the VPC, the only thing separating your data from a hacker is an alphanumeric username and password combination. Sure, you could have a strong password but you have to really be comfortable with that risk. Most people aren't.
Even better, just use a NAT gateway. A NAT gateway allows a Lambda in a VPC to be assigned a publicly addressable IP address. Once you make it past the finicky nature of setting up this configuration, it works really well. The Lambda is publicly accessible, can query the database and return a response to the user, just like you drew it up. One caveat though, there is no free tier. To this point, you're likely still paying 0$ for your database and Lambda. However, the NAT gateway is going to cost you a whopping $30 monthly. That is steep, especially if you are trying to build out a serverless stack on the cheap. For me, it just wasn't worth it - there had to be a better way.

This is where it got difficult. There didn't seem to be another way. It looked fairly certain that I would either have to sacrifice security or pay a prohibitive cost. However, I dug to the deepest depths of the internet (Stackoverflow) and eventually the heavens opened up revealing a closely guarded secret, https://stackoverflow.com/a/41559669, which led me to the following,

Use a public Lambda to call the private Lambda. That sounds so obvious in hindsight. But wait, I thought the VPC Lambda wasn't publicly accessible. That certainly is true, but apparently the Invoke API does not adhere to such mortal rules. Now, you can have your cake and eat it too. We retain the security of having the database in the VPC, while also keeping costs at zero. The only downside is you will have to manage several Lambdas which does add some operational overhead. It also slightly complicates the debugging process, as each Lambda has its own logging group and figuring out which one is behaving poorly can be a tad tedious.

Anyways, hope this was education and/or helps someone who finds themself in a similar situation. Curious to see what else the community has to say about VPC, Lambdas, and the like.

Thanks for reading!

A small contribution

peetss — Sat, 05 Feb 2022 03:13:00 +0000

Hi everyone,

Today I was working on adding some HTTP requests to an existing Rust project.

I didn't really know anything about Rust (or Cargo) and something like this would've definitely saved me some frustration and allowed me to get moving more quickly.

https://github.com/peetss/rust-reqwest-example

Maybe if you are looking to try out Rust this might be a nice and easy way to get started. Who doesn't like making HTTP requests?

To be fair, most of this code came from a stackoverflow post, I just turned it into a project that you could import into your IDE of choice.

At the end of the day we are all just standing on the shoulders of giants anyways, right?

Don't give up.

peetss — Sun, 14 Feb 2021 14:43:33 +0000

It may feel disheartening when you continue to throw new ideas at the wall and none of them stick.

It may feel disheartening when you work your ass off for days and weeks and months and no one seems to notice or care.

Don't give up hope, do not stop persisting. Keep throwing ideas at the wall, keep trying to be the change you want to see in the world.

People will notice. Maybe not today, maybe not tomorrow, but someday, for sure, people will notice.