AWS Cloudwatch: why is it so hairy to pop data out?

Hudson Tavares — Sat, 19 Mar 2022 11:41:22 +0000

For a system using the Amazon cloud provider, it is unavoidable that you may experience some issue that requires checking the logs for some service like API Gateway, Elastic Beanstalk, EC2, or similar, to identify the problem.

I don't know about you, but I always leave this kind of task with a feeling like "we could have a tool inside of a nice UI working integrated with some log groups, allowing me to [insert some killer features here]."

But then I remember that these logs are in Cloudwatch. And wonder when will be the next time I face its terminal 🥲

Some may argue that Cloudwatch has an Athena connector - but it isn't something you get out-of-the-box, and querying it with Athena is not always the kind of latency your application may afford.

You can also export the logs to S3, both manually or via CLI - which brings this task either to your Google Calendar or using some CRON-like Cloudwatch Event to trigger a lambda to trigger the CLI.

Another option is to stream the logs data to the OpenSearch, formerly ElasticSearch. But it can become quickly expensive.

My drive against all the options above is the outrage: how Amazon dares to offer you a hard limit of only ten transactions per second for querying what is often the most extensive data stream from your application?

Hard to answer. Hard to query at scale, also.

Do not delegate the security to your code if you have don't have to

Hudson Tavares — Sun, 06 Mar 2022 15:28:38 +0000

In modern web-based development, we moved from an era of custom hosting approaches based on physical servers to a services-based system, where cloud-based technologies host the different composing pieces of our software and manage the communication between them.

The own hosting definitions also became code in this new era, making technologies like Terraform, Pulumi, and CDK very popular (31.5k, 11.6k, and 8.5k Github stars, respectively) due to their deployment and orchestration capabilities.

As we deploy our code to the inherently secure cloud environments of Amazon, Google, Oracle, and others, we're prone to embrace the misleading assumption that these cloud providers will handle the main security points. At the same time, we remain responsible just for the code programming details.

This shallow thinking often leads to too permissive applications, which can cause issues like data loss or unintended access to other's data because of the underuse of the permissions systems provided on the own environment where they run. These security definitions outside the application code make them easier to test and more complicated to break by the code changes required over the software lifecycle.

The main points to turn these definitions into software development practices are:

It requires good team-level knowledge about the security options offered by the selected cloud provider. For instance, Amazon has Cognito as its user identity service and IAM for identity and access management.
The developers need a clear understanding of the boundaries of authentication and authorization and how they impact access to resources. Complex systems like SaaS applications may become more secure if they can segment the data they handle in different places per tenant instead of single storage relying on system rules. The access to the own data containers becomes tied to the user's related authorization before activating any code-level permission.
Ensure that automation can reliably evaluate these permissions as part of the continuous integration process, ensuring that the permissions granting and denial are in line with the expected behavior as the codebase evolves.

DEV Community: Hudson Tavares

AWS Cloudwatch: why is it so hairy to pop data out?

Do not delegate the security to your code if you have don't have to