DEV Community: Renato Losio 💭💥

You Know What About Me? Decoding My Digital Trail Across Major Platforms

Renato Losio 💭💥 — Sun, 08 Jun 2025 16:04:42 +0000

Under European rules, users can request personal data from platforms, but few do, and the results are often hard to use. I accessed and parsed data from TikTok, Amazon, Google, and Instagram, uncovering surprising insights and useful tips.

We often hear the phrase "if you're not paying for the product, you are the product." Yet despite our concerns about data privacy and corporate surveillance, very few of us know what data these platforms collect about us. Even fewer take advantage of our legal right to access this information.

While European regulations are sometimes criticized for focusing on mundane issues like standardizing charging cables, there's one European rule that has quietly spread worldwide and genuinely empowers users: the right to data portability under GDPR Article 15.

The Right You Often Don't Remember You Have

Under Article 15 of the GDPR, individuals (data subjects) have the right to obtain a copy of their personal data held by a data controller, as well as information on how and why it is being processed.

This isn't just a European privilege, most global services implement the same data takeout process for all users worldwide, regardless of location, because it's too complicated to maintain separate systems.

The process is straightforward but asynchronous. Companies have up to 30 days to provide your data (extendable to 3 months with good reason), and it must be provided in a structured, machine-readable format—typically CSV files, JSON, or text files. Best of all, it's usually free of charge unless you make excessive requests.

From Duolingo to Dropbox, from small German cloud services to tech giants like Google, virtually every digital service now offers some form of data takeout. You simply navigate to your account settings, request your data, and receive download links via email within days or weeks.

The Reality: Nobody's Looking

Despite this powerful right, actual usage remains remarkably low. In my experience working as a software architect implementing these systems, only a tiny fraction of users ever request their data. The reasons are twofold: users don't know about this option or find it cumbersome, while service providers have no incentive to promote it. After all, data egress from cloud services is expensive, and encouraging users to download their data represents a pure cost with no business benefit.

But the results can be eye-opening. File sizes range from a few megabytes for simple services to multiple terabytes for users with extensive cloud storage or long platform histories.

951825487 Dez 5 12:04 data-takeout-amazon-renato-losio-20241205.zip 5195593 Mai 6 19:51 duolingo.zip 2514113 Nov 30 12:55 TikTok_Data_1732739019.zip 2426598 Apr 6 14:59 TikTok_Data_1743792620.zip 6546575709 Mär 29 2018 gmail.zip 1102106991 Mär 28 2018 google-20180328T154946Z-001.zip

Inside the Black Box: What TikTok Collects

TikTok, despite its controversial reputation, actually provides one of the most user-friendly data takeouts I've encountered. The folder structure is intuitive, with clearly labeled directories like "Activity" containing the most interesting information.

The login history alone reveals extensive tracking: every login is recorded with location data, network type (WiFi vs. mobile), carrier information, and device details. But the real revelation lies in the watch history data.

Date: 2024-10-06 05:43:41 IP: 109.42.113.107 Device Model: iPhone13,3 Device System: iOS 17.5.1 Network Type: 4G Carrier:

For one account I analyzed, the numbers were staggering:

188,000 videos shown through the feed
43,000 videos watched to completion (about 25%)
An average of over 500 videos per day
Peak days reaching nearly 2,000 videos

(Note: data may have delays of up to several days) Videos shared since account registration: 166 Videos watched to the end since account registration: 43494 Videos commented on since account registration: 1375

Using simple data analysis tools (even ChatGPT for non-developers), you can uncover viewing patterns by hour and day of the week, identify inactive periods, and calculate session durations. Most surprisingly, the claim that TikTok's algorithm only needs 260 videos to create addiction proved conservative; this threshold could be reached in less than 17 minutes of usage.

Google's Omniscient Eye

Google's data takeout is perhaps the most comprehensive, covering its vast ecosystem of services. For location data alone, the scale is breathtaking. A typical Android user's location history contains hundreds of thousands of data points spanning years.

In one analysis of 10 years of location data, I found:

332,000 location data points
Roughly 100+ location pings per day
Timestamps are accurate to the millisecond
Activity inference (in vehicle, walking, stationary)
Complete movement patterns mapping every journey

This data is so detailed that you could reconstruct someone's entire decade of movement, identify their home and work locations, track their travel patterns, and even infer their lifestyle habits. The precision is both impressive and unsettling.

{ "timestampMs" : "1505486494021", "latitudeE7" : 525277253, "longitudeE7" : 133817500, "accuracy" : 20, "altitude" : 75, "activity" : [ { "timestampMs" : "1505486658517", "activity" : [ { "type" : "STILL", "confidence" : 71 }, { "type" : "UNKNOWN", "confidence" : 18 }, { "type" : "IN_VEHICLE", "confidence" : 12 } ] }

Amazon's Data Labyrinth

Amazon's data takeout presents a stark contrast to TikTok's user-friendly approach. The folder structure is bewildering, with dozens of cryptically named directories requiring careful exploration to understand their contents.

Hidden within this maze are surprising insights:

Complete lists of advertising audiences you've been placed in
Prime Video location tracking showing which countries you've streamed from
Every PDF invoice from decades of purchases
Detailed records of all product searches and browsing history

For one user's Prime Video data over a two-year period, location tracking revealed that 75% of their time was spent in Germany, 18% in Italy, and in various other locations, creating an accurate picture of their international movement based solely on streaming activity.

Beyond Complaint: Taking Action

The power of data takeouts extends beyond satisfying curiosity. This information enables:

Better Privacy Awareness: Understanding exactly what data companies collect helps inform decisions about privacy settings and service usage.

Parental Conversations: Having concrete data about social media usage patterns provides a factual foundation for discussions with children about screen time and digital habits.

Personal Insights: Identifying usage patterns, behavioral trends, and even recovering lost information (like old receipts buried in years of purchase history).

Corporate Accountability: Companies can only be held accountable for data practices when users understand what data is being collected.

Practical Tips for Data Exploration

For those interested in exploring their digital footprint:

Start Simple: Begin with smaller services before tackling comprehensive platforms like Google or Amazon.

Sanitize Sensitive Data: When using AI tools for analysis, remove personal identifiers and URLs to protect privacy.

Focus on Patterns: Look for trends and usage patterns rather than processing every data point.

Use Available Tools: Non-developers can leverage ChatGPT or similar tools to analyze data patterns without coding skills.

Expect Variety: There's no standard format across platforms, but most use JSON, CSV, or plain text files that are reasonably accessible.

The Path Forward

Privacy begins with awareness. We cannot meaningfully discuss data rights, corporate responsibility, or digital privacy without first understanding what information we're sharing. The tools exist (legally mandated and technically accessible) to pull back the curtain on our digital footprints.

Rather than simply complaining about data issues, we can take concrete action. Request your data, explore what's there, and use that knowledge to make informed decisions about your digital life. The companies won't advertise this option, but they're legally required to provide it when asked.

Your data belongs to you. The first step in taking control is knowing what's there to control.

The author presented this session at re:publica 25, demonstrating live data analysis from major platforms while protecting personal privacy through data sanitization techniques. The first draft of this article has been generated starting from the session audio, using Amazon Transcribe and Claude Sonnet 4.

Around the World in 15 Buckets

Renato Losio 💭💥 — Sun, 15 Dec 2024 15:48:49 +0000

Today, we’ll embark on a little journey around the world using S3. We’ll copy various objects across AWS regions worldwide, aiming to answer two simple questions:

Does the storage class affect the speed?
How does the size of the object influence the transfer time?

Verne’s Journey

Imagine wagering your entire fortune on a wild, globe-trotting adventure. That’s exactly what Phileas Fogg, a meticulous and mysterious Englishman, does in Jules Verne’s Around the World in 80 Days, racing against time and overcoming a few unexpected obstacles.

While Fogg sets out to prove he can circumnavigate the planet in just 80 days, we want to prove we can circle the globe through AWS data centers in just a few seconds—copying S3 objects worldwide in the fastest way possible. Fogg begins in London, traveling through Europe, Asia, the United States, and back to Ireland before returning to London. Can we achieve something similar with Amazon S3? AWS doesn’t (yet) have 80 regions, but we don’t need them. Instead of using all the available regions, we’ll stick to fifteen, just enough to follow Fogg’s route.

Ruling Out S3 Replication

First off, do we need to implement the logic ourselves? Can’t we just useS3 replication? As a lazy cloud architect, that would definitely be my favorite approach. It’s the first thing I’d consider to send my objects worldwide in 80 buckets.

However, replication does not work on cascade and it is an asynchronous operation that can take up to 24–48 hours. You can reduce the replication time to an SLA of 15 minutes by paying a bit more—but even that’s still not the fastest way to move objects globally. I want to be as fast as possible when moving my data, so I’ve decided to implement this journey myself.

A Simple S3 Setup

First, we need a bucket in every region we plan to use. Here’s the list of regions for our experiment.

verne_buckets=("eu-west-2" "eu-west-3" "eu-south-1" "il-central-1" "me-south-1" "me-central-1" "ap-south-1" "ap-southeast-1" "ap-east-1" "ap-northeast-1" "us-west-1" "us-west-2" "us-east-2" "eu-west-1" "eu-west-2")

It’s just a subset—we could add a few more (like Northern Virginia), but doing so wouldn’t significantly affect the results. If you want to replicate this experiment yourself, keep in mind that some newer regions aren’t enabled by default, so you’ll need to take care of that first.

Another hurdle is that you can’t use the same bucket name in different regions, which means all our bucket names need to be unique. How can we manage that? The simplest—though not the most elegant—approach is to create a reasonably unique prefix and append the region code to the name. For example verne-demo-us-east-1. Once we have an array of the regions we’re using, we can use the AWS CLI to create those buckets.

s3prefix="verne-demo"

for region in "${verne_buckets[@]}"

do

   bucket_name="$s3prefix-$region"

   echo "aws s3api create-bucket --bucket $bucket_name --region $region --create-bucket-configuration LocationConstraint=$region"

done

We can then upload files of different sizes to our very first region, London—the city where Verne’s journey begins, and so does ours. In theory, we could upload objects up to 5TB (the maximum size of an object on Amazon S3). However, to keep the test simple—and the storage and transfer costs negligible—we’ll stick to the following items and sizes.

How can we now copy data between buckets around the world? The most obvious and elegant way is to go serverless, using AWS Lambda and Amazon S3 event notifications. When an object lands in a bucket, you can trigger a Lambda function to copy it to another bucket, and so on.

But I’m a lazy developer, and I just need to answer a few simple questions. I don’t want to deploy Lambda functions everywhere and deal with a lot of setup. Thankfully, with the AWS CLI and AWS CloudShell, the task is straightforward and the overhead minimal. When you copy an object between two regions using the CLI, the binary doesn’t pass through your terminal—it’s handled directly between AWS data centers.

for (( i=0; i<$array_length-1; i++ ))

do

aws s3api copy-object --copy-source $s3prefix-${verne_buckets[$i]}/$objectname --key $objectname --bucket $s3prefix-${verne_buckets[$((i+1))]} --region ${verne_buckets[$((i+1))]} --storage-class $testclass

done

On Our Way

Let’s run our simple tests. You’ll immediately notice that using the S3 copy-object operation makes our journey impressively fast. You can circle the globe in just 23 seconds! However, as the object size increases, transfer times start to scale linearly. For example, compare the times for 100 MB versus 500 MB. At smaller sizes, the overhead of the commands becomes more noticeable.

	s3api copy-object	s3 cp (multipart)
verne-1K	23s
verne-1M	31s
verne-10M	56s
verne-100M	5m25s	3m40s
verne-500M	25m42s	5m27s

What happens when you switch to the multipart scenario? The growth is no longer linear, and it’s impressive that you can copy a 500 MB object across 15 buckets in about 5 minutes using the default client configuration. No tuning of the S3 client configuration was performed.

storage class	s3api copy-object 1M
STANDARD	30.4s
STANDARD-IA	31.0s
ONEZONE-IA (*)	31.2s
GLACIER-IR	30.9s
REDUCED_REDUNDANCY (*)	29.6s

With multiple storage classes available, does the storage class matter? Two storage classes—the deprecated S3 Reduced Redundancy and S3 One Zone-Infrequent Access—don’t use three separate AZs. Would that offer any speed advantage? The short answer is no. All storage classes that provide immediate access to the object, including Glacier Instant Retrieval, behave similarly.

Conclusions

This isn’t a benchmark or even a proper test. It’s just a fun experiment to validate some ideas about Amazon S3: the storage class of your objects doesn’t affect the time needed to copy them across regions, and the transfer time grows linearly with the object size—unless you use multipart upload.

Also, Amazon S3 and AWS networks are impressively fast.

S3 Lifecycle or Intelligent-Tiering? Object Size Always Matters

Renato Losio 💭💥 — Fri, 13 Dec 2024 12:02:41 +0000

As a cloud architect and storage expert, I often hear the question: Which storage class should I use on Amazon S3? The answer is simple: as always, it depends. Today, we’ll explore some anti-patterns you might overlook when transitioning data between storage classes, whether using lifecycle rules or a managed class like S3 Intelligent-Tiering.

A Short Journey Back in Time

Amazon S3 was announced almost 19 years ago. It’s one of the oldest services on the platform, second only to Amazon SQS. What we see today as Amazon S3 vastly differs from what was available many years ago.

Even just a few years ago, there was no S3 Storage Lens, no conditional writes (introduced earlier this year), and no S3 Intelligent-Tiering storage class. There were no S3 triggers, there was no replication across buckets or regions. When I started working with S3 (around 2010), it was a fairly limited service but, in some ways, simpler than it is today.

At the same time, we tend to forget how expensive it was. As part of the announcement 19 years ago, Jeff Barr wrote:

“1 GB of data for 1 month costs just 15 cents.”

The same storage class today costs just 2.3 cents (or less) per GB in the US East region. With additional storage classes designed to optimize costs, it can go below 0.1 cents per GB.

Storage Class	Storage Price per GB
Standard	2.3 cents (or less)
Standard Infrequent Access	1.25 cents
Glacier Instant Retrieval	0.4 cents
Glacier Deep Archive	Less than 0.1 cents

AWS keeps iterating, adding new features and options. As a cloud architect, your job is to stay up to date. If you deployed and optimized costs on S3 five years ago, it’s time to revisit your setup. There’s a good chance your deployment is no longer optimized, and you might be leaving money on the table. Keep iterating!

How to Choose a Storage Class

Choosing the right storage class depends on the scenario, use case, and associated costs. The price isn’t just about storage. Costs also depend on how you access your data, how you upload it, and how you move it around. So, deciding on the best storage class isn’t always straightforward.

Hot storage refers to data that is frequently accessed and requires fast, consistent response times. On the other hand, cold storage is designed for data accessed infrequently and without the urgency required for hot data. The challenge? The same data often transitions from hot to cold over time.

Different classes come with varying retrieval costs, latency, availability, and minimum time commitments. Some classes store every copy of the data in a single Availability Zone, while others replicate data across three different ones. Different requirements mean different use cases. So, the big question is: how do you choose your storage class?

There are some common patterns. Typically, new data is “warmer.” Over time, the likelihood of accessing that data decreases. This isn’t unique to AWS or object storage. For example, Dropbox has shown that even in backup solutions, most retrieved data is newly uploaded:

“As we have observed that 90% of retrievals are for data uploaded in the last year and 80% of retrievals happen within the first 100 days”

Ideally, you want recent data in hotter storage classes and older data in colder ones.

Lifecycle Rules Help

How can you move data across storage classes? On S3, the easiest way is with S3 Lifecycle. These rules are incredibly powerful—you can decide which storage class to use, the size of objects to move, and when to move them. For example, I can move my data to a cheaper storage class after 45 days because I know I will not need to access it anymore. I can also specify that only large files are moved to colder storage classes, while smaller files (like thumbnails) stay in the Standard class to avoid a higher chance of retrieval fees.

Why do I focus on size? Object size matters for lifecycle rules and, as we will see shortly, for S3 Intelligent-Tiering as well. You pay for operations (transitions) by the number of objects, while storage is charged by size. What does that mean in practice? Let’s break it down with a simple example.

Imagine I have 1 PB of data to move from Standard Infrequent Access (Standard-IA) to Glacier Instant Retrieval (Glacier-IR). How much does the transition cost? How does it impact my overall S3 costs? And how long does it take to recoup my transition expenses?

	1GB size objects	1MB size objects	200KB size objects
Transition to GIR:	1M	1000M	5000M
Cost of transitions (one off):	20 USD	20000 USD	100000 USD
Storage saving (monthly):	17000 USD	17000 USD	17000 USD
Recoup time:	< 1 day	1+ month	6 months

If my 1 PB consists of 1 GB objects, I’ll pay about $20 for the transition, while monthly storage costs would be approximately $17000. However, as object sizes decrease, transition costs become less trivial. You might still choose to move the data, but it’s important to understand your upfront costs.

The best way to estimate these costs is to use S3 Storage Lens. Ideally, you want most small objects in the Standard class and larger objects in colder classes. Below is an example of a distribution for an almost 20 PB bucket.

A more detailed example was shared by Canva last year on the AWS Storage Blog: How Canva saves over $3 million annually in Amazon S3 costs

What about S3 Intelligent-Tiering?

Why develop your logic when Amazon provides a managed class that handles this for you, without charging transition or retrieval fees? S3 Intelligent-Tiering is indeed the ideal choice for most use cases and the default choice for many workloads.

However, it’s still important to understand how the S3 Intelligent-Tiering class works under the hood and why object size remains a key factor. Let’s start with the AWS documentation:

“For a small monthly object monitoring and automation charge S3 Intelligent-Tiering moves objects that have not been accessed for 30 consecutive days to the Infrequent Access tier for savings of 40%; and after 90 days of no access they’re moved to the Archive Instant Access tier with savings of 68%. If the objects are accessed later S3 Intelligent-Tiering moves the objects back to the Frequent Access tier .”

The bold emphasis is mine. Now, how much is the “small monthly object monitoring and automation charge”? What are the implications of moving objects back? Let’s revisit the example of 1 PB of data. How much would this small management fee cost?

1GB objects	1MB objects	200KB objects
1M	1000M	5000M
2.5 USD	2500 USD	12500 USD

Once more, the answers depend on the average size of your objects. The management fee might be negligible, or it might cost you more each month than the storage itself.

With S3 Intelligent-Tiering, there are no retrieval fees, but the object is moved back to the most expensive Frequent Access Tier (Standard class) for 30 days and then stays for another 60 days in the Infrequent Access tier. Retrieving one object means you’re not paying a retrieval fee, but you are paying more for the storage of the object for the next 90 days. How much is that extra overhead?

	1GB objects	1MB objects	200K objects
Intelligent-Tiering	48.5 USD	48.5 USD	48.5 USD
GIR (once)	30 USD	30 USD	30 USD
GIR (10x)	300 USD	300 USD	300 USD

Here, the key metric is the number of times you expect to retrieve that specific (colder) object after the first retrieval. Should you not use the managed storage class? Far from it. S3 Intelligent-Tiering is an amazing option, but—as with any managed service—it’s critical to understand the cost implications. Your retrieval patterns and the average storage size of your object matter.

Whenever you develop your logic with Lifecycle Rules or you delegate the logic to the S3 Intelligent-Tiering class, the average size of your objects on S3 is a significant factor in your storage costs. Move larger objects first; that’s where most of the storage saving lies.

Amazon Aurora is Now 60 Times Faster than RDS for MySQL. Really.

Renato Losio 💭💥 — Wed, 26 Jul 2023 16:52:06 +0000

Today I will perform ~~a simple test~~ a proper benchmark, comparing Aurora MySQL and RDS for MySQL.

Unlike traditional benchmarks that are lengthy and complex, this evaluation will be concise, comprising only about 200 words and taking just a couple of minutes to review the results.

First, let’s create two instances, using the AWS CLI. To ensure a fair comparison, we will create two instances using default values. Both instances will have the same class and size, db.t4g.medium, and will be located in the same AWS region, Frankfurt.

# Aurora

$ aws rds create-db-cluster --db-cluster-identifier benchmark --engine aurora-mysql --engine-version 8.0 --master-username renato --master-user-password ******** --db-subnet-group-name renato --vpc-security-group-ids sg-030c9f25422a13fea
$ aws rds create-db-instance --db-instance-identifier benchmark --db-cluster-identifier benchmark --engine aurora-mysql --db-instance-class db.t4g.medium

# RDS

$ aws rds create-db-instance --db-instance-identifier benchmark-rds --engine mysql --db-instance-class db.t4g.medium --master-username renato --master-user-password ******** --allocated-storage 100 --vpc-security-group-ids sg-030c9f25422a13fea

The benchmark process will involve creating a database, a table, and a MySQL procedure on both endpoints. This procedure will simply insert 100K records into the table, generating random values to introduce some data variety.

CREATE DATABASE renato;
USE renato;

CREATE TABLE renato
(id bigint(20) NOT NULL AUTO_INCREMENT,
datetime TIMESTAMP NULL DEFAULT CURRENT_TIMESTAMP,
value float DEFAULT NULL,
PRIMARY KEY (id));

DELIMITER $$
CREATE PROCEDURE load_renato()
BEGIN
  DECLARE i INT DEFAULT 0;
  WHILE i < 100000 DO
    INSERT INTO renato (datetime,value) VALUES (
      FROM_UNIXTIME(UNIX_TIMESTAMP('2023-07-01 01:00:00')+FLOOR(RAND()*31536000)),
      ROUND(RAND()*100,2));
    SET i = i + 1;
  END WHILE;
END$$
DELIMITER ;

Finally, we will call the procedure on both databases, Remember: default configuration, same region, same class, same size. No networking is involved. No magic. No tricks.

# Aurora

mysql> call load_renato;
Query OK, 1 row affected (5,41 sec)

mysql> select count(*) from renato;
+----------+
| count(*) |
+----------+
| 100000 |
+----------+

# RDS

mysql> call load_renato;
Query OK, 1 row affected (5 min 20,41 sec)

mysql> select count(*) from renato;
+----------+
| count(*) |
+----------+
| 100000 |
+----------+

The results are striking: Amazon Aurora outperforms RDS significantly, completing the task in just 5.4 seconds compared to RDS’s 5 minutes and 20 seconds. This remarkable 60x difference clearly demonstrates the superiority of Aurora in this specific scenario.

WOW, you are kidding me!

Just remember this simple test every time you read “50% faster”, “47% faster and costing up to 43% less” or “127% better IOPS”. This benchmark serves just as a reminder that real-world tests are crucial in understanding the true capabilities of a database.

As they used to say, there are three kinds of lies: lies, damned lies, and ~~statistics~~ database benchmarks.

What Do Cloud Services and Jam Have in Common?

Renato Losio 💭💥 — Tue, 25 Apr 2023 10:38:23 +0000

In this short video I talk about the similarities between cloud services and jam. Yes, jam.

I explain how cloud services offer a variety of options for users to choose from, but too many options can lead to decision paralysis. Yes, the famous jam experiment.

Unlocking the Secrets of the Magic Number 35

Renato Losio 💭💥 — Tue, 14 Mar 2023 15:56:29 +0000

When you run a database on RDS, regardless of the engine you use, you can set the automatic backup retention period to any number between 0 and 35 days. Running a MySQL 8.0? You can do a point in time up to 35 days in the past. Not 40 days, not 50 days, 35 days is the maximum. Oracle? 35 days. SQL Server? The continuous backups have a maximum retention of 35 days.

But why 35? What is so special about the number 35?

This is an inherent hard limit tailored for automated backups and providing point-in-time recovery, not for the purpose of archiving data for the long term. And 35 days has been the maximum for over 10 years since AWS increased the backup retention period from a maximum of 8 days in 2012. Jeff Barr wrote at that time:

After fielding a number of customer requests, we have increased the maximum retention period from 8 days to 35 days. A number of our customers have asked for this new limit in order to retain at least one month’s worth of backups for compliance purposes.

Source: https://aws.amazon.com/blogs/aws/relational-database-service-increased-snapshot-retention-period/

You might have the two-pizza team rule in AWS, but the number became quickly the limit in other services too, not RDS only.

Point-in-time recovery for DynamoDB? You can restore that table to any point in time during the last 35 days. What about Aurora? You cannot disable automated backups on Aurora, but you can choose any number as long as it is between 1 and 35. DocumentDB? It continuously backs up your data to S3 for 1 to 35 days. The number is now a de-facto standard, not only on AWS.

What about other cloud providers? Azure has 35 days:

Log backups can occur as often as every 15 minutes and can be retained for up to 35 days.

What about Google? AlloyDB recently announced:

Continuous backups have a retention period of between 1 and 35 days, depending upon the configuration that you choose when setting up this feature.

And other services on GCP and Azure match the 35 days too. But why is everyone on 35? Why not 30? Or 33 days? We can only guess:

Once a service uses 35, it is easier to converge on one value. For all the services of the same provider at least. Even Azure confirmed that when they increased the value for an existing service: “This change reflects our goal of providing a consistent user experience across service tiers”
To retain at least one month’s worth of backups for compliance purposes, you need to consider the longest month (31 days) and add a buffer for a (long) weekend that would cover an 8x5 monitoring and support scenario. Not everyone is on 24x7.
35 is a tetrahedral number. I am sure it matters.

Do you have a better explanation for the magic 35? Let me know below or on Linkedin.

A new CRT HTTP Client, a Reference Architecture for Deployment Pipelines and the VPC Resource Map

Renato Losio 💭💥 — Tue, 28 Feb 2023 10:31:37 +0000

As an editor for InfoQ, every month I follow and (try to) cover the most interesting news in the cloud space. From VPC Resource Map to the new CRT HTTP Client in AWS SDK for Java 2.x, below are the announcements and news in the AWS space that caught my attention in February. You can read the full articles on InfoQ.

AWS Adds VPC Resource Map to Simplify Management of Virtual Networks

The VPC section of the AWS Management Console now provides visualization of VPC resources, such as the relationships between a VPC and its subnets, routing tables, and gateways. The map displays existing VPC resources and their routing on a single page, allowing a better understanding of the networking layout.

AWS Publishes Reference Architecture and Implementations for Deployment Pipelines

AWS recently released a reference architectureand a set of reference implementations for deployment pipelines. The recommended architectural patterns are based on best practices and lessons collected at Amazon and customer projects.

Faster Startup Time and Lower Memory Usage: New CRT HTTP Client in AWS SDK for Java

AWS recently announced the general availability of the Common Runtime (CRT) HTTP Client in the AWS SDK for Java 2.x. The new asynchronous client provides faster SDK startup time and a smaller memory footprint improving Lambda serverless workloads.

AWS Patches Undocumented APIs Bypassing CloudTrail Event Logging

AWS recently patched undocumented IAM APIs that bypassed CloudTrail logging. The vulnerability allowed a malicious user to perform reconnaissance activities on IAM without recording events in CloudTrail or being detected by Amazon GuardDuty.

To read my other articles on InfoQ, follow me.

My Journey Through the AWS Blog: Lessons Learned and Random Statistics

Renato Losio 💭💥 — Thu, 26 Jan 2023 20:41:03 +0000

Last week I checked the (primary) AWS blog. I went through all the articles from the welcome post in 2004 until the very last one. I did not read all of them. I just collected some data using Python and a Bash script.

Random Numbers

Let’s start with some random numbers. There are until today 3925 posts.

The year with the most posts? 2015 with 320 posts, almost one a day. AWS then realized more “secondary” blogs were needed. You now have more content to read, but spread across multiple blogs.

Top month? Well, November of course, 571 posts. The re:Invent effect, with significant spikes in the last few years. Quieter month? February with 229.

I plot the month-by-month over the years we can see how the role of the primary AWS blog has shifted and how it is impacted by announcements in Vegas.

The top day ever? A re:Invent day (December 3rd, 2019) with 27 articles.

Top author? Jeff Barr, of course. He appears as the author of 80% of the posts (3136) but this includes a small number of guest posts and posts that in earlier years were coming from other authors but were posted by Jeff himself.

Frugality or Complexity?

I love discussing useless numbers but you might wonder, what was I looking for?

Trigger by jeff Barr’s recent “Leadership Principles for AWS News Bloggers”, I was curious about how much longer (and usually more complex) posts became over the years.

As an InfoQ editor and as a cloud architect, I struggle to keep myself up-to-date even with just the biggest announcements and the main AWS blog alone. Jeff Barr writes:

Be frugal with your own time and with that of your readers (…) Keep posts concise and don’t spend the readers’ time unnecessarily.

Without any complex analysis, I just checked the length of the page in bytes, removing the fixed size of the page’s look and feel. With a few exceptions aside, including a Transcript of Jeff Barr’s Web 2.0 Interview, all the longest articles are very recent.

Posts are getting longer. Are the authors not frugal enough? The platform gets more and more updates and features, increasing the complexity of the announcements too.

S3 and Route53

Let’s compare a couple of articles, from the early times and from recent posts.

Introducing Amazon Route 53 Application Recovery Controller (2021)

Almost 3000 words and over 30 screenshots and diagrams are required to describe how the new feature of Route53 works.

Amazon Route 53 – The AWS Domain Name Service (2010)

About 600 words , not a single screenshot, and a rudimental Route53 logo were enough to announce the service and its main features.

What about an even shorter post?

Amazon S3 (2006)

Just over 200 words , one of the shortest posts ever, were enough for arguably one of the most important announcements in the history of AWS, Amazon S3. Jeff Barr had (literally) a plane to catch:

I’ve got to catch a plane to Silicon Valley in a few minutes, or I’d write a lot more.

Amazon S3 Encrypts New Objects By Default (2023)

By comparison, the most recent announcement about S3, the change in the default of a checkbox, required four times more words.

I wonder how long would be an S3 roundup 17 years later.

Conclusions

Long gone are the days of deployments of web applications relying solely on EC2, RDS, and S3. AWS now claims “over 200 fully featured services from data centers globally” each with multiple options, pricing models, and deployment options.

There’s never been a better time to be a developer. But there’s never been a harder time to make the correct choices. Even following just the primary AWS blog requires a lot of effort. And it is only one of the different blogs offered by the cloud provider.

There is nothing wrong with the authors, I am sure they try to be as frugal as possible but it is getting harder to keep ourselves up to date.

Enough, we are at just over 700 words already.

The post My Journey Through the AWS Blog: Lessons Learned and Random Statistics appeared first on cloudiamo.com.

The Best AWS re:Invent Ever? Looking Back at Eleven Editions

Renato Losio 💭💥 — Fri, 13 Jan 2023 11:00:39 +0000

There is no year that I do not read many “This was the best re:Invent ever!” comments, posts, or articles in the weeks after the AWS conference in Las Vegas. While it is likely true in terms of fun, the number of announcements, or personal networking, I tend to think we are biased by the latest edition.

What was the biggest re:Invent announcement ever? Keep in mind that crucial services like EC2, EBS, RDS, S3, SQS, or IAA were available well before the first edition. Even DynamoDB was announced at the beginning of 2012 before any AWS global conference.

Let’s look back and see which services have stood the test of time, starting from the first edition in 2012, highlighting one announcement per year.

2012

At the very first re:Invent there was the preview of the data warehouse service Amazon Redshift. Here was the original post.

2013

AWS CloudTrail was announced to address audit requirements. Maybe not an exciting service but I cannot think of any other one that is so widespread and used almost ten years later. Few lines were enough for the announcement.

Photo by David Vives on Unsplash

2014

Yes, there were Amazon Aurora, ECS, AWS Config, and AWS Key Management Service. Who cares, nothing matters, this was the year of AWS Lambda. You can go back in time here.

2015

Kinesis Firehose? QuickSight? AWS Mobile Hub? Maybe Amazon Inspector? I find it hard to highlight a single one that stands out. I vote for AWS Snowball, the rugged appliance for efficient data storage and transfer.

2016

This was the year of the first so-called Amazon AI services, including the Rekognition image recognition service and almost all Amazon Lex. But if I have to choose one announcement, I would say, Amazon Athena, today still more popular than the Snowmobile truck that appeared on stage. Really.

2017

There were Alexa for Business, the Serverless App repository, and AWS Cloud9. Aurora Serverless (the now v1) and Amazon Neptune were interesting but it does not matter. This was the year of Kubernetes, EKS, and Fargate.

2018

Quantum Ledger Database? Amazon Managed Blockchain? Global Accelerator? I have to highlight S3 Intelligent Tiering. Yes, not a new service but a paradigm shiftfor the object storage service.

Photo by Grant Cai on Unsplash

2019

It starts to get harder to pick up one announcement. There are too many and it is too early to say the long-term impact. Tempted by Amazon SageMaker Studio or Amazon EventBridge, I vote for the new type of infrastructure deployment, Local Zone.

2020

My very first recap of the conference for InfoQ, a subdue virtual-only but free edition. As an AWS Data Hero and a MySQL enthusiast, I am biased to highlight the preview of Aurora Serverless v2. I had higher hopes for AWS Fault Injection Simulator but I will single out (yes, no kidding) Cloudshell.

2021

Back in person in Las Vegas, this was the year of the Graviton3 chip, with more instances still to come.

2022

We are finally at 2022. What will be the key announcement that will last long term? Who knows, SimSpace Weaver, Clean Rooms, CodeCatalyst, Amazon Omics? My first feeling is that nothing will match the early editions, but I vote for Security Lake.

Was 2022 the best re:Invent ever? What is your favorite edition or announcement? Let me know your thoughts and follow me for more posts.

The article appeared first on cloudiamo.com.

AWSome December, from OpenSearch Serverless to Application Composer

Renato Losio 💭💥 — Sun, 01 Jan 2023 09:30:00 +0000

As an editor for InfoQ, every month I follow and (try to) cover the most interesting news in the cloud space. From Blue/Green Deployments for RDS to Clean Rooms, below are the announcements and news in the AWS space that caught my attention in December. You can read the full articles on InfoQ.

AWS Announces GA of DocumentDB Elastic Clusters

At the recent re:Invent conference, AWS announced the general availability of DocumentDB Elastic Clusters, a service that manages the elasticity and sharding for MongoDB workloads.

AWS Previews VPC Lattice for Service-to-Service Communication

To simplify networking for service-to-service communication, AWS recently announced the preview of Amazon VPC Lattice. The new capability of Virtual Private Cloud (VPC) abstracts network complexity and creates a logical application layer network that connects clients and services across different VPCs and accounts.

AWS Previews Application Composer to Visualize and Create Serverless Workloads

At the recent re:Invent conference, AWS announced the preview of Application Composer, a visual designer to build serverless applications from multiple AWS services. The new option helps create the architecture by dragging, grouping, and connecting services in a visual canvas.

AWS Announces Clean Rooms for Secure Collaboration with Analytics Data

During the recent re:Invent conference, AWS announced the preview of Clean Rooms for analytics data. The new service provides safe environments where multiple customers can securely share and analyze data with control of how the data is used, reducing the risk of sharing personal data.

Amazon Announces Preview of OpenSearch Serverless

AWS recently announced the preview of OpenSearch Serverless, a new option of OpenSearch service that automatically provisions and scales the resources for data ingestion and query responses. The minimum capacity required for the serverless option raised some concerns in the community.

AWS Announces Blue/Green Deployments for MySQL on Aurora and RDS

At the beginning of the re:Invent conference, AWS announced the general availability of RDS Blue/Green Deployments, a new feature for Aurora with MySQL compatibility, RDS for MySQL, and RDS for MariaDB to perform blue/green database updates.

To read my other articles on InfoQ, follow me.

Blue/Green Deployments for RDS: How Fast is a Switchover?

Renato Losio 💭💥 — Tue, 13 Dec 2022 11:46:23 +0000

AWS recently announced the general availability of RDS Blue/Green Deployments, a new feature for RDS and Aurora to perform blue/green database updates. One of the aspects that caught my eye is how fast a switchover is.

In as fast as a minute, you can promote the staging environment to be the new production environment with no data loss.

Let’s give it a try! How can we check how long a switchover really takes? We will perform the simplest test, using only a MySQL client and a while loop in Bash:

while true; do mysql -s -N -h renato-cluster.***.eu-west-1.rds.amazonaws.com -u renato 
-e "select now()"; sleep 1; done

(...)
2022-12-09 07:58:55
2022-12-09 07:58:56
2022-12-09 07:58:57
2022-12-09 07:58:58
(...)

We keep asking every second for the current date and time to the RDS server and we will check what happens when we trigger a switchover. We will compare the results with a reboot and a reboot with failover. To perform the test we will use an m6g.large Multi-AZ instance with some production traffic, a cluster where the Seconds_Behind_Master value of a replica hardly goes above a single-digit number.

Rebooting an instance

Using the CLI, we can perform a reboot of an RDS instance without forcing a failover of the database server.

aws rds reboot-db-instance --db-instance-identifier renato-cluster --no-force-failover

(...)
2022-12-09 08:00:01
2022-12-09 08:00:02
2022-12-09 08:00:03
2022-12-09 08:00:23
2022-12-09 08:00:24
2022-12-09 08:00:25
(...)

That is just about 20 seconds for a reboot of an RDS instance.

Rebooting with failover

We can now repeat the test forcing a failover:

aws rds reboot-db-instance --db-instance-identifier renato-cluster --force-failover

(...)
2022-12-09 08:01:11
2022-12-09 08:01:12
2022-12-09 08:01:13
2022-12-09 08:03:28
2022-12-09 08:03:29
2022-12-09 08:03:30
(...)

This time it takes longer, including the time required to switch the CNAME of our database to the new IP address: 2 minutes and 15 seconds.

Switching a blue/green deployment

Finally, we can trigger a switchover in our blue/green deployment calling:

aws rds switchover-blue-green-deployment --blue-green-deployment-identifier renato-cluster-bg

(...)
2022-12-09 08:06:59
2022-12-09 08:07:00
2022-12-09 08:07:01
2022-12-09 08:08:07
2022-12-09 08:08:08
2022-12-09 08:08:09
(...)

The switchover time of the RDS Blue/Green Deployments is just about a minute: 1 minute and 6 seconds.

Recap

Reboot: 20 seconds
Reboot with failover: 135 seconds
Switchover: 66 seconds

The switchover of an RDS for MySQL instance is “as fast as a minute” and significantly faster than a reboot with failover of a Multi-AZ instance. Of course, the switchover can take significantly longer according to the write load of your production database. This was only a basic test to validate the minimum downtime for a switchover.

Questions? Comments? Contact or follow me.
Originally published on cloudiamo.com.

My 7 Favorite Announcements from AWS re:Invent 2022

Renato Losio 💭💥 — Mon, 05 Dec 2022 14:27:01 +0000

Were you overwhelmed by 100s of announcements at re:Invent last week? Here is the ultimate list of the 7 most important ones 💥.

Just kidding, this is only my very personal selection (as a cloud architect and an AWS Data Hero) of the announcements that I found curious, useful, or a significant step forward 🙌.

You can find my full recap of re:Invent 2022 on InfoQ.

Let me know what you think and if I missed something obvious!

RDS Blue/Green Deployments

At the beginning of the conference, AWS announced the general availability of RDS Blue/Green Deployments, a new feature for Aurora with MySQL compatibility, RDS for MySQL, and RDS for MariaDB to perform blue/green database updates. I love MySQL and I love RDS, this has to be on the top of my list.

Lambda SnapStart

Lambda SnapStart is now available for Java. It has been one of the announcements better received by developers as it addresses one of the limitations of implementing serverless Java applications.

SimSpace Weaver

The new SimSpace Weaver service targets a niche market that runs large-scale spatial simulations at scale, avoiding that a developer is limited by the computing and memory of the hardware.

Aurora zero-ETL integration with Redshift

Enabling near real-time analytics and machine learning, Aurora's "zero-ETL" integration with Redshift is now available in preview.

OpenSearch Serverless

Controversially priced, OpenSearch Serverless manages the provisioning and scaling of the resources to deliver data ingestion and query responses using ElastingSearch-compatible APIs. Not everybody is convinced that naming “serverless” every autoscaling service is a good idea.

DataZone

DataZone is a management service to share, search, and discover data at scale across organizational boundaries. All the data in DataZone is governed by access and use policies that the organization can define.

CloudWatch Internet Monitor

Monitoring data about internet traffic before it reaches the application layer, Internet Monitor uses the connectivity data that AWS captures from their global networking to determine a baseline of performance and availability.

Clean Rooms

At the end of the first keynote, Adam Selipsky announced Clean Rooms. With a questionable name for a company that recently acquired iRobot, the new service (in preview) helps collaborate with other companies on AWS without sharing or revealing underlying data.

This is just a very personal and small selection of the top announcements of AWS re:Invent 2022. You can find my full recap of the conference on InfoQ.

What were your favorite announcements? I would love to know what you think 💚.