A data lake is an architectural approach we can use to centralize all our data. We can then access, transform, and analyze that data with the use of AWS services and tools.
Here is what we build:
Data lakes provide several benefits:
- Scalability: you can store large amounts and many types of data in one place.
- Discoverability: you can find out what data resides in the lake through crawling, cataloging, and indexing.
- Security: you can protect all your data assets with encryption and access controls.
- Shareability: data scientists, data developers, and business analysts can use the same data with the analytics and machine learning (ML) tools of their choice.
- Agility: serverless services remove the undifferentiated heavy lifting of managing infrastructure, so you can focus on the data instead of servers.
In this project we build a serverless data lake. The serverless data lake uses storage with Amazon S3. The serverless data lake also uses event‑driven processing with AWS Lambda and Amazon EventBridge. The next natural steps are cataloging the serverless data lake with AWS Glue. Querying the serverless data lake, with Amazon Athena.
- Data-Generator (Lambda) creates a CSV of random sales orders and writes it to the raw bucket.
- Amazon S3 notifies Data-Processor (Lambda), which totals the orders by category and writes the result to the consumer bucket.
- S3 also sends the same event to Amazon EventBridge. The SalesReportApp rule invokes Sales-Report-App (Lambda), which reads the same raw file, totals the orders by region and category, and writes a report to the consumer bucket.
- Amazon CloudWatch collects the logs of all three functions.
Create the S3 buckets
We use two buckets, one for each zone of the data lake. S3 bucket names are globally unique, so choose your own suffix instead of 313 and use your names everywhere this article mentions a bucket (the IAM policies, the environment variables, and the EventBridge rule). Create all resources in the same AWS Region.
-
Raw bucket (
raw-bucket-demo-313): the raw zone. It stores data in its original format as ingested from the sources, which can be structured, semi-structured, or unstructured: databases, backups, archives, images, and files. In this project, a Lambda function generates the raw data: random sales orders, each with an ID, a product category (a number from 1 to 10), a region, and an amount. -
Consumer bucket (
consumer-bucket-demo-313): the consumption zone. It stores the transformed data. Transformations might include aggregating data from different sources or changing the file format.
To create each bucket:
- Open the Amazon S3 console and choose Create bucket.
- For Bucket name, enter the name.
- Keep the other settings as they are and choose Create bucket.
Confirm the S3 Bucket Security settings
New S3 buckets already block public access and encrypt new objects with SSE-S3 by default. Confirm that both settings are in place on both buckets:
- Open the S3 bucket.
- Navigate to the Permissions tab.
- Under Block public access, confirm that all public access is blocked.
- Under Bucket policy, confirm that no unnecessary public access is granted.
- Navigate to the Properties tab.
- Under Default encryption, confirm that the encryption type is Server-side encryption with Amazon S3 managed keys (SSE-S3)
Create the IAM roles
Each Lambda function needs an execution role that lets it write logs and access the buckets. We create two roles that follow the principle of least privilege:
| Role | Used by | Needs to |
|---|---|---|
Data-Generator-Role |
Data-Generator | write to the raw bucket |
Data-Processor-Role |
Data-Processor and Sales-Report-App | read from the raw bucket and write to the consumer bucket |
Create each role as follows:
- Open the IAM console, choose Roles, and then choose Create role.
- For Trusted entity type, choose AWS service. For Use case, choose Lambda, and then choose Next.
- Select the AWSLambdaBasicExecutionRole managed policy, which lets the function write logs to CloudWatch, and then choose Next.
- Enter the role name from the table and choose Create role.
- Open the new role. On the Permissions tab, choose Add permissions, and then choose Create inline policy.
- Switch to the JSON editor, paste the matching policy below, and choose Next. Enter a policy name and choose Create policy.
If you chose different bucket names, replace them in the policies.
Data-Generator-Role
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::raw-bucket-demo-313/*"
}
]
}
Data-Processor-Role
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::raw-bucket-demo-313/*"
},
{
"Effect": "Allow",
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::consumer-bucket-demo-313/*"
}
]
}
Create the Data-Processor Lambda function
This Lambda function reads a CSV from an S3 bucket, groups the rows by one or more columns, applies one aggregate (sum, mean, count, and so on) to another column, and writes the result to an output bucket as a new CSV.
To create the function:
- In the Lambda console, choose Create function and keep Author from scratch selected.
- For Function name, enter
Data-Processor. - For Runtime, choose Python 3.12.
- Expand Change default execution role, choose Use an existing role, and select
Data-Processor-Role. - Choose Create function.
- On the Code tab, replace the default code in
lambda_function.pywith the Data-Processor code below, and then choose Deploy.
The Lambda code:
import logging
import os
from urllib.parse import unquote_plus
import boto3
import pandas as pd
logger = logging.getLogger(__name__)
logger.setLevel(os.environ.get("LOG_LEVEL", "INFO").upper())
s3 = boto3.client("s3")
ALLOWED_AGGS = {"sum", "mean", "median", "min", "max", "count", "nunique", "std"}
REQUIRED = ("input_bucket", "input_key", "output_bucket", "group_by", "value_column")
DEFAULTS = {
"agg": "sum",
"top_n": None,
"ascending": False,
"output_key": None,
"output_column": None,
}
SETTINGS = (*REQUIRED, *DEFAULTS)
def lambda_handler(event, context):
event = event or {}
s3_files = files_from_s3_event(event)
jobs = [{**event, "input_bucket": bucket, "input_key": key} for bucket, key in s3_files] or [event]
results = [process(job, started_by_s3=bool(s3_files)) for job in jobs]
return results[0] if len(results) == 1 else results
def process(event, started_by_s3=False):
cfg = load_settings(event, started_by_s3)
df = read_csv_from_s3(
cfg["input_bucket"],
cfg["input_key"],
columns=[*cfg["group_by"], cfg["value_column"]],
)
result = aggregate(
df,
group_by=cfg["group_by"],
value_column=cfg["value_column"],
agg=cfg["agg"],
output_column=cfg["output_column"],
top_n=cfg["top_n"],
ascending=cfg["ascending"],
)
write_csv_to_s3(result, cfg["output_bucket"], cfg["output_key"])
summary = {
"input": f"s3://{cfg['input_bucket']}/{cfg['input_key']}",
"output": f"s3://{cfg['output_bucket']}/{cfg['output_key']}",
"rows_read": len(df),
"rows_written": len(result),
}
logger.info("Finished: %s", summary)
logger.debug("First rows:\n%s", result.head(15).to_string(index=False))
return summary
def files_from_s3_event(event):
files = [
(record["s3"]["bucket"]["name"], unquote_plus(record["s3"]["object"]["key"]))
for record in event.get("Records", [])
if "s3" in record
]
if event.get("source") == "aws.s3" and event.get("detail-type") == "Object Created":
detail = event["detail"]
files.append((detail["bucket"]["name"], detail["object"]["key"]))
return files
def load_settings(event, started_by_s3=False):
cfg = {name: _setting(event, name, DEFAULTS.get(name)) for name in SETTINGS}
missing = [name for name in REQUIRED if not cfg[name]]
if missing:
raise ValueError(
f"Missing setting(s): {', '.join(missing)} (set them in the event or as environment variables)"
)
cfg["group_by"] = _as_list(cfg["group_by"])
cfg["ascending"] = _as_bool(cfg["ascending"])
if cfg["agg"] not in ALLOWED_AGGS:
raise ValueError(f"agg must be one of {sorted(ALLOWED_AGGS)}, got {cfg['agg']!r}")
if cfg["top_n"] is not None:
cfg["top_n"] = int(cfg["top_n"])
if cfg["top_n"] < 1:
raise ValueError("top_n must be a positive integer or null")
if not cfg["output_key"]:
root, ext = os.path.splitext(cfg["input_key"])
cfg["output_key"] = f"{root}_aggregated{ext or '.csv'}"
if not cfg["output_column"]:
cfg["output_column"] = f"{cfg['agg']}_{cfg['value_column']}"
if (cfg["input_bucket"], cfg["input_key"]) == (cfg["output_bucket"], cfg["output_key"]):
raise ValueError("Output location equals the input file; refusing to overwrite it")
if started_by_s3 and cfg["input_bucket"] == cfg["output_bucket"]:
raise ValueError(
"Started by S3 with the same input and output bucket: the output file would start this function again"
)
return cfg
def _setting(event, name, default=None):
if event.get(name, "") != "":
return event[name]
return os.environ.get(name) or default
def _as_list(value):
if isinstance(value, str):
return [part.strip() for part in value.split(",") if part.strip()]
return list(value)
def _as_bool(value):
if isinstance(value, str):
return value.strip().lower() in {"1", "true", "yes"}
return bool(value)
def read_csv_from_s3(bucket, key, columns):
body = s3.get_object(Bucket=bucket, Key=key)["Body"]
return pd.read_csv(body, usecols=columns)
def aggregate(df, group_by, value_column, agg, output_column, top_n=None, ascending=False):
result = (
df.groupby(group_by)[value_column]
.agg(agg)
.rename(output_column)
.reset_index()
.sort_values(output_column, ascending=ascending, kind="stable")
)
return result if top_n is None else result.head(top_n)
def write_csv_to_s3(df, bucket, key):
s3.put_object(
Bucket=bucket,
Key=key,
Body=df.to_csv(index=False).encode("utf-8"),
ContentType="text/csv",
)
Add the required Lambda layer
The Data-Processor function uses Pandas to read, aggregate, and write CSV files.
The standard Python 3.12 Lambda runtime does not include Pandas. Therefore, you must add a compatible Lambda layer.
- Scroll down to the Layers section.
- Choose Add a layer.
- Choose AWS layers.
- Select AWSSDKPandas-Python312.
- Choose the latest compatible version.
- Choose Add.
This layer provides Pandas and its associated dependencies, including NumPy.
Set the environment variables
- On the Configuration tab, choose Environment variables, and then choose Edit.
- Choose Add environment variable and add each of these:
- Key
output_bucket, valueconsumer-bucket-demo-313 - Key
group_by, valuecategory - Key
value_column, valueamount
- Key
- Choose Save. A banner confirms that the function was updated.
Because agg isn't set, the function uses its default, sum, and names the output after the input file: generated_data_aggregated.csv.
Configure general settings
- Open the Configuration tab and choose General configuration.
- Choose Edit.
- Set Memory to 512 MB.
- Set Timeout to 1 min 0 sec.
- Choose Save.
Create an S3 event notification for the raw zone bucket
On the raw zone bucket page, click the Properties tab and scroll down to Event notifications.
- In the Event notifications section, click Create event notification.
- For Event name, type: Process Event
- In the Event types section, for Object creation, choose Put.
- Scroll down to Destination.
- For Destination, choose Lambda function, and then select Data-Processor as the function.
- Choose Save changes.
Amazon S3 can also send events to Amazon EventBridge whenever certain events happen in a bucket. Unlike other destinations, you don't select event types: all events are delivered. To turn this on for the raw bucket:
- In the Event notifications section, find Amazon EventBridge and choose Edit.
- For Send notifications to Amazon EventBridge for all events in this bucket, choose On.
- Choose Save changes. A banner confirms that the event notifications were edited.
The raw bucket now sends every new object to two places: directly to Data-Processor through the S3 event notification, and to Amazon EventBridge
Create the Data-Generator Lambda function
To create the function:
- In the Lambda console, choose Create function and keep Author from scratch selected.
- For Function name, enter
Data-Generator. - For Runtime, choose Python 3.12.
- Expand Change default execution role, choose Use an existing role, and select
Data-Generator-Role. - Choose Create function.
- On the Code tab, replace the default code in
lambda_function.pywith the Data-Generator code below, and then choose Deploy.
The Lambda code:
import logging
import os
import boto3
import numpy as np
import pandas as pd
logger = logging.getLogger(__name__)
logger.setLevel(os.environ.get("LOG_LEVEL", "INFO").upper())
s3 = boto3.client("s3")
DEFAULT_COLUMNS = {
"id": {"type": "sequence"},
"category": {"type": "int", "min": 1, "max": 10},
"region": {"type": "choice", "values": ["north", "south", "east", "west"]},
"amount": {"type": "float", "min": 1, "max": 500, "decimals": 2},
}
DEFAULTS = {
"output_key": "generated_data.csv",
"rows": 1000,
"seed": None,
"columns": DEFAULT_COLUMNS,
}
def lambda_handler(event, context):
cfg = load_settings(event)
df = generate_data(cfg["columns"], cfg["rows"], cfg["seed"])
write_csv_to_s3(df, cfg["output_bucket"], cfg["output_key"])
summary = {
"output": f"s3://{cfg['output_bucket']}/{cfg['output_key']}",
"rows": len(df),
"columns": list(df.columns),
}
logger.info("Finished: %s", summary)
logger.debug("First rows:\n%s", df.head().to_string(index=False))
return summary
def load_settings(event):
cfg = {**DEFAULTS, **(event or {})}
cfg["output_bucket"] = cfg.get("output_bucket") or os.environ.get("output_bucket")
if not cfg["output_bucket"]:
raise ValueError("Missing setting: output_bucket (it can also be an environment variable)")
cfg["rows"] = int(cfg["rows"])
if cfg["rows"] < 1:
raise ValueError("rows must be a positive integer")
if cfg["seed"] is not None:
cfg["seed"] = int(cfg["seed"])
if not cfg["columns"]:
raise ValueError("columns must define at least one column")
return cfg
def generate_data(columns, rows, seed=None):
rng = np.random.default_rng(seed)
fake = _make_faker(seed) if any(spec.get("type") == "faker" for spec in columns.values()) else None
data = {}
for name, spec in columns.items():
try:
data[name] = make_column(spec, rows, rng, fake)
except KeyError as missing:
raise ValueError(f"Column '{name}': spec is missing {missing}") from None
return pd.DataFrame(data)
def _make_faker(seed):
try:
from faker import Faker
except ImportError:
raise ValueError("A faker column needs the faker package. Add a layer that contains it.") from None
fake = Faker()
fake.seed_instance(seed)
return fake
def make_column(spec, rows, rng, fake):
kind = spec["type"]
if kind == "sequence":
return spec.get("start", 1) + spec.get("step", 1) * np.arange(rows)
if kind == "int":
return rng.integers(spec["min"], spec["max"], size=rows, endpoint=True)
if kind == "float":
return rng.uniform(spec["min"], spec["max"], size=rows).round(spec.get("decimals", 2))
if kind == "choice":
values = spec["values"]
picks = rng.choice(len(values), size=rows, p=spec.get("weights"))
return [values[i] for i in picks]
if kind == "faker":
method = getattr(fake, spec["method"], None)
if not callable(method):
raise ValueError(f"Unknown Faker method {spec['method']!r}")
args = spec.get("args", {})
return [method(**args) for _ in range(rows)]
raise ValueError(f"Unknown column type {kind!r}; use sequence, int, float, choice or faker")
def write_csv_to_s3(df, bucket, key):
s3.put_object(
Bucket=bucket,
Key=key,
Body=df.to_csv(index=False).encode("utf-8"),
ContentType="text/csv",
)
Add the required Lambda layer
The Data-Generator function imports Pandas and NumPy.
These libraries are not included in the standard Python 3.12 Lambda runtime.
To provide these dependencies:
- Scroll down to the Layers section.
- Choose Add a layer.
- Choose AWS layers.
- Select AWSSDKPandas-Python312.
- Choose the latest compatible version.
- Choose Add.
Configure general settings
- Open the Configuration tab and choose General configuration.
- Choose Edit.
- Set Memory to 512 MB.
- Set Timeout to 1 min 0 sec.
- Choose Save.
Configure and test the Data-Generator function
- On the Configuration tab, choose Environment variables, and then choose Edit.
- Choose Add environment variable. For Key, enter
output_bucket. For Value, enterraw-bucket-demo-313. - Choose Save. A banner confirms that the function was updated.
- Choose the Test tab. For Event name, enter
TestEvent. - In the Event JSON box, replace the existing JSON with the following:
{
"rows": 1000,
"output_key": "generated_data.csv",
"seed": 42
}
- Choose Test.
Expected output:
{
"output": "s3://raw-bucket-demo-313/generated_data.csv",
"rows": 1000,
"columns": [
"id",
"category",
"region",
"amount"
]
}
The upload to the raw bucket also starts Data-Processor through the S3 event notification. We check its result in the next section.
Review the processing layer
In this section, we review the processing layer. After the raw data is ingested, this layer transforms it and sends the result to the consumption zone bucket. In our data lake, the Data-Processor function does this work.
Check the Data-Processor run
The Data-Generator test already uploaded generated_data.csv to the raw bucket, and the S3 event notification invoked Data-Processor.
- In the Lambda console, open the Data-Processor function.
- In the Function overview section, confirm that S3 is listed as the trigger.
- Choose the Monitor tab, choose View CloudWatch logs, and open the latest log stream.
- Confirm that the log contains a
Finished:line that shows'rows_read': 1000and'rows_written': 10.
Review the processed data in the consumer bucket
- In the search bar at the top of the AWS Management Console, search for and choose S3.
- In the General purpose buckets section, choose
consumer-bucket-demo-313. - On the Objects tab, find the
generated_data_aggregated.csvfile. If it isn't listed, choose the refresh icon.
- Select the check box next to the file and choose Download.
- Open the downloaded file in an editor of your choice.
The processing layer works: generated_data_aggregated.csv has one row per category (10 rows) with its total sum_amount, highest first, and the raw file stays untouched in the raw bucket.
Understanding the processing output
The Data-Processor function transforms the original raw CSV file into an aggregated dataset.
The original dataset contains four columns:
- id
- category
- region
- amount
The processed dataset contains:
- category
- sum_amount
The aggregation groups all sales records by product category and calculates the total sales amount for each category. The resulting dataset is sorted in descending order, meaning the categories with the highest total sales appear first. The raw data remains stored in the raw bucket, while the transformed data is stored in the consumer bucket. This separation ensures that the original source data is preserved and can be processed again using different transformation logic when required.
Build the consumption layer
In this section, we build the consumption layer: a reporting application that turns the raw data into something business users can read. The Sales-Report-App function reads the original CSV from the raw bucket, groups the orders by region and category, and writes the total amount for each combination to the consumer bucket. It reads the raw file instead of the Data-Processor output because the processed file no longer has a region column. Analysts can use the report to decide which product categories to promote in each region.
Create the Sales-Report-App function
Sales-Report-App uses the same code as Data-Processor. Only its settings and its trigger are different.
- Choose Create function. For Function name, enter
Sales-Report-App. For Runtime, choose Python 3.12. - Expand Change default execution role, choose Use an existing role, and select
Data-Processor-Role. Choose Create function. - On the Code tab, replace the code in
lambda_function.pywith the Data-Processor code, and then choose Deploy. - In the Layers section, choose Add a layer, choose AWS layers, select
AWSSDKPandas-Python312, choose the latest version, and then choose Add. - On the Configuration tab, choose General configuration, and then choose Edit. Set Memory to
512MB and Timeout to1min0sec, and then choose Save.
Create the Amazon EventBridge rule
- In the search bar at the top of the AWS Management Console, search for and choose Amazon EventBridge.
- In the left navigation pane, choose Event buses, choose the default event bus, choose Rules, and then choose Create rule.
In the Define rule detail step, in the Rule detail section, configure the following:
- For Name, enter SalesReportApp.
- For Event bus, keep the default event bus.
- For Rule type, choose Rule with an event pattern.
- Choose Next.
In the Build event pattern step, configure the following:
- For Event source, choose AWS events or EventBridge partner events if it is not already selected.
- Scroll down to the Event pattern section, and for Event source, keep AWS services.
- For AWS service, use the dropdown menu and choose S3 (Simple Storage Service).
- For Event type, use the dropdown menu and choose Amazon S3 Event Notification.
- In the Event pattern text box, review that the detail-type JSON structure has multiple event types that should trigger the rule for any bucket.
- For Event Type Specification 1, choose Specific event(s).
- For Specific event(s), on the dropdown menu, choose Object Created.
- For Event Type Specification 2, choose Specific bucket(s) by name.
- For Specific bucket(s) by name, enter raw-bucket-demo-313.
- Choose Next.
- In the Select target(s) step, in the Target 1 section, configure the following:
- For Target types, choose AWS service.
- For Select a target, use the dropdown menu, and choose Lambda function.
- For Function, use the dropdown menu, and choose Sales-Report-App.
Under Permissions, clear (uncheck) Use execution role (recommended). EventBridge invokes a Lambda function through a resource-based policy on the function, which the console adds for you, so this target needs no separate role.
Choose Next.
In the Configure tags - optional step, choose Next.
In the Review and create step, review the Event pattern and confirm it looks similar to the following example:
- Choose Create rule. The rule appears in the list with the status Enabled.
Configure the Sales-Report-App function
- At the top of the AWS Management Console, in the search bar, search for and choose Lambda.
- In the Functions list, choose the Sales-Report-App function.
- In the Function overview section, review to ensure that EventBridge is listed as the function trigger.
- Scroll down to the Code tab.
- In the lambda_function.py window, review the Sales-Report-App function code.
Set the Sales-Report-App environment variables
Before testing the reporting application, set its environment variables:
- On the Configuration tab, choose Environment variables, and then choose Edit.
- Choose Add environment variable and add each of these:
- Key
output_bucket, valueconsumer-bucket-demo-313 - Key
group_by, valueregion,category - Key
value_column, valueamount - Key
output_key, valuesales_report.csv
- Key
- Choose Save. A banner confirms that the function was updated.
Run the pipeline again
- On the top breadcrumb menu, choose Functions.
- Choose the Data-Generator function and open the Test tab.
- Choose Test to run
TestEventagain. This upload now starts both Data-Processor (through the S3 event notification) and Sales-Report-App (through the EventBridge rule).
Review data in the consumption zone S3 bucket
At the top of the AWS Management Console, in the search bar, search for and choose S3.
In the General purpose buckets section, choose the consumption zone S3 bucket (bucket named consumer-bucket-demo-313).
On the Objects tab, find the sales_report.csv file. If it isn't listed, choose the refresh icon, and then select the check box next to the file.
Choose Download, and save the sales_report.csv file to your device.
On your device, open the sales_report.csv file using an editor of your choice.
The consumption layer works: sales_report.csv has one row for each region and category combination (4 regions × 10 categories = 40 rows) with its total sum_amount.
The consumption layer provides datasets that are ready to be accessed by reporting applications and business users.
In this project, Sales-Report-App generates a report containing the total sales amount for each combination of region and product category.
This allows business analysts to identify which product categories are performing well in particular regions.
The resulting CSV file can be downloaded and used with spreadsheet applications or other analytical tools.
In a more advanced architecture, the consumption layer could also provide data to services such as Amazon Athena, Quick Sight, or other business intelligence applications.
How the two event paths work
In this architecture, two different event-driven mechanisms are used to process the same raw data.
S3 event notification
The S3 event notification directly invokes the Data-Processor Lambda function whenever an object is uploaded to the raw bucket using the configured Put event type.
The Data-Processor function receives the bucket name and object key from the S3 event. It then reads the CSV file, aggregates the data, and writes the resulting CSV file to the consumer bucket.
Amazon EventBridge
Amazon S3 also sends events to EventBridge because event delivery was enabled on the raw bucket.
The SalesReportApp rule listens for Object Created events associated with raw-bucket-demo-313.
When a matching event occurs, EventBridge invokes the Sales-Report-App Lambda function.
The Sales-Report-App function independently reads the original CSV file from the raw bucket, groups the data by region and category, and writes its report to the consumer bucket.
This design demonstrates how multiple serverless applications can respond independently to the same data ingestion event.
Why two mechanisms?
We use both on purpose. An S3 bucket can't send the same event type to two destinations unless their prefix or suffix filters differ, because the filters would overlap. EventBridge has no such limit: one event can match several rules, each with its own target. If you only need one consumer, the S3 event notification alone is enough.
Monitoring and troubleshooting
Monitoring is an important part of maintaining a serverless data lake. Amazon CloudWatch collects the logs of every Lambda execution, so you can find problems during ingestion and processing.
To review the logs of a function:
- Open the AWS Lambda console and select the function.
- Choose the Monitor tab.
- Choose View CloudWatch logs.
- Open the latest log stream and review the execution logs.
CloudWatch logs can help you identify:
- Lambda execution failures
- Missing environment variables (for example,
Missing setting(s): output_bucket, group_by, value_column) - Incorrect S3 bucket names
- Missing Python dependencies, such as a function without the Pandas layer
- IAM permission errors
- CSV parsing errors
- Timeout issues
Conclusion
In this project, we build a basic serverless data lake on AWS. Data-Generator writes raw sales data to the raw bucket. The same upload starts two independent applications: Data-Processor, through an S3 event notification, and Sales-Report-App, through an EventBridge rule. Both write their results to the consumer bucket, and the raw data stays untouched. CloudWatch logs help us monitor and troubleshoot every step. To extend the project, catalog the data with AWS Glue, query it with Amazon Athena, and build dashboards on top.














Top comments (0)