DEV Community

Sushanta Paudel
Sushanta Paudel

Posted on Edited on

Building a Basic Serverless Data Lake on AWS with S3, Lambda, and EventBridge

A data lake is an architectural approach we can use to centralize all our data. We can then access, transform, and analyze that data with the use of AWS services and tools.

Here is what we build:

Data lakes provide several benefits:

  • Scalability: you can store large amounts and many types of data in one place.
  • Discoverability: you can find out what data resides in the lake through crawling, cataloging, and indexing.
  • Security: you can protect all your data assets with encryption and access controls.
  • Shareability: data scientists, data developers, and business analysts can use the same data with the analytics and machine learning (ML) tools of their choice.
  • Agility: serverless services remove the undifferentiated heavy lifting of managing infrastructure, so you can focus on the data instead of servers.

In this project we build a serverless data lake. The serverless data lake uses storage with Amazon S3. The serverless data lake also uses event‑driven processing with AWS Lambda and Amazon EventBridge. The next natural steps are cataloging the serverless data lake with AWS Glue. Querying the serverless data lake, with Amazon Athena.

  1. Data-Generator (Lambda) creates a CSV of random sales orders and writes it to the raw bucket.
  2. Amazon S3 notifies Data-Processor (Lambda), which totals the orders by category and writes the result to the consumer bucket.
  3. S3 also sends the same event to Amazon EventBridge. The SalesReportApp rule invokes Sales-Report-App (Lambda), which reads the same raw file, totals the orders by region and category, and writes a report to the consumer bucket.
  4. Amazon CloudWatch collects the logs of all three functions.

Create the S3 buckets

We use two buckets, one for each zone of the data lake. S3 bucket names are globally unique, so choose your own suffix instead of 313 and use your names everywhere this article mentions a bucket (the IAM policies, the environment variables, and the EventBridge rule). Create all resources in the same AWS Region.

  • Raw bucket (raw-bucket-demo-313): the raw zone. It stores data in its original format as ingested from the sources, which can be structured, semi-structured, or unstructured: databases, backups, archives, images, and files. In this project, a Lambda function generates the raw data: random sales orders, each with an ID, a product category (a number from 1 to 10), a region, and an amount.
  • Consumer bucket (consumer-bucket-demo-313): the consumption zone. It stores the transformed data. Transformations might include aggregating data from different sources or changing the file format.

To create each bucket:

  1. Open the Amazon S3 console and choose Create bucket.
  2. For Bucket name, enter the name.
  3. Keep the other settings as they are and choose Create bucket.

Confirm the S3 Bucket Security settings

New S3 buckets already block public access and encrypt new objects with SSE-S3 by default. Confirm that both settings are in place on both buckets:

  • Open the S3 bucket.
  • Navigate to the Permissions tab.
  • Under Block public access, confirm that all public access is blocked.
  • Under Bucket policy, confirm that no unnecessary public access is granted.
  • Navigate to the Properties tab.
  • Under Default encryption, confirm that the encryption type is Server-side encryption with Amazon S3 managed keys (SSE-S3)

Create the IAM roles

Each Lambda function needs an execution role that lets it write logs and access the buckets. We create two roles that follow the principle of least privilege:

Role Used by Needs to
Data-Generator-Role Data-Generator write to the raw bucket
Data-Processor-Role Data-Processor and Sales-Report-App read from the raw bucket and write to the consumer bucket

Create each role as follows:

  1. Open the IAM console, choose Roles, and then choose Create role.
  2. For Trusted entity type, choose AWS service. For Use case, choose Lambda, and then choose Next.
  3. Select the AWSLambdaBasicExecutionRole managed policy, which lets the function write logs to CloudWatch, and then choose Next.
  4. Enter the role name from the table and choose Create role.
  5. Open the new role. On the Permissions tab, choose Add permissions, and then choose Create inline policy.
  6. Switch to the JSON editor, paste the matching policy below, and choose Next. Enter a policy name and choose Create policy.

If you chose different bucket names, replace them in the policies.

Data-Generator-Role

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::raw-bucket-demo-313/*"
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode

Data-Processor-Role

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::raw-bucket-demo-313/*"
    },
    {
      "Effect": "Allow",
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::consumer-bucket-demo-313/*"
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode

Create the Data-Processor Lambda function

This Lambda function reads a CSV from an S3 bucket, groups the rows by one or more columns, applies one aggregate (sum, mean, count, and so on) to another column, and writes the result to an output bucket as a new CSV.

To create the function:

  1. In the Lambda console, choose Create function and keep Author from scratch selected.
  2. For Function name, enter Data-Processor.
  3. For Runtime, choose Python 3.12.
  4. Expand Change default execution role, choose Use an existing role, and select Data-Processor-Role.
  5. Choose Create function.
  6. On the Code tab, replace the default code in lambda_function.py with the Data-Processor code below, and then choose Deploy.

The Lambda code:

import logging
import os
from urllib.parse import unquote_plus

import boto3
import pandas as pd

logger = logging.getLogger(__name__)
logger.setLevel(os.environ.get("LOG_LEVEL", "INFO").upper())

s3 = boto3.client("s3")

ALLOWED_AGGS = {"sum", "mean", "median", "min", "max", "count", "nunique", "std"}
REQUIRED = ("input_bucket", "input_key", "output_bucket", "group_by", "value_column")

DEFAULTS = {
    "agg": "sum",
    "top_n": None,
    "ascending": False,
    "output_key": None,
    "output_column": None,
}

SETTINGS = (*REQUIRED, *DEFAULTS)


def lambda_handler(event, context):
    event = event or {}
    s3_files = files_from_s3_event(event)

    jobs = [{**event, "input_bucket": bucket, "input_key": key} for bucket, key in s3_files] or [event]
    results = [process(job, started_by_s3=bool(s3_files)) for job in jobs]
    return results[0] if len(results) == 1 else results


def process(event, started_by_s3=False):
    cfg = load_settings(event, started_by_s3)

    df = read_csv_from_s3(
        cfg["input_bucket"],
        cfg["input_key"],
        columns=[*cfg["group_by"], cfg["value_column"]],
    )
    result = aggregate(
        df,
        group_by=cfg["group_by"],
        value_column=cfg["value_column"],
        agg=cfg["agg"],
        output_column=cfg["output_column"],
        top_n=cfg["top_n"],
        ascending=cfg["ascending"],
    )
    write_csv_to_s3(result, cfg["output_bucket"], cfg["output_key"])

    summary = {
        "input": f"s3://{cfg['input_bucket']}/{cfg['input_key']}",
        "output": f"s3://{cfg['output_bucket']}/{cfg['output_key']}",
        "rows_read": len(df),
        "rows_written": len(result),
    }
    logger.info("Finished: %s", summary)
    logger.debug("First rows:\n%s", result.head(15).to_string(index=False))
    return summary


def files_from_s3_event(event):
    files = [
        (record["s3"]["bucket"]["name"], unquote_plus(record["s3"]["object"]["key"]))
        for record in event.get("Records", [])
        if "s3" in record
    ]
    if event.get("source") == "aws.s3" and event.get("detail-type") == "Object Created":
        detail = event["detail"]
        files.append((detail["bucket"]["name"], detail["object"]["key"]))
    return files


def load_settings(event, started_by_s3=False):
    cfg = {name: _setting(event, name, DEFAULTS.get(name)) for name in SETTINGS}

    missing = [name for name in REQUIRED if not cfg[name]]
    if missing:
        raise ValueError(
            f"Missing setting(s): {', '.join(missing)} (set them in the event or as environment variables)"
        )

    cfg["group_by"] = _as_list(cfg["group_by"])
    cfg["ascending"] = _as_bool(cfg["ascending"])
    if cfg["agg"] not in ALLOWED_AGGS:
        raise ValueError(f"agg must be one of {sorted(ALLOWED_AGGS)}, got {cfg['agg']!r}")
    if cfg["top_n"] is not None:
        cfg["top_n"] = int(cfg["top_n"])
        if cfg["top_n"] < 1:
            raise ValueError("top_n must be a positive integer or null")

    if not cfg["output_key"]:
        root, ext = os.path.splitext(cfg["input_key"])
        cfg["output_key"] = f"{root}_aggregated{ext or '.csv'}"
    if not cfg["output_column"]:
        cfg["output_column"] = f"{cfg['agg']}_{cfg['value_column']}"

    if (cfg["input_bucket"], cfg["input_key"]) == (cfg["output_bucket"], cfg["output_key"]):
        raise ValueError("Output location equals the input file; refusing to overwrite it")
    if started_by_s3 and cfg["input_bucket"] == cfg["output_bucket"]:
        raise ValueError(
            "Started by S3 with the same input and output bucket: the output file would start this function again"
        )
    return cfg


def _setting(event, name, default=None):
    if event.get(name, "") != "":
        return event[name]
    return os.environ.get(name) or default


def _as_list(value):
    if isinstance(value, str):
        return [part.strip() for part in value.split(",") if part.strip()]
    return list(value)


def _as_bool(value):
    if isinstance(value, str):
        return value.strip().lower() in {"1", "true", "yes"}
    return bool(value)


def read_csv_from_s3(bucket, key, columns):
    body = s3.get_object(Bucket=bucket, Key=key)["Body"]
    return pd.read_csv(body, usecols=columns)


def aggregate(df, group_by, value_column, agg, output_column, top_n=None, ascending=False):
    result = (
        df.groupby(group_by)[value_column]
        .agg(agg)
        .rename(output_column)
        .reset_index()
        .sort_values(output_column, ascending=ascending, kind="stable")
    )
    return result if top_n is None else result.head(top_n)


def write_csv_to_s3(df, bucket, key):
    s3.put_object(
        Bucket=bucket,
        Key=key,
        Body=df.to_csv(index=False).encode("utf-8"),
        ContentType="text/csv",
    )
Enter fullscreen mode Exit fullscreen mode

Add the required Lambda layer

The Data-Processor function uses Pandas to read, aggregate, and write CSV files.

The standard Python 3.12 Lambda runtime does not include Pandas. Therefore, you must add a compatible Lambda layer.

  • Scroll down to the Layers section.
  • Choose Add a layer.
  • Choose AWS layers.
  • Select AWSSDKPandas-Python312.
  • Choose the latest compatible version.
  • Choose Add.

This layer provides Pandas and its associated dependencies, including NumPy.

Set the environment variables

  • On the Configuration tab, choose Environment variables, and then choose Edit.
  • Choose Add environment variable and add each of these:
    • Key output_bucket, value consumer-bucket-demo-313
    • Key group_by, value category
    • Key value_column, value amount
  • Choose Save. A banner confirms that the function was updated.

Data-Processor environment variables

Because agg isn't set, the function uses its default, sum, and names the output after the input file: generated_data_aggregated.csv.

Configure general settings

  • Open the Configuration tab and choose General configuration.
  • Choose Edit.
  • Set Memory to 512 MB.
  • Set Timeout to 1 min 0 sec.
  • Choose Save.

Create an S3 event notification for the raw zone bucket

On the raw zone bucket page, click the Properties tab and scroll down to Event notifications.

  • In the Event notifications section, click Create event notification.
  • For Event name, type: Process Event
  • In the Event types section, for Object creation, choose Put.
  • Scroll down to Destination.
  • For Destination, choose Lambda function, and then select Data-Processor as the function.
  • Choose Save changes.

Amazon S3 can also send events to Amazon EventBridge whenever certain events happen in a bucket. Unlike other destinations, you don't select event types: all events are delivered. To turn this on for the raw bucket:

  1. In the Event notifications section, find Amazon EventBridge and choose Edit.
  2. For Send notifications to Amazon EventBridge for all events in this bucket, choose On.
  3. Choose Save changes. A banner confirms that the event notifications were edited.

The raw bucket now sends every new object to two places: directly to Data-Processor through the S3 event notification, and to Amazon EventBridge

Create the Data-Generator Lambda function

To create the function:

  1. In the Lambda console, choose Create function and keep Author from scratch selected.
  2. For Function name, enter Data-Generator.
  3. For Runtime, choose Python 3.12.
  4. Expand Change default execution role, choose Use an existing role, and select Data-Generator-Role.
  5. Choose Create function.
  6. On the Code tab, replace the default code in lambda_function.py with the Data-Generator code below, and then choose Deploy.

The Lambda code:

import logging
import os

import boto3
import numpy as np
import pandas as pd

logger = logging.getLogger(__name__)
logger.setLevel(os.environ.get("LOG_LEVEL", "INFO").upper())

s3 = boto3.client("s3")

DEFAULT_COLUMNS = {
    "id": {"type": "sequence"},
    "category": {"type": "int", "min": 1, "max": 10},
    "region": {"type": "choice", "values": ["north", "south", "east", "west"]},
    "amount": {"type": "float", "min": 1, "max": 500, "decimals": 2},
}

DEFAULTS = {
    "output_key": "generated_data.csv",
    "rows": 1000,
    "seed": None,
    "columns": DEFAULT_COLUMNS,
}


def lambda_handler(event, context):
    cfg = load_settings(event)

    df = generate_data(cfg["columns"], cfg["rows"], cfg["seed"])
    write_csv_to_s3(df, cfg["output_bucket"], cfg["output_key"])

    summary = {
        "output": f"s3://{cfg['output_bucket']}/{cfg['output_key']}",
        "rows": len(df),
        "columns": list(df.columns),
    }
    logger.info("Finished: %s", summary)
    logger.debug("First rows:\n%s", df.head().to_string(index=False))
    return summary


def load_settings(event):
    cfg = {**DEFAULTS, **(event or {})}

    cfg["output_bucket"] = cfg.get("output_bucket") or os.environ.get("output_bucket")
    if not cfg["output_bucket"]:
        raise ValueError("Missing setting: output_bucket (it can also be an environment variable)")

    cfg["rows"] = int(cfg["rows"])
    if cfg["rows"] < 1:
        raise ValueError("rows must be a positive integer")
    if cfg["seed"] is not None:
        cfg["seed"] = int(cfg["seed"])
    if not cfg["columns"]:
        raise ValueError("columns must define at least one column")
    return cfg


def generate_data(columns, rows, seed=None):
    rng = np.random.default_rng(seed)
    fake = _make_faker(seed) if any(spec.get("type") == "faker" for spec in columns.values()) else None

    data = {}
    for name, spec in columns.items():
        try:
            data[name] = make_column(spec, rows, rng, fake)
        except KeyError as missing:
            raise ValueError(f"Column '{name}': spec is missing {missing}") from None
    return pd.DataFrame(data)


def _make_faker(seed):
    try:
        from faker import Faker
    except ImportError:
        raise ValueError("A faker column needs the faker package. Add a layer that contains it.") from None
    fake = Faker()
    fake.seed_instance(seed)
    return fake


def make_column(spec, rows, rng, fake):
    kind = spec["type"]
    if kind == "sequence":
        return spec.get("start", 1) + spec.get("step", 1) * np.arange(rows)
    if kind == "int":
        return rng.integers(spec["min"], spec["max"], size=rows, endpoint=True)
    if kind == "float":
        return rng.uniform(spec["min"], spec["max"], size=rows).round(spec.get("decimals", 2))
    if kind == "choice":
        values = spec["values"]
        picks = rng.choice(len(values), size=rows, p=spec.get("weights"))
        return [values[i] for i in picks]
    if kind == "faker":
        method = getattr(fake, spec["method"], None)
        if not callable(method):
            raise ValueError(f"Unknown Faker method {spec['method']!r}")
        args = spec.get("args", {})
        return [method(**args) for _ in range(rows)]
    raise ValueError(f"Unknown column type {kind!r}; use sequence, int, float, choice or faker")


def write_csv_to_s3(df, bucket, key):
    s3.put_object(
        Bucket=bucket,
        Key=key,
        Body=df.to_csv(index=False).encode("utf-8"),
        ContentType="text/csv",
    )
Enter fullscreen mode Exit fullscreen mode

Add the required Lambda layer

The Data-Generator function imports Pandas and NumPy.

These libraries are not included in the standard Python 3.12 Lambda runtime.

To provide these dependencies:

  • Scroll down to the Layers section.
  • Choose Add a layer.
  • Choose AWS layers.
  • Select AWSSDKPandas-Python312.
  • Choose the latest compatible version.
  • Choose Add.

Configure general settings

  • Open the Configuration tab and choose General configuration.
  • Choose Edit.
  • Set Memory to 512 MB.
  • Set Timeout to 1 min 0 sec.
  • Choose Save.

Configure and test the Data-Generator function

  1. On the Configuration tab, choose Environment variables, and then choose Edit.
  2. Choose Add environment variable. For Key, enter output_bucket. For Value, enter raw-bucket-demo-313.
  3. Choose Save. A banner confirms that the function was updated.
  4. Choose the Test tab. For Event name, enter TestEvent.
  5. In the Event JSON box, replace the existing JSON with the following:
{
  "rows": 1000,
  "output_key": "generated_data.csv",
  "seed": 42
}
Enter fullscreen mode Exit fullscreen mode
  1. Choose Test.

TestEvent with its Event JSON on the Data-Generator Test tab

Expected output:

{
  "output": "s3://raw-bucket-demo-313/generated_data.csv",
  "rows": 1000,
  "columns": [
    "id",
    "category",
    "region",
    "amount"
  ]
}
Enter fullscreen mode Exit fullscreen mode

The upload to the raw bucket also starts Data-Processor through the S3 event notification. We check its result in the next section.

Review the processing layer

In this section, we review the processing layer. After the raw data is ingested, this layer transforms it and sends the result to the consumption zone bucket. In our data lake, the Data-Processor function does this work.

Check the Data-Processor run

The Data-Generator test already uploaded generated_data.csv to the raw bucket, and the S3 event notification invoked Data-Processor.

  1. In the Lambda console, open the Data-Processor function.
  2. In the Function overview section, confirm that S3 is listed as the trigger.
  3. Choose the Monitor tab, choose View CloudWatch logs, and open the latest log stream.
  4. Confirm that the log contains a Finished: line that shows 'rows_read': 1000 and 'rows_written': 10.

Review the processed data in the consumer bucket

  1. In the search bar at the top of the AWS Management Console, search for and choose S3.
  2. In the General purpose buckets section, choose consumer-bucket-demo-313.
  3. On the Objects tab, find the generated_data_aggregated.csv file. If it isn't listed, choose the refresh icon.

generated_data_aggregated.csv in the consumer bucket

  1. Select the check box next to the file and choose Download.
  2. Open the downloaded file in an editor of your choice.

The processing layer works: generated_data_aggregated.csv has one row per category (10 rows) with its total sum_amount, highest first, and the raw file stays untouched in the raw bucket.

Contents of generated_data_aggregated.csv

Understanding the processing output

The Data-Processor function transforms the original raw CSV file into an aggregated dataset.

The original dataset contains four columns:

  • id
  • category
  • region
  • amount

The processed dataset contains:

  • category
  • sum_amount

The aggregation groups all sales records by product category and calculates the total sales amount for each category. The resulting dataset is sorted in descending order, meaning the categories with the highest total sales appear first. The raw data remains stored in the raw bucket, while the transformed data is stored in the consumer bucket. This separation ensures that the original source data is preserved and can be processed again using different transformation logic when required.

Build the consumption layer

In this section, we build the consumption layer: a reporting application that turns the raw data into something business users can read. The Sales-Report-App function reads the original CSV from the raw bucket, groups the orders by region and category, and writes the total amount for each combination to the consumer bucket. It reads the raw file instead of the Data-Processor output because the processed file no longer has a region column. Analysts can use the report to decide which product categories to promote in each region.

Create the Sales-Report-App function

Sales-Report-App uses the same code as Data-Processor. Only its settings and its trigger are different.

  1. Choose Create function. For Function name, enter Sales-Report-App. For Runtime, choose Python 3.12.
  2. Expand Change default execution role, choose Use an existing role, and select Data-Processor-Role. Choose Create function.
  3. On the Code tab, replace the code in lambda_function.py with the Data-Processor code, and then choose Deploy.
  4. In the Layers section, choose Add a layer, choose AWS layers, select AWSSDKPandas-Python312, choose the latest version, and then choose Add.
  5. On the Configuration tab, choose General configuration, and then choose Edit. Set Memory to 512 MB and Timeout to 1 min 0 sec, and then choose Save.

Create the Amazon EventBridge rule

  1. In the search bar at the top of the AWS Management Console, search for and choose Amazon EventBridge.
  2. In the left navigation pane, choose Event buses, choose the default event bus, choose Rules, and then choose Create rule.

In the Define rule detail step, in the Rule detail section, configure the following:

  • For Name, enter SalesReportApp.
  • For Event bus, keep the default event bus.
  • For Rule type, choose Rule with an event pattern.
  • Choose Next.

In the Build event pattern step, configure the following:

  • For Event source, choose AWS events or EventBridge partner events if it is not already selected.
  • Scroll down to the Event pattern section, and for Event source, keep AWS services.
  • For AWS service, use the dropdown menu and choose S3 (Simple Storage Service).
  • For Event type, use the dropdown menu and choose Amazon S3 Event Notification.
  • In the Event pattern text box, review that the detail-type JSON structure has multiple event types that should trigger the rule for any bucket.

  • For Event Type Specification 1, choose Specific event(s).
  • For Specific event(s), on the dropdown menu, choose Object Created.
  • For Event Type Specification 2, choose Specific bucket(s) by name.
  • For Specific bucket(s) by name, enter raw-bucket-demo-313.

  • Choose Next.
  • In the Select target(s) step, in the Target 1 section, configure the following:
  • For Target types, choose AWS service.
  • For Select a target, use the dropdown menu, and choose Lambda function.
  • For Function, use the dropdown menu, and choose Sales-Report-App.
  • Under Permissions, clear (uncheck) Use execution role (recommended). EventBridge invokes a Lambda function through a resource-based policy on the function, which the console adds for you, so this target needs no separate role.

  • Choose Next.

  • In the Configure tags - optional step, choose Next.

  • In the Review and create step, review the Event pattern and confirm it looks similar to the following example:

  • Choose Create rule. The rule appears in the list with the status Enabled.

Configure the Sales-Report-App function

  • At the top of the AWS Management Console, in the search bar, search for and choose Lambda.
  • In the Functions list, choose the Sales-Report-App function.
  • In the Function overview section, review to ensure that EventBridge is listed as the function trigger.
  • Scroll down to the Code tab.
  • In the lambda_function.py window, review the Sales-Report-App function code.

Set the Sales-Report-App environment variables

Before testing the reporting application, set its environment variables:

  • On the Configuration tab, choose Environment variables, and then choose Edit.
  • Choose Add environment variable and add each of these:
    • Key output_bucket, value consumer-bucket-demo-313
    • Key group_by, value region,category
    • Key value_column, value amount
    • Key output_key, value sales_report.csv
  • Choose Save. A banner confirms that the function was updated.

Sales-Report-App environment variables

Run the pipeline again

  • On the top breadcrumb menu, choose Functions.
  • Choose the Data-Generator function and open the Test tab.
  • Choose Test to run TestEvent again. This upload now starts both Data-Processor (through the S3 event notification) and Sales-Report-App (through the EventBridge rule).

Review data in the consumption zone S3 bucket

At the top of the AWS Management Console, in the search bar, search for and choose S3.

In the General purpose buckets section, choose the consumption zone S3 bucket (bucket named consumer-bucket-demo-313).

On the Objects tab, find the sales_report.csv file. If it isn't listed, choose the refresh icon, and then select the check box next to the file.

Choose Download, and save the sales_report.csv file to your device.

On your device, open the sales_report.csv file using an editor of your choice.

The consumption layer works: sales_report.csv has one row for each region and category combination (4 regions × 10 categories = 40 rows) with its total sum_amount.

The consumption layer provides datasets that are ready to be accessed by reporting applications and business users.

In this project, Sales-Report-App generates a report containing the total sales amount for each combination of region and product category.

This allows business analysts to identify which product categories are performing well in particular regions.

The resulting CSV file can be downloaded and used with spreadsheet applications or other analytical tools.

In a more advanced architecture, the consumption layer could also provide data to services such as Amazon Athena, Quick Sight, or other business intelligence applications.

How the two event paths work

In this architecture, two different event-driven mechanisms are used to process the same raw data.

S3 event notification

The S3 event notification directly invokes the Data-Processor Lambda function whenever an object is uploaded to the raw bucket using the configured Put event type.

The Data-Processor function receives the bucket name and object key from the S3 event. It then reads the CSV file, aggregates the data, and writes the resulting CSV file to the consumer bucket.

Amazon EventBridge

Amazon S3 also sends events to EventBridge because event delivery was enabled on the raw bucket.

The SalesReportApp rule listens for Object Created events associated with raw-bucket-demo-313.

When a matching event occurs, EventBridge invokes the Sales-Report-App Lambda function.

The Sales-Report-App function independently reads the original CSV file from the raw bucket, groups the data by region and category, and writes its report to the consumer bucket.

This design demonstrates how multiple serverless applications can respond independently to the same data ingestion event.

Why two mechanisms?

We use both on purpose. An S3 bucket can't send the same event type to two destinations unless their prefix or suffix filters differ, because the filters would overlap. EventBridge has no such limit: one event can match several rules, each with its own target. If you only need one consumer, the S3 event notification alone is enough.

Monitoring and troubleshooting

Monitoring is an important part of maintaining a serverless data lake. Amazon CloudWatch collects the logs of every Lambda execution, so you can find problems during ingestion and processing.

To review the logs of a function:

  1. Open the AWS Lambda console and select the function.
  2. Choose the Monitor tab.
  3. Choose View CloudWatch logs.
  4. Open the latest log stream and review the execution logs.

CloudWatch logs can help you identify:

  • Lambda execution failures
  • Missing environment variables (for example, Missing setting(s): output_bucket, group_by, value_column)
  • Incorrect S3 bucket names
  • Missing Python dependencies, such as a function without the Pandas layer
  • IAM permission errors
  • CSV parsing errors
  • Timeout issues

Conclusion

In this project, we build a basic serverless data lake on AWS. Data-Generator writes raw sales data to the raw bucket. The same upload starts two independent applications: Data-Processor, through an S3 event notification, and Sales-Report-App, through an EventBridge rule. Both write their results to the consumer bucket, and the raw data stays untouched. CloudWatch logs help us monitor and troubleshoot every step. To extend the project, catalog the data with AWS Glue, query it with Amazon Athena, and build dashboards on top.

Top comments (0)