DEV Community: jordan gonzález

Web Scraping with AWS Lambda in Rust

jordan gonzález — Tue, 04 Feb 2025 04:00:38 +0000

Recently, I have been working on delivering best-in-class observability products for AWS Lambda in Rust. Yet I haven't had the chance to build an actual function using the language.

For a first project I decided to code a web scraper which would parse the AWS docs and alert me when a new AWS Lambda runtime is supported or deprecated.

It might feel scary to work with an environment which is not directly provided – unlike NodeJS or Python – but the Rust community is growing rapidly and AWS even maintains other packages besides their SDKs.

An impressive advantage of using Rust for microservices is that the initialization time will be extremely small, compared to other popular runtimes. As per latest benchmarks from Maxime David, from his popular site lambda-perf, the average for hello-world is sitting at ~14ms.

Hello World

Let me first teach you how to easily set up an AWS Lambda in Rust using the vetted tool Cargo Lambda.

After installing the tool in your environment, just run the following command and you should be good to go!

cargo lambda new hello-world

You should now have a new directory, which looks very similar to a Cargo project, and your main file should look like this.

// main.rs
use lambda_runtime::{service_fn, Error, LambdaEvent};
use serde_json::{json, Value};

#[tokio::main]
async fn main() -> Result<(), Error> {
    let func = service_fn(func);
    lambda_runtime::run(func).await?;
    Ok(())
}

async fn func(event: LambdaEvent<Value>) -> Result<Value, Error> {
    let (_event, _context) = event.into_parts();

    Ok(json!({ "message": "Hello Ferris!" }))
}

Test locally by running the emulator with the watch command.

cargo lambda watch

Invoke in another terminal with the command invoke.

cargo lambda invoke --data-ascii "{ \"hello\": \"world\" }"
# {"message":"Hello Ferris!"}

Simple enough, right? We will dive into deploying to AWS later.

Web Scraping

There are multiple ways of scraping the web with Rust, I have chosen to use scraper due to the simplicity of my use case.

I just need to query a page, extract some tables, and see if there have been any changes since my last snapshot.

AWS Lambda with HTTP adapter

I will modify the previous example, because I want to be able to trigger this function through a Lambda URL.

cargo add lambda_http

I’ll separate the handler from the boilerplate needed to execute our HTTP adapter.

// main.rs
use lambda_http::{run, service_fn, tracing, Error};
mod http_handler;
use http_handler::handler;

#[tokio::main]
async fn main() -> Result<(), Error> {
    tracing::init_default_subscriber();

    run(service_fn(handler)).await
}

Now our handler looks more similar to an HTTP API in Rust.

// http_handler.rs
use lambda_http::{Body, Error, Request, Response};

pub(crate) async fn handler(_event: Request) -> Result<Response<Body>, Error> {
    let resp = Response::builder()
        .status(200)
        .header("content-type", "text/html")
        .body(“hello world”.into())
        .map_err(Box::new)?;
    Ok(resp)
}

Using `scraper`

Moreover, we need to install an HTTP client and our scraper.

cargo add reqwest scraper

To start scraping, we simply have to make a request to the site we are extracting the data from, convert that response to text, and then navigate through the document!

// http_handler.rs
use lambda_http::{Body, Error, Request, Response};
use reqwest::{header::HeaderMap, Client};
use scraper::{Html, Selector};
// … rest of code

async fn scrape(client: Client) {
    let response = client.get(AWS_LAMBDA_RUNTIMES_URL).send().await?;

    // Get the page as text
    let body = response.text().await?;

    // Parse the document
    let document = Html::parse_document(&body);
}

In my case, the elements I care about in the AWS Lambda docs page are the HTML tables, so I need to create a selector for it, and then find the references in the document.

This is how a row in the table looks like.

// scrape method

// Select the table element
let table_selector = Selector::parse(“table”).unwrap();

// Get the reference to the first table element in the document
let tables_ref = document.select(&table_selector).next().unwrap();

I will repeat the same process of creating a selector and then using it to get to every row of the table.

// scrape method

// Get the table body
let tbody_selector = Selector::parse("tbody").unwrap();
let tbody_ref = tables_ref.select(&tbody_selector).next().unwrap();

// Get the table rows from the table body
let tr_selector = Selector::parse("tr").unwrap();
let tr_elements = tbody_ref.select(&tr_selector);

// On every row, filter out empty spaces and breaklines
let rows = tr_elements.for_each(|e| {
   e.text()
    .map(|t| t.trim())
    .filter(|t| !t.is_empty())
    .collect::<Vec<_>>();
});

These will allow me to have an array of rows as Vec<Vec<String>>.

[ 
   [
      “Node.js 22”,”nodejs22.x”,”Amazon Linux 2023”,”Not scheduled”,”Not scheduled”,”Not scheduled”
   ],
   …
]

Deploying with AWS CDK

It is always a good practice to keep our cloud resources controlled through Infrastructure as Code (IaC). AWS provides the Cloud Development Kit (CDK) so developers can manage their resources more efficiently.

The same organization of developers of cargo-lambda provide a CDK construct – an abstraction that represents AWS CloudFormation resources and their configuration – for an AWS Lambda developed in Rust.

In your CDK project, simply install the cargo-lambda-cdk construct, available in multiple runtimes. Here we will use NPM to install it.

npm install cargo-lambda-cdk

And just use it in your stack like you would for any other AWS Lambda.

import { RustFunction } from 'cargo-lambda-cdk';

new RustFunction(stack, 'Rust function', {
  manifestPath: 'path/to/package/directory/with/Cargo.toml',
});

That’s pretty much it. My average initialization time, dependencies included, ended up being around ~40ms, while an invocation for processing took an average of ~300ms. Pretty good numbers overall, although I did not invest much in performance here.

I open sourced my project, so it can be used as a guide to your needs. There I include more processing in order to compare snapshots of scraped data. Please check it out!

[rust] add runtime scraper #4

duncanista posted on Feb 03, 2025

What?

Adds a function that scrapes the AWS Lambda docs to check the supported and deprecated runtimes.

How?

Used the following crates:

scraper: web scraping
similar: to diff data
prettytable-rs: to create easy-to-print tables
lambda_http: adapter for HTTP triggered AWS Lambdas

View on GitHub

Summary

Setting up Rust in AWS Lambda is far simpler than one might think. Combined with its astonishing start up times, I would say it is a powerful contender for microservices. This does not mean that you have to rewrite everything in Rust.

This guide is here to demonstrate a simple use case for an element inside a page, but you can build much more complex and powerful scrapers based on this framework. Always make sure to read more and research on your own. I hope this encourages you to explore scraping in Rust!

🇲🇽 This post is also available in Spanish in my personal blog

Web Scraping con AWS Lambda en Rust | Jordan González – Blog

Aquí aprenderás como hacer web scraping con Rust, el paquete será desplegado en una AWS Lambda, haciendo utilidad de las herramientas de cargo-lambda.

jordangonzalez.dev

References

[1] David, M. (2025). Lambda Cold Start benchmarks. Lambda-perf.

[2] Cargo Lambda. (2025). Rust functions on AWS Lambda made simple. Cargo Lambda.

[3] Rust Scraper. (2024) HTML parsing and querying with CSS selectors. scraper.

[4] Cargo Lambda. (2024). About CDK Construct to build Rust functions with Cargo Lambda. Cargo Lambda CDK construct.

Reduce AWS Lambda Cold Starts in .NET

jordan gonzález — Tue, 26 Nov 2024 17:03:12 +0000

Performance is a key concern for engineers, as it directly impacts spending, user experience, scalability, and reliability. Initialization time falls within this spectrum when working with any Serverless environments.

When executed for the first time, or after a long break, a Serverless workload requires provisioning resources. This is what we call a cold start. Initialization duration, the time spent initializing code and runtime, is part of the cold start.

Here’s a great developer guide on how AWS defines cold starts for AWS Lambda.

A few months ago, I was tasked with reducing Datadog’s .NET tracing overhead. The first thing that came to my mind was what my colleague Rey Abolofia did for Python in:

Reducing AWS Lambda Cold Starts

Rey Abolofia for AWS Community Builders ・ May 24

#coldstart #python #aws #serverless

So I thought, since .NET is a framework which requires C# to be compiled, there has to be a way to reduce the amount of work being done during runtime. After reading more about how .NET compilation works, I set out to compile our tracer ahead of time.

As a result, I achieved a 25% performance improvement in cold starts. Let me explain how I accomplished this.

.NET Compilation

Understanding how the .NET framework works is crucial, it will allow you to improve how your code is delivered. Many engineers overlook this aspect, mainly because they prioritize shipping code – but investing time in understanding how it works will pay dividends over time, as there are optimizations one can miss without this crucial knowledge.

Default Compilation

.NET applications are compiled into a language-agnostic Common Intermediate Language (CIL). Compiled code is stored in assemblies: files with a .dll or .exe file extension.

During runtime, the Common Language Runtime (CLR) is in charge of taking the assemblies and using a Just-In-Time (JIT) compiler to turn the Intermediate Language code into native code for the local machine to run. [1]

So, even though .NET applications are required to be compiled, there’s another compilation step during runtime – which requires compute power, and which subsequently translates into execution time.

There are two main techniques on how we can improve our Cold Starts, but the main idea behind them is Ahead-Of-Time (AOT) compilation. One is ReadyToRun, and the other one is Native AOT.

ReadyToRun

ReadyToRun (R2R) is a form of ahead-of-time (AOT) compilation. The binaries produced improve the startup performance by reducing the amount of work that the JIT compiler needs to do as our application loads. [2]

The main disadvantage is that R2R binaries are much larger because they contain both IL code and the native version of the same code.

Native AOT

Native AOT compilation produces an app that has been ahead-of-time compiled into native code for a specific architecture. Therefore, these applications will not use the JIT compiler during runtime. Not only will they have a faster startup time, but also a smaller memory footprint.

Another great advantage is that these binaries do not require the local machine to have the .NET runtime installed at all. Although a limitation is that you cannot cross-compile. [3]

Choosing a Compilation Strategy

The easy pick would be to compile with Native AOT all the time, right? Because it doesn’t require the .NET runtime, nor a JIT compiler. Unfortunately, there will be scenarios which you simply cannot do. [4]

For example, if you are doing dynamic loading, through Assembly.LoadFile, or runtime code generation using reflection, with System.Reflection.Emit, when compiling to Native AOT, you will find warnings during the process and your app will behave unexpectedly.

In my specific task, I couldn’t take advantage of Native AOT compilation because the Datadog .NET tracer uses dynamic loading and reflection. Due to the amount of required changes needed for this to work, I had to settle with R2R until we update the tracer.

How to?

R2R

To enable ReadyToRun compilation, simply add the following property in your .csproj:

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <!-- ...other properties -->
    <PublishReadyToRun>true</PublishReadyToRun>

Native AOT

For Native AOT compilation, you can set the property, also in your .csproj:

<PublishAot>true</PublishAot>

To ensure that your application is Native AOT compatible, you can set this property in the same file:

<IsAotCompatible>true</IsAotCompatible>

Benchmarks

To get data around cold starts, the methodology I used is simple: force a new sandbox for AWS Lambda every certain point in time, and emit telemetry by using an observability tool.

If you want a quick project to quick start and try it out for yourself, go to my example repository which uses the AWS CDK to benchmark a Hello World app with these strategies.

Hello World

For a simple AWS Lambda serializing an API Gateway HTTP payload and returning it, we see almost no benefits when using R2R, at around ~10ms removed. But when compiling to Native AOT, we can see an improvement of 75%, with around ~400ms being saved.

Due to lack of cross-compilation, I couldn't show the data for x86_64 for this test.

Datadog Tracer

For an immense codebase like the Datadog .NET tracer, publishing a release with ReadyToRun enabled improved positively the performance, as said before, a 25% cut during initialization.

This code is publicly available, feel free to check it out in DataDog/dd-trace-dotnet#5962.

[build] Build tracer with ReadyToRun #5962

duncanista posted on Aug 29, 2024

Summary of changes

Allows tracer publishing to be compiled with ReadyToRun to improve Serverless workloads init duration.

Reason for change

It has showcased a 500ms init duration improvement for AWS Lambda. Potentially could be used for other workloads in the future.

Implementation details

Followed #4573 and ReadyToRun docs.

Test coverage

TBD
Tested manually in AWS Lambda.

Other details

Increases tracer size by 3x.

View on GitHub

Summary

In general, understanding how the compiler works will open a lot of doors for you to become a better engineer, and give you the foundational knowledge to think of ways to improve your applications’ performance.

The clear benefit of applying this in Serverless workloads is that your applications will be able to serve faster, and save money at the same time.

For more improvements, like stripping and trimming, I'd recommend deep diving into the referenced content and the AWS developer guide to compile .NET into Native AOT.

🇲🇽 This post is also available in Spanish in my personal blog

Reduce Cold Starts en .NET para AWS Lambda | Jordan González – Blog

Aprende como mejorar drasticamente el rendimiento de tus aplicaciones .NET durante un inicio en frío en AWS Lambda, o cualquier otro entorno sin servidor.

jordangonzalez.dev

References

Thanks to Lucas Pimentel, who explained to me that this was possible.

[1] Microsoft. (2024). What is .NET Framework: Architecture of .NET Framework. Microsoft Learn.

[2] davidwrighton, gewarren, & Miskelly. (2022, June). ReadyToRun Compilation. Microsoft Learn.

[3] LakshanF et al. (2024, October 15). Native AOT Deployment. Microsoft Learn.

[4] stevewhims & mattwojo (2022, October). .NET Native and compilation. Microsoft Learn.

DEV Community: jordan gonzález

Web Scraping with AWS Lambda in Rust

Hello World

Web Scraping

AWS Lambda with HTTP adapter

Using scraper

Deploying with AWS CDK

[rust] add runtime scraper #4

What?

How?

Summary

Web Scraping con AWS Lambda en Rust | Jordan González – Blog

References

Reduce AWS Lambda Cold Starts in .NET

Reducing AWS Lambda Cold Starts

Rey Abolofia for AWS Community Builders ・ May 24

.NET Compilation

Default Compilation

ReadyToRun

Native AOT

Choosing a Compilation Strategy

How to?

R2R

Native AOT

Benchmarks

Hello World

Datadog Tracer

[build] Build tracer with ReadyToRun #5962

Summary of changes

Reason for change

Implementation details

Test coverage

Other details

Summary

Reduce Cold Starts en .NET para AWS Lambda | Jordan González – Blog

References

Using `scraper`