loading...

Why I chose F# for our AWS Lambda project

l1x profile image Istvan ・3 min read

The dilemma

I wanted to create a simple Lambda function to be able to track how our users use the website and the web application without a 3rd party and a ton of external dependencies, especially avoiding 3rd party Javascript and leaking out data to mass surveillance companies. The easiest way is to use a simple tracking 1x1 pixel or beacon that collects just the right amount of information (strictly non-PII). This gives us enough information for creating basic funnels, that covers most of our needs.

First Option: Python

My default language (regardless of what I am going to work on) is Python. It has many great features and it is easy to prototype in it and the performance is great once you are using a C++ or Rust backed library. This also introduces a few issues when you are trying to deploy to AWS Lambda. I develop mainly on macOS and Lambda runs on Linux. Once you need to compile anything it is hard to get it right because Python does not support compiling to a different platform.

https://stackoverflow.com/questions/44490197/how-to-cross-compile-python-packages-with-pip

I was running into packaging issues because on Mac it is not easy to cross-compile and package Python code, maybe if I would create a proper package but I could not find a simple way without Docket. It would extremely valuable if Python had a way to compile a package that you upload to AWS and it works, 100%. I was running into problems that it was working on my Mac and did not work on AWS. I haven't had enough time to investigate.

Second Option: Rust

Rust became the rising star over the years and I try to use it as much as possible with mixed success. My biggest problem is with Rust the low-level nature and the quirky features, that are hard to reason about. From AWS Lambda examples:

use lambda::handler_fn;
use serde_json::Value;

type Error = Box<dyn std::error::Error + Send + Sync + 'static>;

#[tokio::main]
async fn main() -> Result<(), Error> {
    let func = handler_fn(func);
    lambda::run(func).await?;
    Ok(())
}

async fn func(event: Value) -> Result<Value, Error> {
    Ok(event)
}

Do you think that everybody understands immediately what is going on here? I don't. Even if I do, how am I going to explain this to a junior dev? How long does it take to get productive in Rust? I know that for extreme performance we might need this, but our current application is super happy without Rust, we do not have a performance problem. It is more important that developers are productive and the code is super simple to understand.

And the winner is: Fsharp

Member of the ML family, running on the .NET platform, pretty mature ecosystem. Developers can pick up quickly, especially the way we use it, simple functions will do with small types. The performance is great out of the box, in case you need more you have great tooling around it.

Our handler function:

  let handler(request:APIGatewayProxyRequest) =

    let httpResource =
      match isNull request.Resource with
      | true  -> "None"
      | _     -> request.Resource

    let httpMethod =
      match isNull request.HttpMethod with
      | true  -> "None"
      | _     -> request.HttpMethod

    let httpHeadersAccept =
      match isNull request.Headers with
      | true  -> "None"
      | _     -> getOrDefault request.Headers  "Accept" "None"

    let acceptImage =
      let pattern = @"image/"
      let m = Regex.Match(httpHeadersAccept, pattern)
      m.Success

    let log = String.Format("{0} :: {1} :: {2}", httpResource, httpMethod, httpHeadersAccept)
    LambdaLogger.Log(log)
    match (httpResource, httpMethod, httpHeadersAccept, acceptImage) with
    | ("/trck",         "POST", "application/json", _    ) -> trckPost(request)
    | ("/trck",         "GET",  _,                  true ) -> trckGet(request)
    | ("/trck/{image}", "GET",  _,                  true ) -> trckGet(request)
    | ("/echo",         "GET",  _,                  _    ) -> echoGet(request)
    | (_,               _,      _,                  _    ) -> notFound(request)

Pretty readable code, sure, you have to deal with nulls but Fsharp gives you great tooling around it. It took me probably a couple of days from having zero experience with .NET to deploy the first working API that has all of the functionality we are looking for. I might not have idiomatic Fsharp yet, but I am happy with the results so far. In the last couple of weeks, I have written many small tools in Fsharp, mostly dealing with the AWS APIs, I like it so much that I replaced my Python first approach and I go and try to implement everything in F# first. I can develop at the same pace as with Python but the result is much more solid code and easier on deployments (goodbye pip).

I think Fsharp is exactly in the sweet spot of programming languages, good enough performance, nice enough features, and a ton of great libraries. It does not have the problem that Python suffers, you can create a single zip that will work on all platforms. It also free from exposing the low-level details that I do not want to care about in business domain code, what Rust does.

Posted on by:

l1x profile

Istvan

@l1x

Data | Cloud | Functional Programming

Discussion

pic
Editor guide
 

Heya, nice post! Glad to hear F# is working well for you.

With regards to idiomatic code, your code is actually pretty close! It's usually pretty easy for people to write iditomatic F# code. Here's a quick suggestion:

The resource values can be written like this:

let httpResource =
    match request.Resource with
    | null  -> "None"
    | resource -> resource
Enter fullscreen mode Exit fullscreen mode

The pattern match on null directly is usually how null checks can be handled. The isNull function is usually for if expressions.

Another nice thing is that when working with purely F#-defined types, you never need to do a null check because it's not allowable to assign null to an F#-defined type (there is technically a way with the AllowNullLiteral attribute, but only on classes).

Your log value can also be written as an interoplated string starting with F# 5, which is releasing this November:

let log = $"{httpResource} :: {httpMethod} :: {httpHeadersAccept}"
Enter fullscreen mode Exit fullscreen mode

Loved the post!

 

Thanks Phillip, much appreciated! We are approaching 100K LOC in the project and I have to say we are pretty happy. We have moved beyond the initial sync approach and working on async calls wherever we can. The code got a lot cleaner in the last while too. We have started to use ROP and created a way of bubbling up errors from low level infra code to user facing functions. We have also created a few Lambda function, some of them is calling other functions to fetch data. It works very well.

 

Very cool! 100k LoC is quite a lot. I'm glad you're having a great time. Is there anything in F# or F# tooling you feel is missing that you'd love to use?

Yeah I should clarify that we have everything in that 100K LOC, devops, frontend (Elm) and many-many backend systems (F#). The actual breakdown I am not even sure about.

Few things about F#:

  • the foundation of the language is rock solid (no nulls, discriminated unions, type params, etc.) We use currying a lot makes some things like logging or shortening a function with many parameters easy.

  • libraries are (mostly) amazing, some gotchas with interop with async C# and nested exceptions coming from C# libraries

  • working with async was not an easy start, 90% of documentation is about hello world example with printf which is not something that you do a lot (at least not us). Documentation should be just: "use this until you understand in depth what you are doing" I know this is a slippery slope but it is funny how this link gave us more than all the other documentation combined:

fssnip.net/hx/title/AsyncAwaitTask...

  • traceability is a bit hard, even though there are great tools like dotnet trace and speedscope (even though there is no out of the box support for, start this application and trace everything it does (you have to work with ./bin... & echo $!)

  • traceability in a cloud / lambda / serverless environment, we had some hoops because of the lack of example how to operate in these environments effectively

  • not really a tooling issue but ROP (Railway Oriented Programming) is really neat and it would be great to see more about it or implement the functions used in ROP a lot (Option.map, Option.bind, etc.) for other types which we currently do ourselves.

  • I personally use dotnet fsi a lot, it is a bit unfortunate that it does not have history by default (maybe there is already some project addressing it)

These are my thoughts, my co-worker chime in, he has additional points I am sure. :) So far I would say this was the best decision to get into F# that I have made in the last 5 years. We are happy with the performance we got and how solid our system is, even though we are dealing with horrific inputs (broken CSV files, emails, etc.) that are traditionally hard to deal with. Since we introduced ROP the code based became super simple with no visible branching and it is very simple to express complex computations, that can be read and modified by juniors with ease.

 

Have you also considered Golang for your usage? For your python problem, I suppose you are not using CI to package your lambda, why so?