Why I chose F# for our AWS Lambda project

The dilemma

I wanted to create a simple Lambda function to be able to track how our users use the website and the web application without a 3rd party and a ton of external dependencies, especially avoiding 3rd party Javascript and leaking out data to mass surveillance companies. The easiest way is to use a simple tracking 1x1 pixel or beacon that collects just the right amount of information (strictly non-PII). This gives us enough information for creating basic funnels, that covers most of our needs.

First Option: Python

My default language (regardless of what I am going to work on) is Python. It has many great features and it is easy to prototype in it and the performance is great once you are using a C++ or Rust backed library. This also introduces a few issues when you are trying to deploy to AWS Lambda. I develop mainly on macOS and Lambda runs on Linux. Once you need to compile anything it is hard to get it right because Python does not support compiling to a different platform.

https://stackoverflow.com/questions/44490197/how-to-cross-compile-python-packages-with-pip

I was running into packaging issues because on Mac it is not easy to cross-compile and package Python code, maybe if I would create a proper package but I could not find a simple way without Docket. It would extremely valuable if Python had a way to compile a package that you upload to AWS and it works, 100%. I was running into problems that it was working on my Mac and did not work on AWS. I haven't had enough time to investigate.

Second Option: Rust

Rust became the rising star over the years and I try to use it as much as possible with mixed success. My biggest problem is with Rust the low-level nature and the quirky features, that are hard to reason about. From AWS Lambda examples:

use lambda::handler_fn;
use serde_json::Value;

type Error = Box<dyn std::error::Error + Send + Sync + 'static>;

#[tokio::main]
async fn main() -> Result<(), Error> {
    let func = handler_fn(func);
    lambda::run(func).await?;
    Ok(())
}

async fn func(event: Value) -> Result<Value, Error> {
    Ok(event)
}

Do you think that everybody understands immediately what is going on here? I don't. Even if I do, how am I going to explain this to a junior dev? How long does it take to get productive in Rust? I know that for extreme performance we might need this, but our current application is super happy without Rust, we do not have a performance problem. It is more important that developers are productive and the code is super simple to understand.

And the winner is: Fsharp

Member of the ML family, running on the .NET platform, pretty mature ecosystem. Developers can pick up quickly, especially the way we use it, simple functions will do with small types. The performance is great out of the box, in case you need more you have great tooling around it.

Our handler function:

  let handler(request:APIGatewayProxyRequest) =

    let httpResource =
      match isNull request.Resource with
      | true  -> "None"
      | _     -> request.Resource

    let httpMethod =
      match isNull request.HttpMethod with
      | true  -> "None"
      | _     -> request.HttpMethod

    let httpHeadersAccept =
      match isNull request.Headers with
      | true  -> "None"
      | _     -> getOrDefault request.Headers  "Accept" "None"

    let acceptImage =
      let pattern = @"image/"
      let m = Regex.Match(httpHeadersAccept, pattern)
      m.Success

    let log = String.Format("{0} :: {1} :: {2}", httpResource, httpMethod, httpHeadersAccept)
    LambdaLogger.Log(log)
    match (httpResource, httpMethod, httpHeadersAccept, acceptImage) with
    | ("/trck",         "POST", "application/json", _    ) -> trckPost(request)
    | ("/trck",         "GET",  _,                  true ) -> trckGet(request)
    | ("/trck/{image}", "GET",  _,                  true ) -> trckGet(request)
    | ("/echo",         "GET",  _,                  _    ) -> echoGet(request)
    | (_,               _,      _,                  _    ) -> notFound(request)

Pretty readable code, sure, you have to deal with nulls but Fsharp gives you great tooling around it. It took me probably a couple of days from having zero experience with .NET to deploy the first working API that has all of the functionality we are looking for. I might not have idiomatic Fsharp yet, but I am happy with the results so far. In the last couple of weeks, I have written many small tools in Fsharp, mostly dealing with the AWS APIs, I like it so much that I replaced my Python first approach and I go and try to implement everything in F# first. I can develop at the same pace as with Python but the result is much more solid code and easier on deployments (goodbye pip).

I think Fsharp is exactly in the sweet spot of programming languages, good enough performance, nice enough features, and a ton of great libraries. It does not have the problem that Python suffers, you can create a single zip that will work on all platforms. It also free from exposing the low-level details that I do not want to care about in business domain code, what Rust does.

Comments

[Phillip Carter]

Heya, nice post! Glad to hear F# is working well for you.

With regards to idiomatic code, your code is actually pretty close! It's usually pretty easy for people to write iditomatic F# code. Here's a quick suggestion:

The resource values can be written like this:

let httpResource =
    match request.Resource with
    | null  -> "None"
    | resource -> resource

The pattern match on null directly is usually how null checks can be handled. The isNull function is usually for if expressions.

Another nice thing is that when working with purely F#-defined types, you never need to do a null check because it's not allowable to assign null to an F#-defined type (there is technically a way with the AllowNullLiteral attribute, but only on classes).

Your log value can also be written as an interoplated string starting with F# 5, which is releasing this November:

let log = $"{httpResource} :: {httpMethod} :: {httpHeadersAccept}"

Loved the post!

[Istvan]

Thanks Phillip, much appreciated! We are approaching 100K LOC in the project and I have to say we are pretty happy. We have moved beyond the initial sync approach and working on async calls wherever we can. The code got a lot cleaner in the last while too. We have started to use ROP and created a way of bubbling up errors from low level infra code to user facing functions. We have also created a few Lambda function, some of them is calling other functions to fetch data. It works very well.

[Phillip Carter]

Very cool! 100k LoC is quite a lot. I'm glad you're having a great time. Is there anything in F# or F# tooling you feel is missing that you'd love to use?

[Istvan]

Yeah I should clarify that we have everything in that 100K LOC, devops, frontend (Elm) and many-many backend systems (F#). The actual breakdown I am not even sure about.

Few things about F#:

• the foundation of the language is rock solid (no nulls, discriminated unions, type params, etc.) We use currying a lot makes some things like logging or shortening a function with many parameters easy.

• libraries are (mostly) amazing, some gotchas with interop with async C# and nested exceptions coming from C# libraries

• working with async was not an easy start, 90% of documentation is about hello world example with printf which is not something that you do a lot (at least not us). Documentation should be just: "use this until you understand in depth what you are doing" I know this is a slippery slope but it is funny how this link gave us more than all the other documentation combined:

fssnip.net/hx/title/AsyncAwaitTask...

• traceability is a bit hard, even though there are great tools like dotnet trace and speedscope (even though there is no out of the box support for, start this application and trace everything it does (you have to work with ./bin... & echo $!)

• traceability in a cloud / lambda / serverless environment, we had some hoops because of the lack of example how to operate in these environments effectively

• not really a tooling issue but ROP (Railway Oriented Programming) is really neat and it would be great to see more about it or implement the functions used in ROP a lot (Option.map, Option.bind, etc.) for other types which we currently do ourselves.

• I personally use dotnet fsi a lot, it is a bit unfortunate that it does not have history by default (maybe there is already some project addressing it)

These are my thoughts, my co-worker chime in, he has additional points I am sure. :) So far I would say this was the best decision to get into F# that I have made in the last 5 years. We are happy with the performance we got and how solid our system is, even though we are dealing with horrific inputs (broken CSV files, emails, etc.) that are traditionally hard to deal with. Since we introduced ROP the code based became super simple with no visible branching and it is very simple to express complex computations, that can be read and modified by juniors with ease.

[Benjamin Houdu]

Have you also considered Golang for your usage? For your python problem, I suppose you are not using CI to package your lambda, why so?

[Istvan]

Have you also considered Golang for your usage?

No. I gave up on Golang long time ago, it is neither memory safe nor high level enough to be used for business code. On the top of that GC is a problem in high performance use cases. It literally does not give us anything that we need.

For your python problem, I suppose you are not using CI to package your lambda, why so?

Sorry but it is not my intention to waste time on a shortcoming of a language to make it barely work. Python is a deep rabbit hole that we do not have time to go down in. I am using Python on a daily basis for other projects and it is not even comparable how much more efficient it is to work in F#. Basically if the compiler happy we have a really high chance of successful operation, while with Python it is always testing in production even if we do all the checks possible on our end. Again, I use both languages on a daily basis and I do not want my next project happen in Python at all. If you do not understand why ML family languages are superior to C family languages than maybe you should look at why Rust, F# and Haskell exists and how do they rank in terms of security, performance, expressiveness and reliability comparing to other languages. I have spent enough time on languages where most of the time is spent on "how the hell am i supposed to do this in X" instead of "what business features should I implement / improve on today". With F# we (on purpose) write simple, junior friendly code and refrain from using things that are hard to explain to newcomers. We have confidence in our software that it does what it supposed to because of the basic fundamental principles backing everything in this language. (discriminated unions, currying, immutability, referential transparency, idempotency, options, railway oriented programming, null safety (you have to work on it when working with C# code) and so on) It is really painful to switch back to Python where function signatures mean nothing, you have no idea if a function returns a new object of mutates an object to pass in (often silently). Python feels like a hockey stick when i need a precision drill.

[Benjamin Houdu]

I actually agree on Python being a nightmare to debug, and testing at runtime only by design. Thanks for your detailed answer. I don't know much about F# so I guess I'll take a look.

I personally use TypeScript for most of my work now (but you want to get rid of numerous dependencies so it doesn't seem a good choice for you, still for lambdas it is a great fit as you can use webpack post compilation and get really small packages in the end with quick cold starts). It seems a middleground choice, but it can communicate with basically anything, runs everywhere, many people know nodejs as a basis and it is "good enough" for me.

[Lucis]

Thank you very much for that post!

I've been thinking in start using F# on AWS Lambda and this is the exactly kind of post I was hoping to find. Very nice comments and a helpful message overall!