DEV Community: Danny Reed

Self-Provisioning Runtimes & Serverless DX

Danny Reed — Wed, 27 Apr 2022 14:43:00 +0000

Introduction

Developer experience (DX) has come a long way since the punch card days, but in the world of serverless development, we're actually approaching a significant regression in DX.

Some of the practices and principles that unlock the power of serverless also make developer experience worse. Self-provisioning runtimes are the solution, and that's only the beginning of their value.

TL;DR:

Using serverless best practices makes for a poor DX
We should care about that because it costs us both money and happiness
We should pursue self-provisioning runtimes for better DX (and many other reasons)

Serverless best practices push us into bad DX

In a very general way, I'm wrong. Serverless affords us the luxury of spending very little time on "undifferentiated heavy lifting" and more time writing the "special sauce" code that makes our projects distinct and competitive. That, however, is just one metric that contributes to the overall DX.

Consider these best practices that double as DX detractors:

Infrastructure as Code

IaC is critical to the successful management and evolution of serverless applications. I think the benefits of IaC are generally well understood, so I'll forego that argument here.

Let's consider CloudFormation. Perhaps you've experienced the very long dev loop involved with writing infrastructure templates. You get an error, make a fix, push to develop, wait, repeat. I think it's fair to say that IaC is hard because of the long dev loop and the challenges of "config languages" like YAML and JSON.

IaC use cases demand a lot from configuration-oriented languages. We aren't just configuring, we are also defining. We want maps, string manipulation, !Ref-ing, and lots of other things that are a stretch for "config." Sometimes even simple things like concatenation can wind up being tricky to read:

!Join
  - ''
  - - 'arn:'
    - !Ref AWS::Partition
    - ':s3:::elasticbeanstalk-*-'
    - !Ref AWS::AccountId

IaC templates are slow and difficult to develop, but because we believe in the benefits of IaC, we dutifully shoulder that burden.

(Don't worry, CDK fans. We'll consider CDK in Appendix A.)

Configuration Over Code

"Configuration over code" falls in line with the maxim, "code is a liability" and that we should prefer to configure a general-purpose service over writing custom code to maintain.

One example of this is using Step Functions instead of Lambda. Based on our practice of "configuration over code" we should (generally) use Lambda only when Step Functions can't reasonably be used instead.

The depressing ramification of "configuration over code" is that now business logic is infrastructure, too.

If YAML was rough for infrastructure, imagine how bad it is at business logic? Whether you use YAML, JSON, or a domain-specific language (DSL), defining business logic as config is a nightmare:

No debugger
Almost no use of IDE tools like IntelliSense
The available linters aren't great
IaC-defined business logic doesn't read at all like a program

We've reset the DX clock by decades.

Look at this (silly) number classifier written in Amazon States Language:

{
  "States": {
    "ClassifyNumber": {
      "Type": "Choice",
      "Choices": [
        {
          "Variable": "$.value",
          "NumericEquals": 0,
          "Next": "IsSmallNumber"
        },
        {
          "Variable": "$.value",
          "NumericGreaterThan": 100,
          "Next": "IsLargeNumber"
        }
      ]
    },
    "IsSmallNumber": {
      "Type": "Pass",
      "Next": "SuccessState"
    },
    "IsLargeNumber": {
      "Type": "Pass",
      "Next": "SuccessState"
    },
    "SuccessState": {
      "Type": "Succeed"
    }
  },
  "StartAt": "ClassifyNumber"
}

Unless you're an ASL guru, I trust you found that difficult to read and quite verbose. Now compare to this code:

function classifyNumber(num) {
    if(num > 100) return 'Large';
    return 'Small';
};

I admit that's not a realistic program, but we can see the extra layer of developer-effort that has to transform simple logical statements into a highly-specialized domain-specific language (DSL).

(Don't worry Workflow Studio fans, I'll address that in Appendix B)

The Costs of Bad DX

Gap-Closing

We might think very highly of ourselves for accepting these tremendous burdens in order to achieve a philosophically-consistent outcome. This sort of digital Asceticism isn't doing us favors.

One metric that contributes to DX is:

How much distance is there between the developer's intent and their code?

Another way to phrase it is:

How different is the pseudocode from the final code?

I think it's safe to say that nobody has ever pseudocoded something that looked remotely similar to the ASL definition we saw above. So there's distance there -- there's distance between intent and code, and the developer has to close that gap. Doing so is not free:

Gap-Closing Costs

Bugs introduced while transforming intent into final code
Higher maintenance costs due to unintuitive/unreadable codebase
Developer time
Training/ramp-up costs (especially with DSL's)
Developer morale, retention, etc.

Is "developer morale" a stretch? Consider this: I personally dislike JavaScript for various reasons, but it's one of my go-to languages. Why? I'm productive with it. We developers crave productivity -- it boosts our work-satisfaction and perceived effectiveness. When we have to use technology with poor DX, we feel less productive/effective/satisfied and that is a cost we shouldn't ignore.

Recruiting

You will pay more to find and retain a developer who is thoroughly-versed in an obscure DSL than you will pay to hire a developer skilled in general-purpose languages like TypeScript, C#, or Python.

If you want to make life easier when it comes to recruiting, don't make your stack special. Make it normal.

Have Your Cake, Eat it Too: Self-Provisioning Runtimes

I first came across the term "Self-Provisioning Runtimes" on this episode of the excellent Serverless Chats podcast by Jeremy Daily. In this episode he interviews Sean (swyx) Wang and a good portion of their discussion centers on the topic. Not knowing what self-provisioning runtimes (let's call them SPR's, OK?) were, I listened as they began to describe how, essentially, you just write your business logic, and then the system looks at your code, figures out what infrastructure you need, and then runs it.

Swyx poses the idea this way:

If the Platonic ideal of Developer Experience is a world where you ”Just Write Business Logic”, the logical endgame is a language+infrastructure combination that figures out everything else.

Jeremy amusingly described his reaction to reading this sentence for the first time:

I put my arms out like this, lights light up, music starts playing, doves fly out from behind me. I'm like, "Yes! Yes! That. Why do more people not get that?"

I had a pretty similar reaction.

This is fake code that doesn't work with any SPR, but it gives you the idea of what we're going for:

const uploadEndpoint = spr.postEndpoint('/upload');

// Upload handler
uploadEndpoint.onUpload(file => {
    spr.store(file)
       .wait(Duration.Days, 14)
       .archive(file)
       .wait(Duration.Years, 1)
       .delete(file);
});

Let's imagine that this snippet gets processed by the SPR, and it spits out an API Gateway, a Step Functions state machine, and an S3 bucket with some lifecycle rules.

The distance between intent and code is extremely short. There's no accompanying infrastructure template to write. We just write business logic, and the SPR figures out how to architect that.

All Your Practices Are Belong to the SPR

Developers no longer need to subject themselves to the pain of writing business logic in config languages, deal with the quirks of YAML, or learn obscure DSL's. The SPR knows all the best practices and will create the best-fit architecture for the business logic at hand.

DX Regained

Now that we're back to writing business logic in programming languages like TypeScript, C#, Python, or whatever you want, you can use your debugger, your IDE tools, your language skills, etc.! This is the DX we want, and it's the DX we should push for.

It's not because we can't handle the pain, but because we are actually worse and more expensive developers when we are in pain.

Rainbows and Unicorns

You're saying, "great, but it's made up, right?" Yes, kind of. Why doesn't this exist? Swyx does a pretty good round-up on the contenders out there in his article. The most mature one is Serverless Cloud, and you should definitely check them out.

I think we need to see more attempts, loftier goals, and more folks throwing energy at this. I think Serverless Cloud is awesome, but I have a mercilessly-ambitious wishlist, about which I'll share in a future article.

Appendix A: CDK

I think of CDK as a stepping stone on the way to where we want to go.

The CDK is great. I found it long before I heard of SPR's and I thought it was the most epic thing I had ever seen. The reasons I liked it basically boiled down to DX pain-relief.

No more YAML/JSON
Constructs let me reuse stuff, leverage abstractions
IntelliSense works
I can catch many errors before running my long pipeline
Easy infrastructure diffs
Productivity

Ben Kehoe and others have raised concerns with CDK for various valid reasons, but I was so hungry for a better DX I was disposed to try to defend it against all the arguments, no matter how good they seemed.

The CDK does help with DX, and that's good, but it's not solving the whole problem, and whatever does solve the whole problem will make CDK-like tools easy to abandon.

Why is CDK just a stepping stone?

For one thing, you still have to write infrastructure separately from your code, so we haven't achieved swyx's "Platonic ideal". Look at this Lambda definition. It clearly points to where the business logic lives, over in lib/lambda/myLambdaFunction. This is an improvement over doing things in YAML/JSON, but it's just a way to do the same thing with less pain.

const myLambda = new lambda.Function(this, 'my-lambda-name', {
    // See? your business logic lives separately!
    code: lambda.Code.fromAsset('lib/lambda/myLambdaFunction'), 
    runtime: lambda.Runtime.NODEJS_14_X,
    architecture: lambda.Architecture.ARM_64, 
    handler: 'index.handler',
    functionName: 'my-lambda-name',
});

What about CDK-defined step functions?

Let's look at a CDK-defined state machine that I created recently. It runs an Athena query, then uploads the results to an external web service using Lambda.

Here, we did the following:

Used a programming language
Practiced "infrasructure as code"
Practiced "config over code"

Home run? Almost. Here's why it's not quite there:

We aren't describing business logic; we are still describing infrastructure.
The developer still has the burden of translating business logic into "state machine form" and then defining the state machine with CDK. We haven't closed the "intent to code" gap yet.

const artifactStateMachine = new stepFn.StateMachine(this, `StateMachine-${artifact.viewName}`, {
  // Step 1: Start the Athena query
  definition: new tasks.AthenaStartQueryExecution(this, `Run Athena Query: ${artifact.viewName}`, {
    queryString: `SELECT * FROM ${artifact.viewName}`,
    resultConfiguration: {
      outputLocation: {
        bucketName: dataPipelineBucket.bucketName,
        objectKey: artifact.viewName
      },
    },
    queryExecutionContext: {
      databaseName: 'MyAthenaDB'
    },
    workGroup: 'MyAthenaWorkGroup',
    integrationPattern: IntegrationPattern.RUN_JOB
    // Step 2: Invoke uploader Lambda
  }).next(new tasks.LambdaInvoke(this, `Upload to destination: ${artifact.viewName}`, {
      lambdaFunction: myLambda,
      inputPath: '$.QueryExecution.ResultConfiguration',
    })),
  role: stateMachineRole,
});

Here's an example of what we might do in an ideal world, with an SPR:

const queryResult = await data.query(viewName);
await externalService.upload(queryResult);

The SPR would "compile" that down into something like what we saw in the CDK example above.

Notice that the developer doesn't need to:

Know what a state machine is or how to define one.
Know if a state machine is the best choice for this workload.
Write the Lambda code separately

Appendix B: Step Functions Workflow Studio

AWS went to the trouble to make us a really nice Workflow Studio for Step Functions, so why doesn't that count as a great DX?

Fair point. Even better when you use the YAML/JSON export features to create your ASL files, and then reference them using serverless transforms in your CloudFormation template:

Resources:
  MyStepFunction:
    Type: AWS::Serverless::StateMachine
    Properties:
      DefinitionUri: aslDef.yml # <-- behold, the generated ASL file
      Events:
        # [omitted for brevity]

This is an improvement on writing those things by hand, but here are my challenges with this arrangement:

You can use the Workflow Studio to make your ASL file, but you still have to dig into it manually when it comes to merge conflicts or other version control-related tasks.
You can't "read" the logic of the state machine easily in ASL format, so you have to copy/paste it into the Workflow Studio to see what it really does. This means as a developer you can't really understand your code by looking at the code in the repo any more.
Copying and pasting YAML back and forth from the browser to the IDE seems error-prone at worst, and clunky at best.

Feedback

Thank you for considering my lengthy thoughts on this topic. I'm very excited to see what can be done in the wonderland of SPR's, and I'm hoping to push out some more articles that focus on defining what an ideal SPR looks like, what characteristics it needs to have, and what obstacles are in the way.

Please share your thoughts on the topic, relevant resources, or ideas for where to go next in my exploration of SPR's.

Debounce in Event-Driven Serverless

Danny Reed — Thu, 31 Mar 2022 12:58:07 +0000

Debounce

Debounce is a technique that seeks to consolidate several near-simultaneous events into one. It's related to fan-in, but debounce usually has a time component. One way to say it is, "If we get X related inputs within N seconds, only trigger one output." My favorite way to think of it is:

Use Case

In my case, I have several sensors that submit events when certain conditions occur. Each event will trigger some computations, but if several sensors submit an event within the same second, I just want to trigger one computation. Having N separate computations outputting N separate results and resulting in N separate rows in a data store can mean some rows are obsolete almost immediately after being created. It also means I have to store lots of extraneous data.

What I need is to debounce those events so that if N events come in "at once" then we just make one new row in the data store.

Challenges

This is a hard problem, especially in serverless. One reason it's hard is that in order to implement debounce, we need to find mechanisms to do two things:

Deduplicate
Delay

Approach

Deduplication: DynamoDB

DynamoDB will let us PutItem conditionally. This is important because we'll use a Condition Expression to prevent overwriting an existing record. Once we put an Item with a given key, we won't overwrite it. This means when we hook up a DynamoDB Stream to this table, it will only fire one event even if we send several PutItem calls down to DynamoDB at nearly the same time.

This snippet shows writing the record conditionally:

await db.send(new PutItemCommand({
    TableName: process.env.tableName,
    Key: marshall({
        pk: event.deviceId,
    }),
    ConditionExpression: 'attribute_not_exists(pk)', 
    Item: marshall({
        pk: event.deviceId,
        timestamp: event.disappearTime,
    }),
    ReturnConsumedCapacity: 'TOTAL',
}));

If the call to PutItem would overwrite an existing record, the call will throw a ConditionalCheckFailedException which you should catch (verify the exception name) and ignore as valid behavior.

It's worth noting that deduplication is a helpful mechanism by itself. If you just want deduplication and not a full debounce solution, you can just use this part and then hook up your Lambda target.

Delay: SQS

This part of the solution isn't without its drawbacks. SQS is one of the only serverless offerings that allows us to delay a message (something we usually avoid in event-driven architectures). Part of the nature of most event-driven services is that they don't hold onto stuff for more time than necessary to process it and pass it on. We actually want to put the brakes on this message for a bit in order to allow any "at the same time" messages to finish coming in. SQS facilitates that.

To configure a delay, we'll just set DelaySeconds to whatever value you feel suits your use case. I chose 1 second for mine. Here's my CloudFormation snippet:

AppDelayQueue:
  Type: AWS::SQS::Queue
  Properties:
    DelaySeconds: 1
    QueueName: AppDelayQueue
    KmsMasterKeyId: alias/aws/sqs # Enable encryption

Connecting it Up: DynamoDB Stream

We'll pipe our PutItem events to a Lambda using a DynamoDB Stream, and then the Lambda will use the SDK to enqueue the message in a delay queue. The code is very simple, and frankly I wish DynamoDB Streams would let you send messages directly to SQS, but alas, the options are currently Lambda and Kinesis Data Streams.

Edit: Nowadays, there's also EventBridge Pipes which can hook up to a DynamoDB Stream and then pipe your data into a wide variety of targets, including SQS Queues. That would be the more "managed" way to do things at this point.

Here's the simple code:

const {
    SQSClient,
    SendMessageCommand,
} = require('@aws-sdk/client-sqs');

const sqs = new SQSClient();
const QUEUE_URL = process.env.queueUrl;

exports.handler = async (event) => {
    const message = event.Records[0];

    console.log(JSON.stringify(message));

    // Parse out the relevant data
    const deviceId = message.dynamodb.NewImage.pk.S;
    const timestamp = parseInt(message.dynamodb.NewImage.timestamp.N);

    // Send message to SQS delay queue
    await sqs.send(new SendMessageCommand({
        MessageBody: JSON.stringify({
            deviceId,
            timestamp,
        }),
        QueueUrl: QUEUE_URL,
    }));
};

Diagram

All together now:

Tips

The nature of debounce involves setting a threshold for what should "count as one" so you'll always have messages falling just outside or just inside the threshold you set. Your app should be designed (downstream) so that getting a repeat event, say, 1.1 seconds after the first doesn't break stuff or corrupt data.
You may want to disable retries on the DynamoDB Stream depending on your use case. Leaving this on will cause failed messages to get recycled forever, which in this case wasn't helpful. This can't be done through the AWS Console, weirdly. You'll need to do it via the CLI (or through IaC). Read about it here.
If you want to measure how many records are getting debounced, you can add some code where you're PUTting to DynamoDB inside the catch. After you've confirmed that the exception name is ConditionalCheckFailedException you can either log a message or PUT to a CloudWatch metric for bonus points :) Just be careful with the latter option if you're dealing with tons of messages since CloudWatch metrics can be expensive!

FIFO Delay Queues with Deduplication?

Why don't you just use a FIFO queue from SQS with a delay? Using a deduplication ID (or content-based deduplication) you can achieve deduplication out-of-the-box. You can also build the delay right in. Honestly that's a really good option for most use cases.

The one really frustrating problem with it is that the deduplication feature doesn't just deduplicate within messages currently in the queue, but it deduplicates against all messages sent in the past 5 minutes. You currently cannot configure that to be shorter. In my use case, I actually need to send messages with the same deduplication ID within 5 minutes. I just want it deduplicated against what's actually in the queue at the time.

Conclusion

Event-driven architectures are very powerful, but debounce is hard to achieve. Thanks to the combination of DynamoDB and SQS, we can achieve this behavior and minimize extraneous data processing and storage.

Thoughts and criticisms welcome! Let me know how you're solving these challenges in your shop!

An attempt was made to access a socket...

Danny Reed — Wed, 12 Jan 2022 01:34:56 +0000

The Error

Today I tried to fire up a project I haven't run in a long time. It's a web app, and it happens to launch on a specific port: 50617.

I was frustrated to see this:

An attempt was made to access a socket in a way forbidden by its access permissions.

The Reason

I use Docker on Windows with the WSL2 engine, which involves Hyper-V. Some quick googling revealed that this error usually comes up when you're trying to access a port that has been reserved by something else. Hyper-V reserves a bunch of ephemeral ports, so it's a common culprit.

To confirm your port has already been reserved, open an admin command prompt and run this:

netsh interface ipv4 show excludedportrange protocol=tcp

This will show you the ranges that have been reserved. My port was in one of the ranges.

The Soltuion

I found the solution on this GitHub comment, so full props to the author for finding the solution. I'm just writing this post to help make this easier to find than digging through long GitHub comment chains :)

The gist of it is:

Open an Administrator command prompt
Stop the Windows NAT Driver service
```
net stop winnat
```
Create a manual reservation for the specific port you're trying to use (50617 in my case).
```
netsh int ipv4 add excludedportrange protocol=tcp startport=50617 numberofports=1
```
Start the Windows NAT Driver service
```
net start winnat
```
Run your project!

Conclusion

This should get your port reserved and let you get your project running. If you have additional tips related to this issue, leave a comment!

Thanks!

2 Important Optimizations for Athena with S3

Danny Reed — Wed, 01 Dec 2021 14:33:09 +0000

Do you use Athena to query data in S3? Coupled with the use of the Parquet file format, this is a really powerful combination. S3 is very cheap, Athena is very cheap, and wait a second....my bill is steadily going up? What happened?

That's what happened to me, and it took a while to identify two key configuration mistakes that could have been avoided easily. I will share them both here so you can do these right the first time and save yourself one major headache and one minor one.

Optimization 1: Athena Query Results

Did you know that by default, Athena will save a CSV of the output of every query you run? It stashes it in S3. If your system runs Athena queries periodically, this can add up (especially in the Standard storage tier). Honestly these serve no purpose for us, so this wound up creating terabytes of unnecessary data.

Fixes

One fix is to create a lifecycle rule on the "folder" in which Athena stores these results. For our purposes, it's sufficient to keep just a few days worth of results for debugging purposes. You'll need to identify where those results are stored, then create the appropriate lifecycle rule to delete old files.
You can also create a rule to at least move these to a cheaper-than-Standard storage tier to reduce costs without deleting any data. This should at least minimize the costs if you truly have a reason to keep all your query results.

Optimization 2: KMS

This one was sneaky. It took me quite a while to figure out why my KMS costs were going through the roof. It turns out that when Athena pulls out my .parquet files from S3, it makes kms:decrypt calls for every file it retrieves. When you're retrieving a hundred thousand files dozens of times per day, this gets really expensive.

Root Cause

The issue was that I had selected S3 Encryption for the bucket in which my .parquet files are stored. This works fine for many use cases, but when frequently retrieving many files, it's expensive, and you should prefer Bucket Keys with SSE-KMS.

Fix

Use a bucket key. I was originally using SSE-S3 encryption, but to use a bucket key, you'll need to move to SSE-KMS with Bucket Key enabled. The use of a bucket key is really the crux of the solution here.

From AWS docs:

Amazon S3 Bucket Keys reduce the cost of Amazon S3 server-side encryption using AWS Key Management Service (SSE-KMS). This new bucket-level key for SSE can reduce AWS KMS request costs by up to 99 percent by decreasing the request traffic from Amazon S3 to AWS KMS. With a few clicks in the AWS Management Console, and without any changes to your client applications, you can configure your bucket to use an S3 Bucket Key for AWS KMS-based encryption on new objects.

Note that at the end of the quote it says, "on new objects."

This is a depressing detail if you've already stored data without bucket key enabled. You'll have to do what I did and follow this process:

Enable the bucket key for new objects
Enable daily manifests on your bucket
Wait a day or two for manifests to be generated
Make a backup of your data (some will be encrypted with bucket key, most will not)
Create a "Batch Operation" job in S3 to COPY your files over top of themselves, and have the job configure bucket key encryption as it overwrites.

This process is scary because you're essentially overwriting your production data. I recommend thorough testing in a test environment and good study of the related documentation.

Outcome

We have seen our S3 storage costs go down by close to 70%, and our KMS costs go down by 83% for an overall cost savings of hundreds of dollars per month.

These small details are easy to overlook since they don't impact the functionality of your application. Until these two pieces are configured correctly, however, you'll miss out on the cost effectiveness that these powerful services can offer.

Resources

AWS Docs: S3 Lifecycle Rules
AWS Docs: Athena Query Results
AWS Docs: Bucket Keys
AWS Docs: S3 Batch Operation to Re-Encrypt

Kalman Filtering in SQL

Danny Reed — Tue, 20 Apr 2021 19:51:10 +0000

Kalman who?

Have you ever used a Kalman filter? My first experience with Kalman filtering was in smoothing motion sensor readings for a homemade Segway I built back in college. Many years ago it was used in the Apollo missions (ok, they win on coolness...). It's job is simple -- take noisy data and smooth it out a bit.

At work, I needed to smooth some latitude/longitude values to help make navigation paths be smoother. Normally, I'd just npm i kalmanjs and be off to the races, but this particular data pipeline presented a challenge.

Challenge

I apply several data-processing operations to the data I'm working with, and several of those operations happen in the database because SQL is a great place to do set-based operations. These operations must take effect in a specific order in order for things to come out right, and I needed to stick the Kalman filtering operation right between two specific steps in my process. Both of those steps were done in SQL.

Option 1

Do the first few operations in SQL, then return results to the API layer (Lambda) to do the Kalman filtering (using Kalmanjs) and the remaining operations. This requires porting the rest of the steps from SQL to JS and doing them in the Lambda.

Advantage: Puts some CPU load on the Lambda instead of the DB, which is a good thing in my case.

Disadvantage: I have to translate several operations from SQL to JS, and some of them are set-based, which means they're better to do in SQL.

Option 2

Try to implement Kalman filtering in SQL. This option is tricky because there's practically no information out there on Kalman filtering in SQL.

Advantage: Preserve data pipeline and existing SQL implementations for subsequent operations.

Disadvantage: Requires writing a Kalman filter in SQL, which has some specific challenges we'll discuss later. Also, I don't do much fancy math, so understanding the way Kalman filtering actually works would be a challenge.

I chose option 2 because I just can't justify doing several operations in the "wrong" place just because that's where the Kalman implementations are available.

Translation

Before attempting the implementation, I took a good look at the code in KalmanJS and tried to understand it. Honestly the code looked pretty simple, and all I really had to do was change syntax around a bit. Here's a snippet of the translation:

Original JS implementation (KalmanJS)

/**
  * Filter a new value
  * @param  {Number} z Measurement
  * @param  {Number} u Control
  * @return {Number}
  */
  filter(z, u = 0) {

    if (isNaN(this.x)) {
      this.x = (1 / this.C) * z;
      this.cov = (1 / this.C) * this.Q * (1 / this.C);
    }
    else {

      // Compute prediction
      const predX = this.predict(u);
      const predCov = this.uncertainty();

      // Kalman gain
      const K = predCov * this.C * (1 / ((this.C * predCov * this.C) + this.Q));

      // Correction
      this.x = predX + K * (z - (this.C * predX));
      this.cov = predCov - (K * this.C * predCov);
    }

    return this.x;
  }

Essentially this just does basic ifs, and some math. In fact, I found it so clear that I decided I could probably just translate the code without even having to dive into understanding the math behind Kalman filtering.

SQL Translation:

IF (@x IS NULL)
BEGIN
    PRINT 'No existing state; proceeding without it';
    SET @x = 1 / @C * @z;
    SET @cov = 1 / @C * @Q * (1 / @C);
END
ELSE
BEGIN
    SET @predX = @A * @x + @B * @u;
    SET @predCov = @A * @cov * @A + @R;

    SET @K = @predCov * @C * (1 / (@C * @predCov * @C + @Q));

    SET @x = @predX + @K * (@z - @C * @predX);
    SET @cov = @predCov - @K * @C * @predCov;

END

I did choose to change uncertainty() and predict() functions to inline formulas, just to minimize the need to create function declarations in SQL. Other than that, you can tell they're pretty much the same.

Speed Bump

This is where I hit a snag with the translation. Kalman works by looking at statistical properties the previous values in order to predict and smooth the next value. This means that Kalman needs to keep track of the state of the filter. In OOP you can just keep an instance of a class around in memory and use instance properties to keep track of state. In SQL, once the invocation completes, nothing is stored in memory.

Solution

Since the only obvious way to store things in SQL is in a table, I decided to create a KalmanState table in my database. I read through the JS code again to determine which values were actually stored and used in the next calculation. From what I gathered, it appeared that all it needs to store are these variables:

x, cov, predX, predCov, K

Now it needs to store one set of these variables for each instance of the filter, so I made a simple table that looks like this:

id          x                      cov                    predX                  predCov                K                      identifier
----------- ---------------------- ---------------------- ---------------------- ---------------------- ---------------------- ----------------
43          26.9861111111111       0.618055555555556      26.9636363636364       1.61818181818182       0.618055555555555      testingLat
44          21.0416666666667       0.618055555555556      21.1090909090909       1.61818181818182       0.618055555555555      testingLon

The last column is "identifier" which is just a string that you can use to name the instance. In my case, I'm using one filter for each of several MAC addresses of devices I'm using. So for me, I can just pass in the MAC address of the device as the identifier. This helps the SPROC retrieve the proper state when it's time to filter a new value.

Usage

Here's how I integrated this into my query.

SELECT TOP 1
    @LatIdentifier = CONCAT(@DeviceId, '-lat')
    ,@LonIdentifier = CONCAT(@DeviceId, '-lon')
    ,@rawLat = Latitude
    ,@rawLon = Longitude
    FROM #temp;

EXEC sp_kalman @LatIdentifier, @rawLat, @predictedLat OUTPUT;
EXEC sp_kalman @LonIdentifier, @rawLon, @predictedLon OUTPUT;

-- Overwrite the unfiltered lat/lon values in the temp table
UPDATE #temp 
    SET
        Latitude = @predictedLat
        ,Longitude = @predictedLon;

Note that this is part of a much larger query, there were contextual reasons to use the temp table for this purpose, and that there is only ever one row in that table when this is executed.

Problems/Caveats

I'm not writing this to say that I've come up with some perfect solution...in fact this solution has some serious challenges.

I was forced to make it a SPROC. I attempted to turn my code into a UDF. Functions are nice because you can use them right in a SELECT or even a JOIN. Sadly (but sensibly) functions are not allowed to alter data. As we discussed earlier, the Kalman filter must alter data in order to save its state. Now to integrate it with my query, I can't just flow it right in with the SELECTs but I have to EXEC the SPROC with parameters and then retrieve the output into a variable. This is just a clunky experience that makes for uglier and less readable code, in my opinion.
You can't use it with ISOLATION LEVEL SNAPSHOT. This is true for the same reason as #1. You can't alter data in a SNAPSHOT-isolated transaction. This makes sense too, but it's annoying! I had to change up how my transaction worked in order to exclude the EXEC calls from it. In my case it was possible to side-step this limitation, but in other cases it may not be!
You have to clean up after yourself By this I mean that there's no mechanism whereby this will ever DELETE records from the KalmanState table, which could allow for uncontrolled growth dependent on the application. In my case, I had to modify my API layer to perform cleanup when a filter is no longer useful to the system.

Should you use this?

It seems to me that the vast majority of use cases for Kalman filtering can be accomplished without doing the math in the DB. It's just easier all around :)

If you find yourself in a position like mine, where doing it in the DB really was the best option, then I hope this gives you a leg up on accomplishing that.

Next Steps

The author of the KalmanJS library (who does some cool stuff -- check out his work) has a contrib filter where he invites translations to other languages. You can find this implementation there now.

Also check out his original blog post about the development of KalmanJS here.

If you can think of improvements, please feel free to have a go at it!

(Cover image sourced from here and used with permission)

Serverless - A Primer

Danny Reed — Tue, 20 Apr 2021 02:39:28 +0000

What is "serverless"

I know serverless isn't exactly a new word any more. The first truly abstract serverless offerings came around more than 10 years ago, which is an eternity in the tech world. The reason I want to write about it is that even though it has been around a while, many folks (especially large, established corporations) who could benefit from it suffer from inertia of mindset, commitment to previous investments, and a lack of serverless-ready skillsets. Perhaps accurately understanding what it is will help set a course for progress.

TL;DR

Serverless is more than just a buzzword applied to a product by the marketing team. It's an architecture paradigm, a mindset, a pricing model, and a commitment to an additional layer of abstraction that keeps your time focused on what makes your project special.

Adding to the confusion, there's also a company called Serverless Inc, offering tools to help you build...serverless apps. We won't dive into that today.

Serverless: An Architecture Paradigm

When designing an architecture for a serverless system, one has a different set of building blocks at her/his disposal. Instead of thinking of CPU and RAM capacities, one thinks in more abstract terms like services, protocols, and data pipelines.

A serverless architecture is one that can scale with little or no intervention because it is built on "elastic" services that scale horizontally as the load increases.

A serverless architecture will avoid stateful transactions where possible. Statelessness lends itself to horizontal scaling, scale-to-zero services that shut down entirely when not needed, and typically result in information being stored in nonvolatile, highly durable places instead of memory.

A serverless architecture will decouple wherever possible. Technically decoupling is a useful feature in any type of architecture, but it's especially beneficial in serverless. Cloud services designed for use in serverless architectures are built to make decoupling easier than ever before.

Here's a great set of serverless reference architectures complete with diagrams that I found very useful.

Serverless: A Mindset

I like to think of serverless as a mindset. Some projects I've worked on start with someone asking something like,

"Ok, so how big of a server do we want and where should we host it."

Someone with a serverless mindset would first ask,

"Can this be done without a server?"

That seems like a simple change, but the payoff can be impressive. I haven't researched the economics of it, but it seems to me that cloud providers charge quite a bit for a VM. Moving to cloud VM's has benefits, but it's very expensive, too!

Ben Kehoe is one of the top serverless gurus out there, and during an interview on the Real World Serverless with theburningmonk podcast (#15), he mentioned the serverless mindset:

Interviewer: What advice would you give to people just looking to start their serverless journey today? Anything that would help them get started quickly and avoid common mistakes?

Ben Keohe: The most important bit is adopting the serverless mindset, which is something you can do even before you can use the technologies that we refer to as serverless. It's about wanting to own less technology to deliver customer value better.

Serverless: A Pricing Model

When you're looking at pricing information for serverless services, you'll probably see prices listed in hundredths or thousandths of a penny. Most serverless offerings have a per-invocation or per-use pricing model. This means that if nobody is using your app, the cost to operate it goes down. If you have a big surge in load, you'll pay for exactly enough horsepower to tackle it.

This pricing model helps you pay for the things that are actually giving you value. A beefy server with no workload applied to it is giving you negative value. It's costing you money without performing any tasks. A serverless function, by comparison, will only wake up when needed, and then evaporate when it's done. You get charged for the milliseconds/seconds that it was awake.

This is a big benefit for startups who need to keep costs low during development and testing, but need to scale quickly as their apps gain traction.

Serverless: A Commitment to Abstraction

One great way to focus on the important things is to alleviate the need to give your attention to other things. This is an unsung strength of serverless! While you may be capable of managing your own servers, that's not the mission of your app! Having spent an hour managing server OS patches will not result in your app being fundamentally better at what it does. Having that hour to start development of that community-suggested feature on your backlog WILL make your app fundamentally better.

In economics this dynamic is called "opportunity cost" and we should be careful to add this to the equation when evaluating the server-based vs. serverless decision.

Technology has long been a game of abstraction. Serverless computing is the next major step in getting us further away from 1's and 0's and closer to the ideas and concepts that we use to build amazing things.

Downsides?

While serverless is awesome, it's not quite a silver bullet. Here are a couple challenges:

Portability: If you built your system in GCP and then one day are required to port it to Azure, it's not just going to be spinning up a new server and copying over your configs. It's going to involve understanding the nuanced differences between GCP's services and Azure's services, and you'll likely spend quite a bit of time adapting your code and architecture to work well with the new cloud provider.
Control: When using a service, you're at the mercy of the owner of that service. While many times a cloud provider will make positive changes to their services to keep the it modern and secure, sometimes your schedule can be clobbered by a notice that says, "you have a resource using [protocol/runtime/service version] that will be retired on [date]." In my experience cloud providers are judicious about the kinds of changes they make and allow for backwards compatibility where reasonable. Notice is typically given well in advance of anything changing. The bottom line remains, however, that they are in control of their service, and your system must be kept in compatibility with it.

Summary

Serverless is important because it presents new opportunities for us to be effective, efficient, and excellent in our work. The impact of serverless can be seen in the stories of many successful businesses, but some of the most dramatic impacts are not yet realized.

I hope this fairly informal post has helped shape your perception of serverless systems and their characteristics, and that you'll be able to use this exciting set of tools to build something amazing!