DEV Community: Jędrzej Szczepaniak

You should want to write unit tests and here is why

Jędrzej Szczepaniak — Tue, 20 Jul 2021 12:30:49 +0000

Main consumer of the unit test suite is a developer himself. That's right! You are writing these tests for yourself. To be more precise you are writing them for your-future-self. It's one of these things that pays dividend later down the road, and here is why.

At some point you will get back to the piece of code you committed and you will scratch your head, thinking what this thing even does. You want to be able to go to the test suite and find out what it was doing.
If your next task is a bug to be fixed - you don't want to commit the fix just for the next person to remove your change because it seems to be nonsense. You want to put unit test that proves the bug is fixed, because you don't want to spend time on this bug again. To quote Google's Beyoncé Rule "If you liked it, then you shoulda put a test on it".
There is good chance that your day-to-day work requires a lot of thinking. There is also good chance that you spend a lot of time reading code. What would you rather read, a piece of code where you have no idea of what set of inputs/outputs are possible, or would you like to read executable spec that tells you exactly what happens under what circumstances? This executable specification is called unit test.
Occasionally your users might start to complain about slowness of your code, or SRE team can come to you and say that you use way too much RAM. This means that you need to get your hands dirty, play with that profiler and make your feature performing better. This is when you will thank yourself that your feature is tested because you can improve (refactor) your code easily - as long as tests pass - you are sure that you didn't break anything with your performance improvement.
When you are writing a piece of library - you want to interact with what you've done. If your API is easy to use - you will taste it in your tests. If it's hard to use it - you will feel the pain just as the user of your API will feel the pain. You want to have this microscope and look at your code from outside world. As it turns out - unit test is a terrific tool for that.
From time to time someone might ask you - hey, what this piece of code is doing? Being able to answer - "hey, just look at the unit tests - they will tell you what is what" is an asset you want to have.
Ability to run the program is very cool. Let's say you want to run a piece of code that lives deep in your repository. How do you do it? If your answer is - I am running the whole application, plug in the debugger (or print some values), and play with it from outside - you are in the disadvantage. Better way is to have a unit test that allows you to run any piece of the code you have in your repository without providing any external dependencies. It saves your time. It's your ally in day to day work. If you need to click through the UI or send some messages to the queue just to execute a piece of code - you should know that there is a better way.
Not only you are a programmer, but you are code reviewer from time to time as well, right? If this is the case, I bet that instead of guessing what are possible inputs and edge cases of given function it's better to read the test suite, and learn more about the piece of code you are reviewing. Not only that, tests allow you to better understand what is the intention of given piece of code. And if you know what is the intention you can review not only correctness of the code but you are able to judge whether things are named properly - which is very important given that this code will be read by many people in the future.
On top of that you are making sure that software you deliver does exactly what it should, and you can prove it easily, quickly and repeatedly.

Are there any more reasons to start writing unit tests?

Serverless and microservices are... pretty much the same?

Jędrzej Szczepaniak — Wed, 06 Jan 2021 11:43:29 +0000

Recently I was asked "Do you have experience with developing microservices?" Well, I don't have experience with microservices per se but I am doing serverless so that must be similar right? I asked myself "Why people do microservices?" We can enumarate some of the reasons.

Each team can be responsible for their microservice(s) end to end.
You can deploy and scale different pieces of the application separately.
Your application can be more fault tolerant. Ideally, breaking one microservice doesn't mean that your whole application is down.

What are the downsides then?

When different pieces of code become services that are separately developed and deployed, you need to be extra careful when dividing your domain so that each service has responsibility clearly set.
Network calls are slower than function calls and they can randomly fail.
Debugging becomes a problem, you need to have things like tracing.

This is the view of the person that didn't work with microservices ever. It's based on what I read, what I think and on presentations I watched.

How serverless fits in?

What is the serverless application in that case?

This is serverless application. Notice the arrow direction. It points from serverless to the application. Serverless depends on the application. What is the application? This is you differentiator. It's your business logic. Your modules live there. I am working on connecting employers with candidates right now so my domain contains modules like job recommendations, ability to apply for a job and candidate profiles.

If this is the case, what is this serverless thing?

It's your deployment platform. I am developing on AWS, so endpoints to my domain are AWS Lambda functions. They can receive HTTP requests, events from the queue, messages from a topic, and many different AWS-specific events.

How this tech fits in the monolith/microservices battle?

With microservices - each team can be responsible for their microservice(s) end to end.

With serverless many teams can be responsible for different serverless applications.

With microservices - you can deploy and scale different pieces of the application separately.

That's true for sure with serverless. Lambda functions scale up and down with incoming traffic, but it won't scale infinitely. If you are starting to hit cloud vendor limits - you should be lucky because your business is getting noticed. Also you should think then about using something else for your very popular endpoints.

With microservices - your application can be more fault tolerant. Ideally, breaking one microservice doesn't mean that your whole application is down.

Definitely - one failing lambda doesn't mean that the rest of the application is unavailable.

With microservices - When different pieces of code become services that are separately developed and deployed, you need to be extra careful when dividing your domain so that each service has responsibility clearly set.

Here I see the real power of the serverless. Software applications are living creatures. Developers try to foresee the future as much as they can and they try to make assumptions of what future needs they will need to satisfy. Product people on the other hand have different agenda. They try to figure out what will work for the customers. They pivot all the time. This is good because it means business is going forward. But for developers it's just nightmare. You try to divide your domains into modules, bounded contexts, subdomains and everything changes all the time. The serverless gives you advantage here. You can start small, couple of lambda functions calling the code they interested in. But as the requirements change, you can alter your domain code as well until it evolves into something mature. With microservices you can end up with the situation where you already divided your domains and different microservices are responsible for different things but when new requirement or feature comes in - you think - where do I put it? In the service A? In the service B? Maybe I should create service C? You need to carefully calculate what other modules will be called. If they live in different services - you might end up with too many network calls and it might not be as reliable as you would like to.

I believe that with serverless live is easier. You develop your application. You add features to it and you are careful about it. You studied "A Philosophy of Software Design" by John Ousterhout and you know how to design beutiful, functional, deep modules but when requirements change you can painlessly change modules as well. Later you create endpoints to the domain code by attaching lambda functions to it. Ideal situation is when you have a software module that is called by couple of lambdas and they form unity. They are separated. Later you can do whatever you want - you can take these lambdas and the domain code and pull them out of the serverless application and even create microservice from it if you want to!

My point here is that with serverless there is no struggle as to when to move from monolith to microservices. Serverless allows your software to evolve in direction designated by product.

Is it something that rarely changes and needs to handle huge load? Pull it out and create microservice.

Is it something that often changes and you need to tweak it frequently? Develop it further with serverless.

My real point is this: Serverless supports evolution.

With microservices - network calls are slower than function calls and they can randomly fail.

I believe I touched on this subject before, if you want to - you call the function from other module.

With microservices - Debugging becomes a problem, you need to have things like tracing.

With serverless you need to have tracing too. When things become really distributed it is invaluable tool to figure what is wrong.

Let's go back to the original question.

Can you develop microservices?

I believe that everyone who once developed serverless application will be able to develop any microservices because principles you need to follow are similar, yet serverless gives a little bit more elasticity.

Let's go back to the title.

Serverless and microservices are... pretty much the same?

They aren't the same, they are similar. There is place for both solutions. If your doing something completely new and you don't know where it will go - defintely choose serverless and benefit from the elasticity it provides. If you exactly know what you want and you will handle really huge traffic - create microservice and scale it infinitely on the platform of your choice.

DynamoDB with Go #12 - Condition on other item from item collection

Jędrzej Szczepaniak — Tue, 24 Nov 2020 08:22:34 +0000

Item collection is a set of attributes that share the same Partition Key. A common pattern is to model one to many relationships with item collections where PK represents commonality among items and SK distinguishes types of items.

As an example we can take information about user and orders placed by the user.

PK	SK
1234	USERINFO
1234	ORDER#2017-03-04 00:00:00 +0000 UTC
1234	ORDER#2017-02-04 00:00:00 +0000 UTC

This is Single Table Design. We put different types of things into single table so that we can satisfy access patterns with minimal interactions with DynamoDB table. In this example we can display on a single page information about user with the latest orders while doing single query. PK represents ID of the user, and SK depending on the case represents basic user information or the order.

Now, that we've established that querying data gets simpler with Single Table Design, let us consider inserting data.

We need to be able to register a user. This use case is fairly simple. We need to make sure, that user with given ID doesn't exist in our system yet. The tool to do just that is to use expression like this one.

expr, err := expression.NewBuilder().
  WithCondition(expression.AttributeNotExists(expression.Name("pk"))).
  Build()

When we have a user, we can place orders. Here is the thing. I would like to be able to place orders only for existing users. I would like to do something so that test passes. We want to get error when placing an order for the user that doesn't exist yet.

func TestInsertingOrderFailsBecauseUserDoesNotExist(t *testing.T) {
  ctx := context.Background()
  tableName := "ATable"
  db, cleanup := dynamo.SetupTable(t, ctx, tableName, "./template.yml")
  defer cleanup()

  _, err := db.PutItemWithContext(ctx, &dynamodb.PutItemInput{
    Item: map[string]*dynamodb.AttributeValue{
      "pk": dynamo.StringAttr("1234"),
      "sk": dynamo.StringAttr("ORDER#2017-03-04 00:00:00 +0000 UTC"),
    },
    TableName: aws.String(tableName),
  })
  assert.Error(t, err)
}

Don't consider this a real world unit test. I am using _test.go file merely for executing queries. Anyway, this test fails. There is no error. Can we write simple condition that makes sure that user for which we want to place order exists?

Unfortunately, we can't do this easily. Writing conditions that relate to other items in the Item Collection is impossible. In order to make sure that user exists while introducing new order we need to use the transaction.

func TestInsertingOrderFailsBecauseUserDoesNotExist(t *testing.T) {
  ctx := context.Background()
  tableName := "ATable"
  db, cleanup := dynamo.SetupTable(t, ctx, tableName, "./template.yml")
  defer cleanup()

  expr, err := expression.NewBuilder().
    WithCondition(expression.AttributeExists(expression.Name("pk"))).
    WithUpdate(expression.Add(expression.Name("orders_count"), expression.Value(1))).
    Build()
  assert.NoError(t, err)

  _, err = db.TransactWriteItemsWithContext(ctx, &dynamodb.TransactWriteItemsInput{
    TransactItems: []*dynamodb.TransactWriteItem{
      {
        Put: &dynamodb.Put{
          Item: map[string]*dynamodb.AttributeValue{
            "pk": {S: aws.String("1234")},
            "sk": {S: aws.String("ORDER#2017-03-04 00:00:00 +0000 UTC")},
          },
          TableName: aws.String(tableName),
        },
      },
      {
        Update: &dynamodb.Update{
          ConditionExpression:       expr.Condition(),
          ExpressionAttributeValues: expr.Values(),
          ExpressionAttributeNames:  expr.Names(),
          UpdateExpression:          expr.Update(),
          Key: map[string]*dynamodb.AttributeValue{
            "pk": {S: aws.String("1234")},
            "sk": {S: aws.String("USERINFO")},
          },
          TableName: aws.String(tableName),
        },
      },
    },
  })

  assert.Error(t, err)
  _, ok := err.(*dynamodb.TransactionCanceledException)
  assert.True(t, ok)
}

Transaction has two items. First one (Put) is what we really want to do here which is placing new order. Additionally, there is an update for item that represents information about the user. I did it so that I can put condition on that update operation. The condition says exactly what we wanted. Transaction will succeed only for existing user.

You can notice however that I am increasing orders_count which wasn't talked about as a requirement for this example. It doesn't really matter what gets updated, but you cannot perform empty update. We could choose to update some helper attribute to "" but I wanted to make it useful. We are killing two birds with one stone. We get orders count for free.

Takeaway from this episode is the following one.

If you want to base the condition of your operation on other item - you need to use transaction.

DynamoDB with Go #11 - Expressions API

Jędrzej Szczepaniak — Tue, 17 Nov 2020 10:58:12 +0000

I am advocating for using the expression API from the first episode of the DynamoDB with Go. In this episode I would like to show you why I enjoy using them so much. We are going to examine three examples. Each of them is going to be implemented using plain text expressions and with expressions API. I hope that after examining these three comparisons you'll be convinced that expression API is the way to go.

Example #1 - Get item collection

When table uses Composite Primary Key (meaning that it has both PK and SK), an item collection is bunch of items that share the same Partition Key. In order to fetch the item collection we need to use DynamoDB Query with Key Condition. Let's start with a test.

t.Run("v1 - get whole item collection", func(t *testing.T) {
  ctx := context.Background()
  tableName := "ATable"
  db, cleanup := dynamo.SetupTable(t, ctx, tableName, "./template.yml")
  defer cleanup()
  insert(ctx, db, tableName, item1, item2)

  collection, err := GetItemCollectionV1(ctx, db, tableName, "1")
  assert.NoError(t, err)
  assert.Subset(t, collection, []Item{item1, item2})
  assert.Len(t, collection, 2)
})

I am using insert helper that sets up the stage for me. We are going to begin each test with this one.

I would like to draw your attention to the fact that GetItemCollection function has V1 prefix. We also have version 2 of the function. For this episode I'll set a convention where all functions that use traditional string based expressions end with V1 and are located in strings_expressions.go file. Functions that use expression API can be found in api_expressions.go and end with V2 prefix.

Let's compare implementations.

out, err := db.QueryWithContext(ctx, &dynamodb.QueryInput{
  KeyConditionExpression: aws.String("#key = :value"),
  ExpressionAttributeNames: map[string]*string{
    "#key": aws.String("pk"),
  },
  ExpressionAttributeValues: map[string]*dynamodb.AttributeValue{
    ":value": {S: aws.String(pk)},
  },
  TableName: aws.String(table),
})

expr, err := expression.NewBuilder().
  WithKeyCondition(expression.KeyEqual(expression.Key("pk"), expression.Value(pk))).
  Build()

out, err := db.QueryWithContext(ctx, &dynamodb.QueryInput{
  KeyConditionExpression:    expr.KeyCondition(),
  ExpressionAttributeNames:  expr.Names(),
  ExpressionAttributeValues: expr.Values(),
  TableName:                 aws.String(table),
})

Certainly second version hides from us details that we need to think about when constructing expressions ourselves.

What we really want to do in this example is to construct condition that checks equality. Attribute "pk" should be equal to the variable pk. Doing it directly is risky because we cannot use reserved keywords of the DynamoDB. This is why we create aliases #key and :value which are later mapped to real values, #key becomes "pk", and :value becomes contents of variable pk. When using expressions API, we don't need to build these by hand, the API is doing it automatically. Let's examine the expression that is built by the expression API.

(expression.Expression) {
 expressionMap: (map[expression.expressionType]string) (len=1) {
  (expression.expressionType) (len=12) "keyCondition": (string) (len=7) "#0 = :0"
 },
 namesMap: (map[string]*string) (len=1) {
  (string) (len=2) "#0": (*string)(0xc0002b63b0)((len=2) "pk")
 },
 valuesMap: (map[string]*dynamodb.AttributeValue) (len=1) {
  (string) (len=2) ":0": (*dynamodb.AttributeValue)(0xc0002768c0)({
  S: "1"
})
 }
}

It is really similar. The only difference is that expression API creates aliases meaningless for humans like #0 and :0. Note that alias for key has to start with : and alias for value with #.

Further parts of both versions of implementations are identical - so I am skipping them here. You can examine them in the repository. The same goes for second and third example. I am going to show only expressions - as they are the only parts that differ.

Example #2 - "update A and unset B but only if B is set to `baz`"

Taken out of context doesn't really make sens, but this scenario is inspired by real life case. For given item that has attributes A and B I want to update A to some value and unset B but only if B is set to baz.

Let's jump into the test.

t.Run("v1 - update A and unset B but only if B is set to `baz`", func(t *testing.T) {
  ctx := context.Background()
  tableName := "ATable"
  db, cleanup := dynamo.SetupTable(t, ctx, tableName, "./template.yml")
  defer cleanup()
  insert(ctx, db, tableName, item1, item2)

  updated, err := UpdateAWhenBAndUnsetBV1(ctx, db, tableName, Key{PK: "1", SK: "1"}, "newA", "baz")
  if assert.Error(t, err) {
    assert.Equal(t, "b is not baz, aborting update", err.Error())
  }
  assert.Empty(t, updated)

  updated, err = UpdateAWhenBAndUnsetBV1(ctx, db, tableName, Key{PK: "1", SK: "2"}, "newA", "baz")
  assert.NoError(t, err)
  assert.Equal(t, "newA", updated.A)
  assert.Empty(t, updated.B)
})

It turns out that item1 has B set to bar thus it's not updated, item2 on the other hand has B set to baz and it is updated.

Let's compare implementations.

out, err := db.UpdateItemWithContext(ctx, &dynamodb.UpdateItemInput{
  ConditionExpression: aws.String("#b = :b"),
  ExpressionAttributeNames: map[string]*string{
    "#b": aws.String("b"),
    "#a": aws.String("a"),
  },
  ExpressionAttributeValues: map[string]*dynamodb.AttributeValue{
    ":b": {S: aws.String(whenB)},
    ":a": {S: aws.String(newA)},
  },
  Key:              marshaledKey,
  ReturnValues:     aws.String("ALL_NEW"),
  TableName:        aws.String(table),
  UpdateExpression: aws.String("REMOVE #b SET #a = :a"),
})

vs.

expr, err := expression.NewBuilder().
  WithCondition(expression.Equal(expression.Name("b"), expression.Value(whenB))).
  WithUpdate(expression.
    Set(expression.Name("a"), expression.Value(newA)).
    Remove(expression.Name("b"))).
  Build()

out, err := db.UpdateItemWithContext(ctx, &dynamodb.UpdateItemInput{
  ConditionExpression:       expr.Condition(),
  ExpressionAttributeNames:  expr.Names(),
  ExpressionAttributeValues: expr.Values(),
  UpdateExpression:          expr.Update(),
  Key:                       marshaledKey,
  ReturnValues:              aws.String("ALL_NEW"),
  TableName:                 aws.String(table),
})

I think that second approach is better because it is less error prone and gives more clarity (which is 100% subjective by the way). This example is fairly simple. The more fields you want to update, and more complex your conditions are - the more you'll appreciate type systems of Golang when constructing expressions with the expressions API.

On a side note, notice how expression builder allows you to mix both condition expressions and update expressions.

Example #3 - Put if doesn't exist

By default PutItem operation overwrites an item if it already exists. It is very common to make sure to not overwrite that, and the tool to do that is the condition expression. Here is the test.

t.Run("v1 - put if doesn't exist", func(t *testing.T) {
  ctx := context.Background()
  tableName := "ATable"
  db, cleanup := dynamo.SetupTable(t, ctx, tableName, "./template.yml")
  defer cleanup()
  insert(ctx, db, tableName, item1, item2)

  err := PutIfNotExistsV1(ctx, db, tableName, Key{PK: "1", SK: "2"})
  if assert.Error(t, err) {
    assert.Equal(t, "Item with this Key already exists", err.Error())
  }

  err = PutIfNotExistsV1(ctx, db, tableName, Key{PK: "10", SK: "20"})
  assert.NoError(t, err)
})

Item with PK=1 and SK=2 was already inserted to the DynamoDB, thus cannot be inserted again and test fails. For PK=10 and SK=20 operation succeeds. Let's compare implementations.

_, err = db.PutItemWithContext(ctx, &dynamodb.PutItemInput{
  ConditionExpression: aws.String("attribute_not_exists(#pk)"),
  ExpressionAttributeNames: map[string]*string{
    "#pk": aws.String("pk"),
   },
   Item:      marshaledKey,
   TableName: aws.String(table),
 })

vs.

expr, err := expression.NewBuilder().
  WithCondition(expression.AttributeNotExists(expression.Name("pk"))).
  Build()

_, err = db.PutItemWithContext(ctx, &dynamodb.PutItemInput{
  ConditionExpression:      expr.Condition(),
  ExpressionAttributeNames: expr.Names(),
  Item:                     marshaledKey,
  TableName:                aws.String(table),
})

If I am being honest - I kind of like first version. It is concise, you immediately know what is going. The thing is that I think that conventions matter, and I'd rather stick with one way of doing things. Other thing is that even though attribute_not_exists(#pk) is cute - it is so simple to make mistake - and you don't have any autocompletion when writing it by hand.

Summary

I think that ability to write expressions by hand matters. I believe that this ability helps along the way when you're trying to figure out when your query breaks. Having said that - when you know what is what - I think in day to day work it is better to stick with expression API as it very convenient and less error prone than plain text expressions.

DynamoDB with Go #10 - Gotcha with empty slices

Jędrzej Szczepaniak — Tue, 10 Nov 2020 14:44:48 +0000

This time we are going to cover the gotcha of AWS SDK for Go that has bitten me recently. It's about storing empty slices in the DynamoDB.

Different flavours of empty slices

First of all - what is the slice? It is three things:

pointer to the underlying data,
length of the slice,
capacity of the slice.

These three things make a slice. If an empty slice means the slice with length of 0 then we can create such entity in
at least three different ways.

1. Zero value slice

Zero value of a slice is the nil. It has capacity of 0, length of 0 and no underlying array.

var stringSlice []string

2. Short declaration

Short declaration produces slice with capacity of 0, length of 0 and pointer to the underlying data.

stringSlice := []string{}

3. Make

Make function similarly to short declaration produces slice with capacity of 0, length of 0
and pointer to the underlying data.

stringSlice := make([]string, 0)

Each method produces slice for which condition len(stringSlice) == 0 holds true. However, condition stringSlice == nil is true only for zero value slice. It is in the spirit of Go for zero value to be useful. Indeed, zero value slice is useful. Sometimes however distinction between nil slice and non-nil empty slice can be handy. Let's imagine a case where you want to collect information about professional experience from the user. Professional experience can be modelled as a slice of workplaces. You might want to distinguish three different situations.

user didn't provide professional experience yet,
user marked I do not have any professional experience yet checkbox,
user provided a list of his past professional endeavors.

Modelling the difference between cases 1. and 2. can be done with distinction of nil slice and non-nil empty slice. Luckily for us when marshaling into JSON - Go distinguishes between these cases.

var stringSlice []string
json.Marshal(stringSlice) // Marshals into `null`

stringSlice := []string{}
json.Marshal(stringSlice) // Marshals into `[]`

Empty slices in the DynamoDB

What happens when we try to save an item into the DynamoDB when one of the attributes is empty slice?

There is no problem with zero value slice. You can save nil slice into DynamoDB and after retrieving it, it will be unmarshalled as a nil slice. A problem occurs when using empty non-nil slices. Look at the test.

t.Run("regular way", func(t *testing.T) {
  t.Skip("this test fails")
  attrs, err := dynamodbattribute.Marshal([]string{})
  assert.NoError(t, err)

  var s []string
  err = dynamodbattribute.Unmarshal(attrs, &s)
  assert.NoError(t, err)

  assert.NotNil(t, s) // fails
  assert.Len(t, s, 0)
})

This failing test shows exactly what is the problem. I mean the problem isn't with the DynamoDB, but rather with how AWS SDK for Go treats slices. The DynamoDB is 100% capable of distinguishing between empty lists and NULL values. In order to see that let's look at attrsvariable from the example above.

(*dynamodb.AttributeValue)(0xc0000d5ea0)({
  NULL: true
})

This is what dynamodbattribute.Marshal function did to non-nil empty slice. It changed into NULL. Actually as you can see NULL is a bool field on dynamodb.AttributeValue type. This is just how AttributeValue represents something that is NULL. Let's keep that in mind.

What we really want to do if we care about distinction between nil slice and non-nil empty slice is to use the custom encoder and decoder and set the EnableEmptyCollections option that will preserve empty list.

t.Run("new way", func(t *testing.T) {
  e := dynamodbattribute.NewEncoder(func(e *dynamodbattribute.Encoder) {
    e.EnableEmptyCollections = true
  })
  attrs, err := e.Encode([]string{})
  assert.NoError(t, err)

  var s []string
  d := dynamodbattribute.NewDecoder(func(d *dynamodbattribute.Decoder) {
    d.EnableEmptyCollections = true
  })
  err = d.Decode(attrs, &s)
  assert.NoError(t, err)
  assert.NotNil(t, s)
  assert.Len(t, s, 0)
})

This test passes. Let's see how attrs variable looks like.

(*dynamodb.AttributeValue)(0xc00024c140)({
  L: []
})

This is truly empty list. This is what we want! Do we have a success? Not yet...

The dynamodbattribute package is used to transform application types into attribute values - which are types that the DynamoDB understands. It is used when performing for example PutItem operation or GetItem operation.

But we have also UpdateItem operations and typically we construct them with expression API. Let's look at the example.

expr, err := expression.NewBuilder().
  WithUpdate(expression.Set(expression.Name("attr_name"), expression.Value([]string{}))).
  Build()

This piece of code is an expresion that should update attr_name to the value of []string{} which is empty non-nil slice. Let's print that expression and observe what are we dealing with.

(expression.Expression) {
 expressionMap: (map[expression.expressionType]string) (len=1) {
  (expression.expressionType) (len=6) "update": (string) (len=12) "SET #0 = :0\n"
 },
 namesMap: (map[string]*string) (len=1) {
  (string) (len=2) "#0": (*string)(0xc000051990)((len=3) "123")
 },
 valuesMap: (map[string]*dynamodb.AttributeValue) (len=1) {
  (string) (len=2) ":0": (*dynamodb.AttributeValue)(0xc0002245a0)({
  NULL: true
})
 }
}

This is how update expression looks under the hood. We have expressionMap, namesMap and valuesMap parameters. Expression itself is SET #0 = :0. Path #0 is substituted with "123" and ":0" is substituted with {NULL: true}. We never really talked about how AWS SDK encodes expressions into the DynamoDB format because we have the expression API, and we didn't need to think about actual representation. At least until now. As you can see, even though we used expression.Value([]string{}) it was transformed into the DynamoDB NULL value which isn't very good because we didn't want to have nil slice there. As a matter of fact expression API will always transform any slice with length of 0 into NULL value. Are we helpless though? No, we are not. Instead of simply using expression.Value we need to construct that value by hand.

v := expression.Value((&dynamodb.AttributeValue{}).SetL([]*dynamodb.AttributeValue{}))
expr, _ := expression.NewBuilder().WithUpdate(expression.Set(expression.Name("attr_name"), v)).Build()

This isn't very pretty, but it works. Now we get:

(expression.Expression) {
 expressionMap: (map[expression.expressionType]string) (len=1) {
  (expression.expressionType) (len=6) "update": (string) (len=12) "SET #0 = :0\n"
 },
 namesMap: (map[string]*string) (len=1) {
  (string) (len=2) "#0": (*string)(0xc000051a20)((len=9) "attr_name")
 },
 valuesMap: (map[string]*dynamodb.AttributeValue) (len=1) {
  (string) (len=2) ":0": (*dynamodb.AttributeValue)(0xc000222780)({
  L: []
})
 }
}

The expression is aware of the empty list.

Summary

Sometimes we need to get our hands dirty and tinker with the internals of the API in order to get things done. But from now on - we'll have complete confidence that we know what is going on with our slices when we use them with Dynamo.

DynamoDB with Go #9 - Switching the toggle, toggling the switch

Jędrzej Szczepaniak — Tue, 03 Nov 2020 09:48:05 +0000

Here is the scenario for this episode. There is a toggle, and it can be switched on and off. Whenever toggle is switched, an event is published and we are in charge of consuming that event. Our task is to save switches of state of the toggle and to be able to tell whether it is on or off at the moment. Bad news is that events can arrive out of order - an old event can appear at any time, and we need to be able to reject it.

It may seem like fictitious example but is 100% real. I dealt with this kind of problem couple of weeks ago. Let me show you how it can be done with the DynamoDB!

We really have simple access pattern here. We want to obtain the latest state of the toggle. Additionally, we should be able to obtain log of switches that happened. Let's jump into the tests that define what we really want to do.

Test suite

t.Run("save toggle", func(t *testing.T) {
  tableName := "ToggleStateTable"
  db, cleanup := dynamo.SetupTable(t, ctx, tableName, "./template.yml")
  defer cleanup()

  toggle := NewToggle(db, tableName)
  err := toggle.Save(ctx, Switch{ID: "123", State: true, CreatedAt: time.Now()})
  assert.NoError(t, err)

  s, err := toggle.Latest(ctx, "123")
  assert.NoError(t, err)
  assert.Equal(t, s.State, true)
})

Can it be simpler? I don't think so - save it first - retrieve later. We want more however, next test proves that we can save many switches, and we can retrieve the latest one.

t.Run("save toggles, retrieve latest", func(t *testing.T) {
  tableName := "ToggleStateTable"
  db, cleanup := dynamo.SetupTable(t, ctx, tableName, "./template.yml")
  defer cleanup()

  toggle := NewToggle(db, tableName)
  now := time.Now()
  err := toggle.Save(ctx, Switch{ID: "123", State: true, CreatedAt: now})
  assert.NoError(t, err)

  err = toggle.Save(ctx, Switch{ID: "123", State: false, CreatedAt: now.Add(10 * time.Second)})
  assert.NoError(t, err)

  s, err := toggle.Latest(ctx, "123")
  assert.NoError(t, err)
  assert.Equal(t, s.State, false)
})

Last test shows that out of order events are not taken into account. If an old event arrives, it doesn't influence the latest state.

t.Run("drop out of order switch", func(t *testing.T) {
  tableName := "ToggleStateTable"
  db, cleanup := dynamo.SetupTable(t, ctx, tableName, "./template.yml")
  defer cleanup()

  toggle := NewToggle(db, tableName)
  now := time.Now()
  err := toggle.Save(ctx, Switch{ID: "123", State: true, CreatedAt: now})
  assert.NoError(t, err)

  err = toggle.Save(ctx, Switch{ID: "123", State: false, CreatedAt: now.Add(-10 * time.Second)})
  assert.NoError(t, err)

  s, err := toggle.Latest(ctx, "123")
  assert.NoError(t, err)
  assert.Equal(t, s.State, true)
})

This is all we want from the system. In the next section I am going to provide you with the solution for this problem that I came up with. If you are able to provide more elegant solution, more efficient one or maybe just ...better, feel free to share it with me! I am keen to look at the problem at a different angle!

A solution

Let's recap. Whenever new Switch is consumed by our function I want to append it at the end of my "log" but only if it is the newest one.

We do not have any table yet! We need to fix that. The table will have Partition Key called pk, and Sort Key called sk. As you saw in test cases the toggle has ID. Let's use that ID as the PK for every item connected with that toggle. In terms of SK - I want to use it twofold. First of all we will have log items. Each log item will have SK that starts with READ prefix followed by time of given switch. There will be additional special item with SK with constant "LATEST_SWITCH". This item will be used both to keep the order when writing a log item and when retrieving the latest switch.

Coding time

func (t *Toggle) Save(ctx context.Context, s Switch) error {
  item := s.asItem()

We'll start with keeping details unexported. Switch is a public thing, let's not pollute it with implementation details. The DynamoDB on the other hand needs to know how to deal with switches. We can also have different contents of sk. Because of that we have asItem() method.

func (s Switch) asLogItem() switchItem {
  return switchItem{
    PK:        s.ID,
    SK:        "SWITCH#" + s.CreatedAt.Format(time.RFC3339Nano),
    CreatedAt: s.CreatedAt,
    State:     s.State,
  }
}

What is switchItem? I am glad you asked. It knows how to marshal/unmarshal items.

type switchItem struct {
  PK string `dynamodbav:"pk"`
  SK string `dynamodbav:"sk"`

  State     bool      `dynamodbav:"state"`
  CreatedAt time.Time `dynamodbav:"created_at"`
}

Since we are speaking about marshaling we need to convert our data into the format that the DynamoDB understands.

attrs, err := dynamodbattribute.MarshalMap(item)

Next thing is the most complicated expression we've ever seen in the DynamoDB with Go because it combines condition and update.

expr, err := expression.NewBuilder().
  WithCondition(expression.LessThan(expression.Name("created_at"), expression.Value(item.CreatedAt))).
  WithUpdate(expression.
    Set(expression.Name("created_at"), expression.Value(item.CreatedAt)).
    Set(expression.Name("state"), expression.Value(item.State))).
  Build()

What is says is

Please update created_at field and state field but only if item we are inserting is younger than what is in the DynamoDB.

_, err = t.db.TransactWriteItemsWithContext(ctx, &dynamodb.TransactWriteItemsInput{
  TransactItems: []*dynamodb.TransactWriteItem{
    {
      Update: &dynamodb.Update{
        Key: map[string]*dynamodb.AttributeValue{
            "pk": {S: aws.String(item.PK)},
            "sk": {S: aws.String("LATEST_SWITCH")},
        },
        ExpressionAttributeNames:  expr.Names(),
        ExpressionAttributeValues: expr.Values(),
        ConditionExpression:       expr.Condition(),
        TableName:                 aws.String(t.table),
        UpdateExpression:          expr.Update(),
        ReturnValuesOnConditionCheckFailure: aws.String("ALL_OLD"),
      },
    },
    {
      Put: &dynamodb.Put{
        Item:      attrs,
        TableName: aws.String(t.table),
      },
    },
  },
})

We are using transaction because we want to append log entry (Put) and update the latest state of the toggle (Update) but only if the condition holds true for the latest state.

Let's put in english the condition mixed with this transaction. This is what is says:

I want to update the created_at and the state fields on the item that represents the latest state of the switch, but only if what I am trying to put into the DynamoDB is younger than what is already there. Additionally, I want to append the log item (because I want to have history of what happened to that toggle).

One more thing - ReturnValuesOnConditionCheckFailure parameter. It is crucial because it allows us to recognize what happened after transaction failure. It can fail for two reasons:

condition failed - we received out of order event - we need to discard it
database doesn't have any item with given pk - it is the first write for given partition key

if err == nil {
  return nil
}

We might not have an error at all - that means that we've just appended new log entry with an update to the latest state.

aerr, ok := err.(*dynamodb.TransactionCanceledException)
if !ok {
  return err
}

We might have an error that wasn't anticipated. In that case - application blows up, and we have an incident to handle.

if len(aerr.CancellationReasons[0].Item) > 0 {
  return nil
}

Based on that condition we can reason that we received an out of order event. Why? Because we filled ReturnValuesOnConditionCheckFailure parameter with ALL_OLD value. This means that when transaction fails the value of aerr.CancellationReasons[0].Item will be an item that was in the DynamoDB before our action. If that value is not empty this means that there is an item in the DynamoDB for given Partition Key. Since we have two reasons for transaction failure we can - by elimination - conclude that we'ce received an out of order event.

Out of order event isn't saved into the DynamoDB so we exit immediately. Now we need to handle the situation when transaction failed because it's the first time DynamoDB sees such Partition Key.

expr, err = expression.NewBuilder().
   WithCondition(
     expression.Not(expression.And(
       expression.Equal(expression.Name("pk"), expression.Value(item.PK)),
       expression.Equal(expression.Name("sk"), expression.Value("LATEST_SWITCH")),
     ))).Build()

This condition is responsible for making sure that the DynamoDB didn't save LATEST_SWITCH for our Partition Key yet. It's a possibility because between failed transaction and now there is time difference. Some other process could have saved such item in between.

We also need to create an item representing the latest state of the toggle.

latestAttrs, err := dynamodbattribute.MarshalMap(s.asLatestItem())

It's similar to asLogItem but it sets sk to LATEST_SWITCH. Next thing we do is yet another transaction.

_, err = t.db.TransactWriteItemsWithContext(ctx, &dynamodb.TransactWriteItemsInput{
  TransactItems: []*dynamodb.TransactWriteItem{
    {
      Put: &dynamodb.Put{
        ConditionExpression:       expr.Condition(),
        ExpressionAttributeNames:  expr.Names(),
        ExpressionAttributeValues: expr.Values(),
        Item:                      latestAttrs,
        TableName:                 aws.String(t.table),
      },
    },
    {
      Put: &dynamodb.Put{
        Item:      attrs,
        TableName: aws.String(t.table),
      },
    },
  },
})

What we want to achieve here is appending first log item and create the latest state of the toggle which holds the same information as the only log item we've saved.

What can happen now?

if err == nil {
  return nil
}

We might have no error - condition passed and both items were saved. This DynamoDB call can fail as well.

_, ok = err.(*dynamodb.TransactionCanceledException)
if !ok {
  return err
}

If it does - and the reason isn't transaction failure - something wrong happened, and we return with an error. If however transaction failed - first switch for the toggle was saved but not by us. What we can do is to call this whole function again. It is completely save and won't create an infinite loop because the Switch is either older or younger than what is saved in the Dynamo. If it is younger it will be saved. If it's older - it will be rejected.

return t.Save(ctx, s)

Very often - when writing is complex - reading must be trivial. This is the case in here!

func (t *Toggle) Latest(ctx context.Context, userID string) (Switch, error) {
out, err := t.db.GetItemWithContext(ctx, &dynamodb.GetItemInput{
  Key: map[string]*dynamodb.AttributeValue{
      "pk": {S: aws.String(userID)},
      "sk": {S: aws.String("LATEST_SWITCH")},
  },
  TableName: aws.String(t.table),
})
if err != nil {
  return Switch{}, err
}
if len(out.Item) == 0 {
  return Switch{}, errors.New("not found")
}

var item switchItem
err = dynamodbattribute.UnmarshalMap(out.Item, &item)
return item.asSwitch(), err
}

We retrieve the latest state of the toggle, unmarshal it and return it to the client!

Summary

Fairly easy test suite received very complex implementation. If you know how to satisfy the same test suite with simpler implementation - let me know - I would love to know how! Nevertheless, we learned two important pieces of DynamoDB API. First of all we know that expression API can combine both updates and conditions. Other than that - now we know that after transaction fails - we can use ReturnValuesOnConditionCheckFailure to obtain more insight on what really happened.

DynamoDB with Go #8 - Implement hierarchical data with Single Table Design

Jędrzej Szczepaniak — Tue, 27 Oct 2020 16:47:07 +0000

This episode is all about implementation. If you didn't read episode 7th - please do because we can't move forward without it. Assuming you've already read it, let's express all the use cases we have with help of unit tests.

Registering a sensor

Registering a sensor gives ability to record new readings of the sensor later on.

t.Run("register sensor, get sensor", func(t *testing.T) {
  tableName := "SensorsTable"
  db, cleanup := dynamo.SetupTable(t, ctx, tableName, "../template.yml")
  defer cleanup()
  manager := sensors.NewManager(db, tableName)

  err := manager.Register(ctx, sensor)
  assert.NoError(t, err)

  returned, err := manager.Get(ctx, "sensor-1")
  assert.NoError(t, err)
  assert.Equal(t, sensor, returned)
})

In order to verify that registration went well - sensor is retrieved afterwards. You might wonder - isn't unit testing about testing single units? Isn't the registration a unit? Well, registration doesn't really matter if we cannot do anything with it, and a unit is single behavior. At the beginning of the journey of testing my code it was a little strange for me. I was thinking - "but now such test has 2 reasons to break". Thinking in terms of above example - when registration fails and when retrieval fails. Later on, I came to the conclusion that there is superiority of such test over - for example checking what method was called underneath because it is better to test behavior than implementation. This is the key aspect of testing for me because change in implementation shouldn't break tests when behavior doesn't change. Now let's go back to the sensor business. Next behavior we want to cover is inability to register a sensor with the same ID twice.

Inability to register the same sensor twice

t.Run("do not allow to register many times", func(t *testing.T) {
  tableName := "SensorsTable"
  db, cleanup := dynamo.SetupTable(t, ctx, tableName, "../template.yml")
  defer cleanup()
  manager := sensors.NewManager(db, tableName)

  err := manager.Register(ctx, sensor)
  assert.NoError(t, err)

  err = manager.Register(ctx, sensor)
  assert.EqualError(t, err, "already registered")
})

When you try to register again you get slapped with an error. One thing I want to mention is sensor variable that I used in both above mentioned snippets. It's just exemplary sensor I've declared on top of the test suite. You can look it up in the repository if you want to.

Recording a sensor reading

Since we know how to register a sensor, let's record a reading of that sensor.

t.Run("save new reading", func(t *testing.T) {
  tableName := "SensorsTable"
  db, cleanup := dynamo.SetupTable(t, ctx, tableName, "../template.yml")
  defer cleanup()
  manager := sensors.NewManager(db, tableName)

  err := manager.Register(ctx, sensor)
  assert.NoError(t, err)

  err = manager.SaveReading(ctx, sensors.Reading{SensorID: "sensor-1", Value: "0.67", ReadAt: time.Now()})
  assert.NoError(t, err)

  _, latest, err := manager.LatestReadings(ctx, "sensor-1", 1)
  assert.NoError(t, err)
  assert.Equal(t, "0.67", latest[0].Value)
})

After saving new reading I need to verify somehow that it worked. In order to do that I am using LatestReadings method that provides me with one latest reading - which hopefully should be the one that was just saved.

Retrieving sensor and its last readings

Let's explore more the API that we already've seen in previous test.

t.Run("get last readings and sensor", func(t *testing.T) {
  tableName := "SensorsTable"
  db, cleanup := dynamo.SetupTable(t, ctx, tableName, "../template.yml")
  defer cleanup()
  manager := sensors.NewManager(db, tableName)

  err := manager.Register(ctx, sensor)

  assert.NoError(t, err)

  err = manager.SaveReading(ctx, sensors.Reading{SensorID: "sensor-1", Value: "0.3", ReadAt: time.Now().Add(-20 * time.Second)})
  assert.NoError(t, err)
  err = manager.SaveReading(ctx, sensors.Reading{SensorID: "sensor-1", Value: "0.5", ReadAt: time.Now().Add(-10 * time.Second)})
  assert.NoError(t, err)
  err = manager.SaveReading(ctx, sensors.Reading{SensorID: "sensor-1", Value: "0.67", ReadAt: time.Now()})
  assert.NoError(t, err)

  sensor, latest, err := manager.LatestReadings(ctx, "sensor-1", 2)
  assert.NoError(t, err)
  assert.Len(t, latest, 2)
  assert.Equal(t, "0.67", latest[0].Value)
  assert.Equal(t, "0.5", latest[1].Value)
  assert.Equal(t, "sensor-1", sensor.ID)
})

The point of this test is to show that we are able to fetch a sensor and latest readings of this sensor at the same time.

Get sensors by location

t.Run("get by sensors by location", func(t *testing.T) {
  tableName := "SensorsTable"
  db, cleanup := dynamo.SetupTable(t, ctx, tableName, "../template.yml")
  defer cleanup()
  manager := sensors.NewManager(db, tableName)

  err := manager.Register(ctx, sensors.Sensor{ID: "sensor-1", City: "Poznan", Building: "A", Floor: "1", Room: "2"})
  err = manager.Register(ctx, sensors.Sensor{ID: "sensor-2", City: "Poznan", Building: "A", Floor: "2", Room: "4"})
  err = manager.Register(ctx, sensors.Sensor{ID: "sensor-3", City: "Poznan", Building: "A", Floor: "2", Room: "5"})

  ids, err := manager.GetSensors(ctx, sensors.Location{City: "Poznan", Building: "A", Floor: "2"})
  assert.NoError(t, err)
  assert.Len(t, ids, 2)
  assert.Contains(t, ids, "sensor-2")
  assert.Contains(t, ids, "sensor-3")
})

This test is the reason why there are two versions of the code in this episode (version 1 and version 2). Both versions have the same test suite we've just described but each of them has different implementation. They differ because I wanted yo show you different approaches of handling hierarchical data modeling.

Before we jump into implementations I wanted to let you know that I am well aware of the fact that the test suite is not complete. There are corner cases that aren't covered, but I hope you'll realize that this is rather the DynamoDB modeling exercise rather than testing exercise. One more thing about testing. Both versions of the code have the same test suite because we are testing behavior - not implementation. I'm repeating myself, but I would like to emphasize the importance of such tests. I know that some times people are discouraged to test their code because whenever they change production code they need to fix a lot of tests too. It's not the case when testing the behavior. That's it. I used the word "behavior" for the last time in this episode. Moving on.

Implementation - version 1

I am going to show you first version of implementation and then we are going to jump into the second one and
compare them. Let me remind you what really is the first version.

PK	SK	Value	City	Building	Floor	Room	Type	ID
SENSOR#S1	READ#2020-03-01-12:30	2
SENSOR#S1	READ#2020-03-01-12:31	3
SENSOR#S1	READ#2020-03-01-12:32	5
SENSOR#S1	READ#2020-03-01-12:33	3
SENSOR#S1	SENSORINFO		Poznań	A	2	13	Gas
CITY#Poznań	LOCATION#A#2#13							S1

Information about a sensor is broken down into two separate items with different Partition Keys. Additionally, every recording is an item that shares the same PK as main item describing the sensor.

Registration

As you can see in the layout, when registering we need to write two different items which means that we are going to need transactions. Let's jump into it.

func (s *sensorManager) Register(ctx context.Context, sensor Sensor) error {
  attrs, err := dynamodbattribute.MarshalMap(sensor.asItem())

What we are doing here is transforming a sensor into something that we can put into the DynamoDB. Before we go any further, let's talk about Sensor type and asItem method. I differentiate here two different types: Sensor which is the public representation of a sensor and additional type sensorItem that is concerned only with how sensor is stored in the DynamoDB. This type is unexported because it is only the implementation detail.

type Sensor struct {
  ID       string
  City     string
  Building string
  Floor    string
  Room     string
}

type sensorItem struct {
  ID string `dynamodbav:"pk"`
  SK string `dynamodbav:"sk"`

  City     string `dynamodbav:"city"`
  Building string `dynamodbav:"building"`
  Floor    string `dynamodbav:"floor"`
  Room     string `dynamodbav:"room"`
}

As you can see Sensor knows nothing about underlying implementation. The asItem method is a transformation that makes sure that PK and SK are set in a proper way.

func (s Sensor) asItem() sensorItem {
  return sensorItem{
    City:     s.City,
    ID:       "SENSOR#" + s.ID,
    SK:       "SENSORINFO",
    Building: s.Building,
    Floor:    s.Floor,
    Room:     s.Room,
  }
}

Notice also that I named Partition Key - PK, and Sort Key - SK. This is because we are using Single Table Design and different items have their own meaning of the PK and SK. In this example SK has value SENSORINFO. It is a constant value. I am setting this that way so that we are able to distinguish a sensor and its readings. Now, back to the implementation. The sensor is in the format that DynamoDB will understand. Next thing we need to take care of is uniqueness. We cannot register the same sensor twice and in order to achieve that we need a condition.

expr, err := expression.NewBuilder().WithCondition(expression.AttributeNotExists(expression.Name("pk"))).Build()

What it says is: "I am going to move further with the operation only if DynamoDB doesn't have an item with pk that I want to store in this operation".

_, err = s.db.TransactWriteItemsWithContext(ctx, &dynamodb.TransactWriteItemsInput{
  TransactItems: []*dynamodb.TransactWriteItem{
    {
      Put: &dynamodb.Put{
        ConditionExpression:       expr.Condition(),
        ExpressionAttributeNames:  expr.Names(),
        ExpressionAttributeValues: expr.Values(),

        Item:      attrs,
        TableName: aws.String(s.table),
      },
    },
    {
      Put: &dynamodb.Put{
        Item: map[string]*dynamodb.AttributeValue{
            "pk": {S: aws.String("CITY#" + sensor.City)},
            "sk": {S: aws.String(fmt.Sprintf("LOCATION#%s#%s#%s", sensor.Building, sensor.Floor, sensor.Room))},
            "id": {S: aws.String(sensor.ID)},
        },
        TableName: aws.String(s.table),
      },
    },
  },
})

We want to put two items into the DynamoDB, sensor itself and the location. First Write Item has a condition that we defined and the other constructs the location. I decided to define it on the fly here because it's not important anywhere else.

Let's have a look at the error handling.

if err != nil {
  _, ok := err.(*dynamodb.TransactionCanceledException)
  if ok {
    return errors.New("already registered")
  }
  return err
}
return nil

It needs to be handled explicitly because we need to verify whether transaction failed because of the failed condition or because something unexpected happened.

Sensor retrieval

In order to retrieve sensor we need to use proper SK and PK which means we need to construct proper Composite Primary Key.

map[string]*dynamodb.AttributeValue{
  "pk": {S: aws.String("SENSOR#" + id)},
  "sk": {S: aws.String("SENSORINFO")},
}

The ID needs to have the prefix, and SK needs to be the constant I choose to mark a sensor. If you want to see whole implementation of Get method please have a look here. There is nothing interesting going on there - just simple data retrieval, so I am not repeating it here.

Saving a reading

Another fairly simple piece of code. It is just a PUT operation of a Reading. What is worth to talk about her is how data structure looks like.

type Reading struct {
  SensorID string
  Value    string
  ReadAt   time.Time
}

type readingItem struct {
  SensorID string `dynamodbav:"pk"`
  Value    string `dynamodbav:"value"`
  ReadAt   string `dynamodbav:"sk"`
}

I used the same pattern as for Sensor. There is Reading that makes sense in the domain, and there is readingItem that defines how implementation is going to look like.

func (r Reading) asItem() readingItem {
  return readingItem{
    SensorID: "SENSOR#" + r.SensorID,
    ReadAt:   "READ#" + r.ReadAt.Format(time.RFC3339),
    Value:    r.Value,
  }
}

This transformation makes sure that PK of an item begins with SENSOR# prefix. We need that because we want readings of the sensor and sensor itself to be in the same Item Collection. Item collection is collection of items that share the same Partition Key. We need that to be able to retrieve sensor and its latest readings with single query. Other thing that is going on here is formatting SK of an item in a way that will be sortable by time.

Retrieving the latest readings and the sensor

We will query two item types at the same time. We need some sort of condition.

expr, err := expression.NewBuilder().WithKeyCondition(expression.KeyAnd(
  expression.KeyEqual(expression.Key("pk"), expression.Value("SENSOR#"+sensorID)),
  expression.KeyLessThanEqual(expression.Key("sk"), expression.Value("SENSORINFO")), 
)).Build()

Let's read it. Attribute pk is the ID prefixed with SENSOR#. This makes sense - we need to fetch whole item collection. Let's keep reading. Attribute sk needs to be less than or equal than SENSORINFO. Wait, what? We wanted to fetch the sensor and it's readings. How on earth such condition is going to achieve that? Bare with me.

PK	SK
SENSOR#S1	READ#2020-03-01-12:30
SENSOR#S1	READ#2020-03-01-12:31
SENSOR#S1	READ#2020-03-01-12:32
SENSOR#S1	READ#2020-03-01-12:33
SENSOR#S1	SENSORINFO

This is excerpt from the table that I showed you before but containing just Composite Primary Key. Items are sorted in ascending order by default. This means that readings are sorted from oldest to the newest, and after readings there is SENSORINFO because S comes after R in the alphabet. What we want to achieve is to read the data backwards starting from the item with SENSORINFO as SK. In order to read the data in this way we need to construct a query with parameter ScanIndexForward set to false.

out, err := s.db.QueryWithContext(ctx, &dynamodb.QueryInput{
  ExpressionAttributeValues: expr.Values(),
  ExpressionAttributeNames:  expr.Names(),
  KeyConditionExpression:    expr.KeyCondition(),
  Limit:                     aws.Int64(last + 1),
  ScanIndexForward:          aws.Bool(false),
  TableName:                 aws.String(s.table),
})

Also, the limit is set to amount of last readings we want to retrieve increased by one so that we will retrieve information about the sensor as well.

What is going on at the end of the method is proper unmarshalling items into domain objects.

var si sensorItem
err = dynamodbattribute.UnmarshalMap(out.Items[0], &si)

var ri []readingItem
err = dynamodbattribute.UnmarshalListOfMaps(out.Items[1:aws.Int64Value(out.Count)], &ri)

var readings []Reading
for _, r := range ri {
  readings = append(readings, r.asReading())
}
return si.asSensor(), readings, nil

We know for a fact that Sensor is first in the item collection, so it is unmarshalled as the Sensor. The rest of the items are treated as Readings.

Get sensors by location

As you remember in this version of implementation - the location is stored as an additional item. Method GetSensors accepts Location type that contains City, Building, Floor and Room. An item representing the location looks like this:

PK	SK	ID
CITY#Poznań	LOCATION#A#2#13	S1

We need to build key condition that will point to PK which is just a City prefixed with CITY# and that has SK that begins with certain prefix. Depending on level of location precision - SK begins with shorter or longer prefix that specify from where we should get the sensors.

expr, err := expression.NewBuilder().WithKeyCondition(expression.KeyAnd(
  expression.KeyEqual(expression.Key("pk"), expression.Value("CITY#"+location.City)),
  expression.KeyBeginsWith(expression.Key("sk"), location.asPath()), 
)).Build()

After building the condition expression we need to use it in the query:

out, err := s.db.QueryWithContext(ctx, &dynamodb.QueryInput{
  ExpressionAttributeNames:  expr.Names(),
  ExpressionAttributeValues: expr.Values(),
  KeyConditionExpression:    expr.KeyCondition(),
  TableName:                 aws.String(s.table),
})

At the end I just prepare list of IDs that should be returned from the method.

var ids []string
for _, i := range out.Items {
    ids = append(ids, aws.StringValue(i["id"].S))
}
return ids, nil

This is it. Complete code for first version of implementation is here.

Implementation - version 2

Second version of the implementation varies a little. The difference lays in how location is stored. In first version queryable location was just additional item. Second version uses Global Secondary Index for that purpose.

PK	SK	City	Building	Floor	Room	Type	GSI_PK	GSI_SK
SENSOR#S1	SENSORINFO	Poznań	A	2	13	Gas	Poznań	A#2#13

Local Secondary Index cannot be used in this scenario because it would need to have the same Partition Key as Primary Key. Because we want to use different Partition Key - we need to use GSI.

I am going to show you only two methods because only they are different - registration of a sensor and retrieving sensors by the location.

Registration

Sensor type stays exactly the same because the domain sense of it doesn't change with implementation. However sensorItem is going to have two additional fields: GSIPK and GSISK.

func (s Sensor) asItem() sensorItem {
  return sensorItem{
    City:     s.City,
    ID:       "SENSOR#" + s.ID,
    SK:       "SENSORINFO",
    Building: s.Building,
    Floor:    s.Floor,
    Room:     s.Room,
    GSIPK:    "CITY#" + s.City,
    GSISK:    fmt.Sprintf("LOCATION#%s#%s#%s", s.Building, s.Floor, s.Room),
  }
}

As you can see GSIPK and GSISK look exactly the same as the additional location item in the first version of implementation. It's the same information but insidesensorItem.

Registration itself holds exactly the same condition as before - which is to make sure that we are not introducing duplicated sensors. What changed is instead of using transactions - we use simple PUT operation.

_, err = s.db.PutItemWithContext(ctx, &dynamodb.PutItemInput{
  ConditionExpression:       expr.Condition(),
  ExpressionAttributeNames:  expr.Names(),
  ExpressionAttributeValues: expr.Values(),

  Item:      attrs,
  TableName: aws.String(s.table),
})

Frankly speaking registration just got very boring. We transform the Sensor into sensorItem and drop it into the DynamoDB with a condition.

Get sensors by location

This method changed just slightly compared to the first version. Let's have a look at the key condition.

expr, err := expression.NewBuilder().WithKeyCondition(expression.KeyAnd(
  expression.KeyEqual(expression.Key("gsi_pk"), expression.Value("CITY#"+location.City)),
  expression.KeyBeginsWith(expression.Key("gsi_sk"), location.asPath()),
)).Build()

It uses exactly the same mechanism as first version but instead of pk and sk, we use gsi_pk and gsi_sk when building key condition expression. What about the query?

out, err := s.db.QueryWithContext(ctx, &dynamodb.QueryInput{
  ExpressionAttributeNames:  expr.Names(),
  ExpressionAttributeValues: expr.Values(),
  KeyConditionExpression:    expr.KeyCondition(),
  TableName:                 aws.String(s.table),
  IndexName:                 aws.String("ByLocation"),
})

It didn't change much either. There is one additional bit which is IndexName that we used. This index has GSI_PK and GSI_SK as its key.

This is the whole difference between two versions.

Summary

We covered a lot this time. Let me enumerate concepts that we used to make this work.

Single Table Design
Fetching two different item type with single query
Modeling hierarchical data in DynamoDB
Sparse Indexes
Transactions

I hope you enjoyed this long journey. Also, I would like to invite you more than ever to fetch this repository and play with examples!

DynamoDB with Go #7 - Modelling hierarchical data with Single Table Design

Jędrzej Szczepaniak — Tue, 13 Oct 2020 06:10:27 +0000

Imagine that you are working for real estate company that offers office space for rent. All the buildings are those smart new ones with a lot of sensors measuring myriad of parameters. You need to design a system that will manage somehow these sensors.

Access patterns

First things first, before we open IDE we need to know what are we going to do with this application. Let's enumerate all actions that should be possible.

We need to be able to register the sensor. I want to introduce a new sensor into the system in a following way:

Register(Sensor{ID: "humidity-sensor-1", City: "Poznan", Building: "A", Floor: "3", Room: "112"})

After succesful registration I expect that there will be possibility to write a new sensor reading.

NewReading(Reading{SensorID: "humidity-sensor-1", Value: "0.67"})

Sensor is registered and there are some data points already in the DB. How can we display this to the user of the system? I would like to be able to get sensors from locations. For example, I would like to say:

give me all sensors in Lisbon,
give me all sensors in Berlin in building D on 4th floor,
give me all sensors in Poznań in building A, 2nd floor in room 432.

I am thinking about the API that would look like this:

GetSensors(Location{City: "Poznań", Building: "A", Floor: "2"})
GetSensors(Location{City: "Poznań", Building: "D", Floor: "1", Room: "123"})

Let's say I searched for: "Poznań, Building A" and received list of 25 different sensors. In terms of UI - I want to click on given sensor and receive detailed information about the sensor with 10 latest readings of this sensor. I imagine a call:

Get("carbon-monoxide-sensor-2", 10)

It seems simple enough. We have four different functions operating on two different types.

Table design

We know WHAT we want achieve. It's time to define the HOW. How can we implement that? I would like to make this episode an exercise for two different things:

Single Table Design,
modelling hierarchical data.

Single Table Design

The idea of single table design is to be able to fetch different types of entities with single query to the DynamoDB. Since there is no possibility to query many tables at the same time, different types needs to be stored together within single table.

Modeling hierarchical data

Being able to fetch data on different level of hierarchy requires some thought upfront. We are going to leverage ability to use begins_with method on the sort key.

Sensors Table

PK	SK	Value	City	Building	Floor	Room	Type
SENSOR#S1	READ#2020-03-01-12:30	2
SENSOR#S1	READ#2020-03-01-12:31	3
SENSOR#S1	READ#2020-03-01-12:32	5
SENSOR#S1	READ#2020-03-01-12:33	3
SENSOR#S1	SENSORINFO		Poznań	A	2	13	Gas

I sketched out first attempt to put data into the table. First thing to notice is that Partition Key and Sort Key don't have a name that conveys domain knowledge. They're just abbreviation and this is because in Single Table Design there are different types of entities, hence PK and SK can mean different things for a different item type.

This layout allows us to query a sensor and obtain detailed information together with the latest reads from it. This is why we are doing the Single Table Design. We only have to query the DynamoDB once to obtain different types of entities.

This isn't bad so far, but there is still one feature missing - querying for many sensors that share the same location. As I said before we are going to use begins_with on the Sort Key. I want to do queries like these (pseudocode):

Query(PK="Poznań", SKbeginswith="A#-1") -> all sensors from garage in building A in Poznań
Query(PK="Berlin") -> all sensors in Berlin
Query(PK="Lisbon", SKbeginswith="F#3#102") -> all sensors in Lisbon in the building F, room 102 on 3rd floor.

In order to do that we need to introduce additional artificial attribute which is a concatenation of different attributes.

First thing that comes to mind is to simply write something like this to the table:

PK	SK	ID
CITY#Poznań	LOCATION#A#2#13	S1

Now I can satisfy the requirement for querying sensors in the given location. One thing to remember though is that we have two items that need to be synchronized. Detailed information about the sensor, and its location. Registering a sensor and changing the location is more complicated with this approach because we need to change two items in the DynamoDB transactionally so that these pieces of information won't diverge.

The other idea is to use Global Secondary Index (GSI).

PK	SK	Value	City	Building	Floor	Room	Type	GSI_PK	GSI_SK
SENSOR#S1	READ#2020-03-01-12:30	2
SENSOR#S1	READ#2020-03-01-12:31	3
SENSOR#S1	READ#2020-03-01-12:32	5
SENSOR#S1	READ#2020-03-01-12:33	3
SENSOR#S1	SENSORINFO		Poznań	A	2	13	Gas	Poznań	A#2#13

Whenever we want to query for sensors in given location we use GSI to do that. Hidden here is yet another concept which is called the Sparse Index. This index is sparse because it contains only some of the items. When querying or scanning that index we won't get any item that has READ# prefix in the SK, because these items aren't in this index (because these items don't have value for GSI_PK and GSI_SK attributes).

Which approach is better? Additional item in a table is very simple approach that just works. One downside it has is that when sensor is being registered or it's location changes - we need to change two items transactionally and programmatic error can cause this data to diverge. On the other hand there is GSI. Its advantage is that we need to change only single item at a time when location changes or when we register new sensor. You need to be aware however that indexes cost extra money and that data is copied to the GSI in a asynchronous way which further means that when reading data from GSI, strongly consistent reads are not an option. All the reads from GSI are eventually consistent.

Summary

I think we've done enough work for now. Let me split this topic into 2 episodes. This one was about DynamoDB, next one will be more with Go. Nevertheless, to summarize, we know what our access patterns are, and we know how to implement that in two different ways. More over we learned what are cons and pros of each approach. What I propose for the next episode is that first we will write unit tests that define behavior we want to obtain. Then we are going to implement both ways: with additional item and with GSI.

DynamoDB with Go #6 - Legacy IDs mapping with transactions

Jędrzej Szczepaniak — Tue, 06 Oct 2020 18:28:29 +0000

I finished last episode with a promise to show you how to approach what we discussed differently. If you want to get most out of this episode I highly recommend you to read episode #5) first. In a nutshell a problem to solve was to map ids from existing legacy system into new system that will use its own ids schema.

Previous approach

Initial solution for this problem was to generate the new id for the legacy id and conditionally put the mapping into DB. The condition stated that we don't want to do that if mapping for given legacy id already exists. When condition succeeded we knew that there is no mapping for given legacy id yet and mapping was inserted into the DynamoDB. If however condition failed we knew that mapping already exists, and we need to grab it from the database.

This approach works very well and is immune to data races. When we call Map function for the first time for given legacy id, we need to make single call to the DynamoDB. The condition succeeds, and we return newly generated id. However, subsequent calls to the Map with the same id will require two calls to the database. First failing call because of the condition we have in place, and second one that grabs already existing new id from the Dynamo.

ReturnValues parameter

It turns out that there is a parameter in the PutItem called ReturnValues. It can be set to NONE (default) or ALL_OLD. ALL_OLD means that if PutItem has overridden an item, then the response from the DynamoDB will contain content of the item before overriding it. This would be great for us. We would like to know - if we failed - what was in the DynamoDB that caused the failure. That would mean that we don't need to call Dynamo for the second time.

Unfortunately - this works only if PutItem succeeds and in case of conditional check failure we don't get an image of an item before the PutItem that failed.

Since that didn't work - maybe error that we get back that informs us about condition failure, contains more context on why it failed?

Unfortunately it doesn't.

We can however use this idea somewhere else to get just what we want.

Transactions

In general, transactions are for something else. You use them when you need to make many changes to the database state and all of them need to be successful in order for transaction to complete. If one of the changes in the transaction fails, then whole transaction fails and none of the changes are being made.

If the transaction fails then the DynamoDB can give you more context of what happened. Let's see how it works.

Code

All the code is the same as in the 5th episode. Only thing that changes is Map function. You can observe here also beauty of automated tests that verify only behaviour. We can change implementation as much as we want and tests do not need to change at all. Moreover, they'll tell us whether new approach works or not.

Let's change Map function to use the transaction.

idsMapping := mapping{OldID: old, NewID: uuid.New().String()}
attrs, err := dynamodbattribute.MarshalMap(&idsMapping)
if err != nil {
  return "", err
}
expr, err := expression.NewBuilder().
  WithCondition(expression.AttributeNotExists(expression.Name("old_id"))).
  Build()
if err != nil {
  return "", err
}

When I told you that only Map function changes I didn't mean all of it. The beginning stays exactly the same. Just to recap, we create the mapping with the old id and the new id that gets generated for us. Then we construct the condition that fails when we want to put an item that already exists.

_, err = m.db.TransactWriteItemsWithContext(ctx, &dynamodb.TransactWriteItemsInput{
  TransactItems: []*dynamodb.TransactWriteItem{
    {
      Put: &dynamodb.Put{
        ConditionExpression:                 expr.Condition(),
        ExpressionAttributeNames:            expr.Names(),
        ExpressionAttributeValues:           expr.Values(),
        Item:                                attrs,
        ReturnValuesOnConditionCheckFailure: aws.String("ALL_OLD"),
        TableName:                           aws.String(m.table),
      },
    },
  },
})

We have a transaction here with single Put operation in it. There is also something that is not an option for regular PutItem method - ReturnValuesOnConditionCheckFailure parameter. Hopefully when condition fails, we will get exactly what we want - which is the new id that already exists.

The key aspect here is error handling.

if err == nil {
  return idsMapping.NewID, nil
}

If there is no error it means that condition succeeded, hence this was first time anyone called Map with given legacy id, and we just return what we've put into the DynamoDB.

aerr, ok := err.(*dynamodb.TransactionCanceledException)
if !ok {
  return "", err
}

If there is an error we need to check its type, and if it is not TransactionCanceledException - something went wrong, and we don't know what it is, so we just return.

return aws.StringValue(aerr.CancellationReasons[0].Item["new_id"].S), nil

Otherwise, we get new_id from CancellationReasons and we can return that to the client without calling Dynamo again!

Summary

We just showed how can we leverage DynamoDB API to give us reason for transaction failure. In our case this means that we can insert into the DynamoDB new mapping or get existing mapping in one step.

Should you use this approach? Is it better than the previous one? As always it depends. When calling the DynamoDB only once you'll save time spent on request/response round trip. Does it come for free then? Absolutely not. Transaction calls to the DynamoFB are more expensive in terms of Capacity Units you'll pay for them. So in this particular case either you pay more for the DynamoDB transaction call, but make fewer calls in general, or pay less for single call but call DynamoDB more times and spend more time waiting for network calls to the DynamoDB. Having said that I am not recommending any of the approaches. It depends on your needs. I am just showing possible options.

DynamoDB with Go #5 - Legacy IDs mapping

Jędrzej Szczepaniak — Tue, 15 Sep 2020 07:09:00 +0000

Imagine that you are developing a brand new software that is based on information from the legacy system. The way you integrate with the legacy is through events that you are listening to. Let's say that you receive information about orders from an event. Each order is identifiable via unique id given by the legacy system that is an incrementally increased integer. One of the requirements for you is to process orders in a way that won't reveal total amount of orders.

It seems that the problem to solve is to map the old id to the new value. One thing to remember here is that we can receive many events about the same order which means that we need to either generate new id or use the id that we already generated for the legacy id. How can we do it with the DynamoDB?

Table definition

Resources:
  LegacyIDsTable:
    Type: AWS::DynamoDB::Table
    Properties:
      AttributeDefinitions:
        - AttributeName: old_id
          AttributeType: S
      KeySchema:
        - AttributeName: old_id
          KeyType: HASH
          Projection:
            ProjectionType: ALL
      BillingMode: PAY_PER_REQUEST
      TableName: LegacyIDsTable

Very straightforward. Just a single attribute that is also partition key.

Failing test first

Let's go here with the TDD approach. I will write a failing test first, then I will try to make it pass!

func TestMapping(t *testing.T) {
  t.Run("generate new ID for each legacy ID", func(t *testing.T) {
    ctx := context.Background()
    tableName := "LegacyIDsTable"
    db, cleanup := dynamo.SetupTable(t, ctx, tableName, "./template.yml")
    defer cleanup()

    mapper := NewMapper(db, tableName)

    first, err := mapper.Map(ctx, "123")
    assert.NoError(t, err)
    assert.NotEmpty(t, first)
    assert.NotEqual(t, "123", first)

    second, err := mapper.Map(ctx, "456")
    assert.NoError(t, err)
    assert.NotEmpty(t, second)
    assert.NotEqual(t, "456", first)

    assert.NotEqual(t, first, second)
  })

  t.Run("do not regenerate ID for the same legacy ID", func(t *testing.T) {
    ctx := context.Background()
    tableName := "LegacyIDsTable"
    db, cleanup := dynamo.SetupTable(t, ctx, tableName, "./template.yml")
    defer cleanup()

    mapper := NewMapper(db, tableName)

    first, err := mapper.Map(ctx, "123")
    assert.NoError(t, err)

    second, err := mapper.Map(ctx, "123")
    assert.NoError(t, err)

    assert.Equal(t, first, second)
  })

}

As you can see I have two requirements I want to cover. First of all I need to generate an id for each incoming legacy id that enters my function. Second of all, if I invoke that function twice with the same legacy id it shouldn't regenerate new id.

Structs setup

Let's take care of basic structs.

type Mapper struct {
  db    dynamodbiface.DynamoDBAPI
  table string
}

type mapping struct {
  OldID string `dynamodbav:"old_id"`
  NewID string `dynamodbav:"new_id"`
}

There is Mapper that holds dependencies to the DynamoDB and the mapping that will be an item we store in the DynamoDB. Moreover external packages need to create the Mapper.

func NewMapper(client dynamodbiface.DynamoDBAPI, table string) *Mapper {
  return &Mapper{db: client, table: table}
}

Solution

Now let's think of the logic of the Map function. If the mapping of an old id and new id already exists in Dynamo - we need to fetch it. If it doesn't, we need to generate new id and save it. At first glance we could just use the GetItemWithContext and if we get nothing we just use thr PutItemWithContext to save newly generated id. Unfortunately this approach won't work. Remember that we can receive many events about the same order. They can arrive any time and can be handled concurrently. If two threads of execution will run the GetItemWithContext in more or less the same time and both will figure out that there is no mapping yet, we will end up with one of the thread overriding the other threads mapping.

Let's see how can we implement that functionality that will work in the world of concurrent execution.

func (m *Mapper) Map(ctx context.Context, old string) (string, error) {
  idsMapping := mapping{OldID: old, NewID: uuid.New().String()}
  attrs, err := dynamodbattribute.MarshalMap(&idsMapping)

At the beginning we create a mapping that has an old id and that generates new id using UUIDv4. Next thing we'll do isn't retrieving an item from Dynamo. Instead we will revert the logic. I want translate into the code the following sentence: Put into the DynamoDB a mapping, but only if it doesn't exist yet. In order to do that, we need to write condition expression.

expr, err := expression.NewBuilder().
  WithCondition(expression.AttributeNotExists(expression.Name("old_id"))).
  Build()

This expression will make sure that the PutItem operation fails if an item with the same partition key as ours with attribute old_item already exists.

_, err = m.db.PutItemWithContext(ctx, &dynamodb.PutItemInput{
  ConditionExpression:       expr.Condition(),
  ExpressionAttributeNames:  expr.Names(),
  ExpressionAttributeValues: expr.Values(),
  Item:                      attrs,
  TableName:                 aws.String(m.table),
})

We ignore output of the function, but care about error very much. If there is no error - then we know that there wasn't any mapping like ours before and we just return to the caller newly generated id.

if err == nil {
  return idsMapping.NewID, nil
}

If however there is an error we need to check what type of error that is. If this error tells us that condition failed, it means that mapping already exists and we can do additional GetItem operation to retrieve it. If this is any other error, something went terribly wrong.

aerr, ok := err.(awserr.Error)
if ok && aerr.Code() == dynamodbErrCodeConditionalCheckFailedException {
  return "", err
}

At this point we know that the PutItem operation failed because our conditional failed.

This means that someone before us already mapped the legacy id to new id and we can retrieve it.

out, err := m.db.GetItemWithContext(ctx, &dynamodb.GetItemInput{
  Key: map[string]*dynamodb.AttributeValue{
    "old_id": {S: aws.String(old)},
  },
  TableName: aws.String(m.table),
})
if err != nil {
  return "", err
}
return aws.StringValue(out.Item["new_id"].S), nil

Summary

We did it - we no longer rely on ids from legacy system and our solution is bulletproof! There is however one downside to it. Only for the first time for given legacy id we will communicate with DynamoDB once. Subsequent calls to Map function require two sequential calls to the DynamoDB. Next time we will try to figure out whether we can do something about it. As always I am inviting you to clone this repository and play with queries yourself!

DynamoDB with Go #4 - Indices

Jędrzej Szczepaniak — Tue, 08 Sep 2020 15:01:08 +0000

Last time we looked at how Composite Primary Keys can improve our ability to query the DynamoDB. This time we'll focus on how indices can help us even more with our access patterns.

In episode #3 of DynamoDB we built an oversimplified filesystem model. Let me share with you a single item from that filesystem so that we are on the same page on how it looked.

Directory	Filename	Size
finances	report2017.pdf	1MB

If you recall we were able to query the filesystem in a following ways:

Give me a single file from a directory
Give me the whole directory

Thanks to the fact that some of the files have year in their names we are able to get files older/younger than given date. Look here how we did it.

As we are testing this on staging, product person looks up our shoulder and sees that we are querying based on the creation time and she absolutely loves it. Except for one thing.

You cannot really expect from our clients to put creation date of the file in the name

Now we have additional requirement. We need to be able to sort files inside directory based on their creation date. This is new access pattern.

This isn't SQL, we cannot simply use WHERE and ORDER BY clauses and query just anything without additional work. The tool for this job in DynamoDB is an index.

Indices

Table in the DynamoDB can have only single pair of main Partition and Sort Keys. It's called Primary Key. They are important because they are entry point for queries.

But why are they the entry point? In order to answer that question let's think of how DynamoDB stores its data. First of all there is a partition key which is used to determine on which partition data is stored. The DynamoDB calculates hash function based on partition key and finds that partition. All items that share the same partition key (so called item collection) are stored within single B-Tree data structure. What is important about B-Tree is that it allows to query data in sorted order. Sort Key is important here because it is the criteria by which items are sorted in the tree. Let's have a look.

I used Binary Search Tree instead of B-Tree to simplify but they share the same property of being sortable and this is enough to build your intuition around the topic.

Data that corresponds to image above is presented here:

Directory (PK)	Filename (SK)	Size	CreatedAt
photos	bike.png	1.2MB	2017-03-04 00:00:00 +0000 UTC
photos	apartment.jpg	4MB	2018-06-25 00:00:00 +0000 UTC
photos	grandpa.png	3MB	2019-04-01 00:00:00 +0000 UTC
photos	kids.png	3MB	2020-01-10 00:00:00 +0000 UTC

Our entry point here is the ability to get items from photos directory and sort them by filename. This is exactly reflected in the image because tree is constructed in a way that allow to sort only by filename. If we would like to have different entry point - we need to introduce an index.

Indices enhance ability to query the table by introducing additional Partition Keys and Sort Keys.

DynamoDB has two types of indices: Local Secondary Indices (LSI) and Global Secondary Indices (GSI). Secondary means that they are an addition to Primary Key.

Local Secondary Index

LSI has the same Partition Key as Primary Key but different Sort Key. This allows us to sort items by additional attribute. In order to do that DynamoDB has to store additional - reorganized tree. Each sorting pattern needs to have it's own tree.

Our additional access pattern was sorting by created_at attribute. This means additional tree sorted by created_at. Let's have a look at the image to warp our heads around that.

LSI needs to have the same Partition Key as the table itself - this is why it's local. We can use the same partition to find both Primary Sort Key and Sort Key for the LSI. Important limitation of LSI is that it needs to be created when the table is created. There can be up to 5 LSIs on single table.

Global Secondary Index

GSI is different. It allows to have arbitrary Partition Key and Sort Key for the same table. Let's construct an index where Partition Key is size and filename is Sort Key. This would allow to do query that would group photos by their size.

We no longer have the possibility to keep an index close to Primary Key because Partition Key on the index is completely different. We need to construct a new data structure. That's why this index is global - it lives somewhere else so to speak. GSIs are important because they allow to query the table in just any way you imagine. As opposed to LSIs, GSIs can be created anytime in the table lifetime. There can be up to 20 GSIs per table.

Database layout

We will concentrate on solving our business problem and we will create LSI that allows to sort by created_at.

Resources:
  FileSystemTable:
    Type: AWS::DynamoDB::Table
    Properties:
      AttributeDefinitions:
        - AttributeName: directory
          AttributeType: S
        - AttributeName: filename
          AttributeType: S
        - AttributeName: created_at
          AttributeType: S
      KeySchema:
        - AttributeName: directory
          KeyType: HASH
        - AttributeName: filename
          KeyType: RANGE
      LocalSecondaryIndexes:
        - IndexName: ByCreatedAt
          KeySchema:
            - AttributeName: directory
              KeyType: HASH
            - AttributeName: created_at
              KeyType: RANGE
          Projection:
            ProjectionType: ALL
      BillingMode: PAY_PER_REQUEST
      TableName: FileSystemTable

This is very similar to what we had before in the third epiode, there are only two differences. First difference is that we have new big section LocalSecondaryIndexes where an index lives. I called it ByCreatedAt because this is essentially what I want to do - I want to query by the creation time. If we look inside we will see something very similar to the Composite Primary Key definition. There is KeySchema with RANGE and HASH keys. Second difference is that there is new attribute definition. Because RANGE key of the CreatedAt index uses created_at attribute it has to be defined.

type item struct {
    Directory string    `dynamodbav:"directory"`
    Filename  string    `dynamodbav:"filename"`
    Size      string    `dynamodbav:"size"`
    CreatedAt time.Time `dynamodbav:"created_at"`
}

The item changed as well. It has new attribute - CreatedAt.

Query #1 - Photos taken from 2019

Initial setup for all queries will be the same. It's just:

ctx := context.Background()
tableName := "FileSystemTable"
db, cleanup := dynamo.SetupTable(t, ctx, tableName, "./template.yml")
defer cleanup()

Now let's build key condition expression.

expr, err := expression.NewBuilder().
  WithKeyCondition(
    expression.KeyAnd(
      expression.KeyEqual(
        expression.Key("directory"),
        expression.Value("photos"),
        ),
      expression.KeyGreaterThanEqual(
        expression.Key("created_at"),
        expression.Value(time.Date(2019, 1, 1, 0, 0, 0, 0, time.UTC)),
      ))).
  Build()

I need to provide the exact directory for the query. Additionally I am using KeyGreaterThanEqual in order to take advantage of the index. From perspective of this expression however the index is not visible. Hypothetically with table where directory is the Partition Key and created_at is the Sort Key it would be valid expression as well. The point here is that we cannot have many sort keys on the table - we need to use indices for that - but it kind of works as additional sort key and it doesn't affect expression very much.

out, err := db.QueryWithContext(ctx, &dynamodb.QueryInput{
  ExpressionAttributeNames:  expr.Names(),
  ExpressionAttributeValues: expr.Values(),
  KeyConditionExpression:    expr.KeyCondition(),
  TableName:                 aws.String(table),
  IndexName:                 aws.String("ByCreatedAt"),
})
assert.Len(t, out.Items, 2)

As you can see we need to specify that we are using the ByCreatedAt index. If we don't specify that - the DynamoDB will complain that key condition expression is missing its sort key (filename) and has other field instead (created_at). At the end I am just checking if result consists of 2 items.

Query #2 - Photos taken from 2017 to 2018

The query looks exactly the same as in first query (hence I'im skipping it), but the Key Condition Expression used to construct this query uses new operator which I wanted to show you.

expr, err := expression.NewBuilder().
  WithKeyCondition(
    expression.KeyAnd(
      expression.KeyEqual(
        expression.Key("directory"),
        expression.Value("photos"),
      ),
      expression.KeyBetween(
        expression.Key("created_at"),
        expression.Value(
          time.Date(2017, 1, 1, 0, 0, 0, 0, time.UTC)),
        expression.Value(
          time.Date(2019, 1, 1, 0, 0, 0, 0, time.UTC))))).
  Build()

It's that simple - you just need to set lower and upper bounds for your search and that's all - you get photos from 2017 to 2018.

Query #3 - Newest photo

Because an index stores items in the order - sorted by its Sort Key (of the index) - we can access data in a new way. We need to have the Key Condition Expression that will specify only the Partition Key - so that we are in photos directory.

expr, err := expression.NewBuilder().
  WithKeyCondition(expression.KeyEqual(expression.Key("directory"), expression.Value("photos"))).
  Build()

As the Key Condition Expression got simpler, the query itself got a little bit more complicated.

out, err := db.QueryWithContext(ctx, &dynamodb.QueryInput{
  ExpressionAttributeNames:  expr.Names(),
  ExpressionAttributeValues: expr.Values(),
  KeyConditionExpression:    expr.KeyCondition(),
  TableName:                 aws.String(table),
  IndexName:                 aws.String("ByCreatedAt"),
  ScanIndexForward:          aws.Bool(false),
  Limit:                     aws.Int64(1),
})

First of all we need to specify that we wan't to traverse index backwards because by default index keeps data in ascending order - we do it with the ScanIndexForward parameter. Second of all we want to limit query results to single item. Without that bit - we would return all of the photos from newest to oldest one.

At the end we just verify whether item we obtained is really the newest one.

var items []item
err =  dynamodbattribute.UnmarshalListOfMaps(out.Items, &items)
assert.Equal(t, 2020, items[0].CreatedAt.Year())

Summary

This episode was about indices. First we built an intuition on how they work and then we used Local Secondary Index to query by additional attribute.

Indices are esessential concept in the DynamoDB and we will see more of them when working with more complicated data models.

Once again - I am inviting you to clone this repository and play with queries by yourself!

DynamoDB with Go #3 - Composite Primary Keys

Jędrzej Szczepaniak — Tue, 01 Sep 2020 15:37:59 +0000

Use this 4 tricks to build filesystem in DynamoDB under 15 minutes!

After reading previous episode you could have an impression that DynamoDB is just simple key-value store. I would like to straighten things out because DynamoDB is much more than that. Last time I also mentioned that keys can be more complicated. Get ready to dive into the topic of keys in DynamoDB with Go #3!

Let's build filesystem

Maybe it won't be full fledged filesystem, but I would like to model tree-like structure (width depth of 1, nested directories not allowed) where inside a directory there are many files. Moreover, I would like to query this filesystem in two ways:

Give me single file from given directory,
Give me all files from given directory.

These are my access patterns. I want to model my table in a way that will allow me to perform such queries.

Composite Primary Key

Composite Primary Key consists of Partition Key and Sort Key. Without going into details (AWS documentation covers this subject thoroughly), a pair of Partition Key and Sort Key identifies an item in the DynamoDB. Many items can have the same Partition Key, but each of them needs to have a different Sort Key. If you are looking for an item in the table and you already know what is the Partition Key, Sort Key narrows down the search to the specific item.

If a table is defined only by a Partition Key; each item is recognized uniquely by its Partition Key. If, however, a table is defined with a Composite Primary Key; each item is recognized by pair of Partition and Sort keys.

Table definition

With all that theory in mind, let's figure out what should be a Partition Key and a Sort Key in our filesystem.

Each item in the table will represent a single file. Additionally, each file must point to its parent directory. As I mentioned, a Sort Key kind of narrows down the search. In this example, knowing already what directory we are looking for, we want to narrow down the search to a single file.

All that suggests that directory should be the Partition Key and filename the Sort Key. Let's express it as a CloudFormation template.

Resources:
  FileSystemTable:
    Type: AWS::DynamoDB::Table
    Properties:
      AttributeDefinitions:
        - AttributeName: directory
          AttributeType: S
        - AttributeName: filename
          AttributeType: S
      KeySchema:
        - AttributeName: directory
          KeyType: HASH
        - AttributeName: filename
          KeyType: RANGE
      BillingMode: PAY_PER_REQUEST
      TableName: FileSystemTable

We need to define two attributes (directory and filename), because both of them are part of the Composite Primary Key. As you can see there is no Sort Key in the template. There is however RANGE key type. Just remember that:

HASH key type corresponds to Partition Key
RANGE key type corresponds to Sort Key

Moving on to the code

This is how a single item in the DynamoDB is going to look.

type item struct {
  Directory string `dynamodbav:"directory"`
  Filename  string `dynamodbav:"filename"`
  Size      string `dynamodbav:"size"`
}

I am going to insert a couple of items to the database so that we have content to query. Code that is doing that is omitted for brevity, you can look it up here. At the end I want to have following table:

Directory	Filename	Size
finances	report2017.pdf	1MB
finances	report2018.pdf	1MB
finances	report2019.pdf	1MB
finances	report2020.pdf	2MB
fun	game1	4GB

Query #1: Give me single file from given directory

We need to start with a database setup.

func TestSingleFileFromDirectory(t *testing.T) {
  ctx := context.Background()
  tableName := "FileSystemTable"
  db, cleanup := dynamo.SetupTable(t, ctx, tableName, "./template.yml")
  defer cleanup()
  insert(ctx, db, tableName)

With a connection to DynamoDB in place and with the testing data inserted, we can move on to the query itself. I want to obtain a single element from the DynamoDB, thus I am going to use GetItemWithContext.

out, err := db.GetItemWithContext(ctx, &dynamodb.GetItemInput{
  Key: map[string]*dynamodb.AttributeValue{
    "directory": {S: aws.String("finances")},
    "filename":  {S: aws.String("report2020.pdf")},
  },
  TableName: aws.String(table),
})

Note that Key consists of two elements: directory (Partition Key) and filename (Sort Key). Let's make sure that output of the query is really what we think it is:

var i item
err = dynamodbattribute.UnmarshalMap(out.Item, &i)
assert.NoError(t, err)
assert.Equal(t, item{Directory: "finances", Filename: "report2020.pdf", Size: "2MB"}, i)

Query #2: Give me whole directory

In this query we cannot use GetItemWithContext because we want to obtain many items from the DynamoDB. Also when we get a single item we need to know whole composite primary key. Here we want to get all the files from the directory so we know only the Partition Key. Solution to that problem is QueryWithContext method with Key Condition Expression.

expr, err := expression.NewBuilder().
  WithKeyCondition(
    expression.KeyEqual(expression.Key("directory"), expression.Value("finances"))).
  Build()
assert.NoError(t, err)

out, err := db.QueryWithContext(ctx, &dynamodb.QueryInput{
  ExpressionAttributeNames:  expr.Names(),
  ExpressionAttributeValues: expr.Values(),
  KeyConditionExpression:    expr.KeyCondition(),
  TableName:                 aws.String(table),
})
assert.NoError(t, err)
assert.Len(t, out.Items, 4)

That is a lot of new things, so let me break it down for you. First part is where we construct key condition expression which describes what we really want to query. In our case this is just "Give me all items whose directory attribute is equal to finances". I am using an expression builder which simplifies construction of expressions by far.

In the next step we are using expression inside the query. We need to provide condition, names, and values. In this example, condition is just an equality comparison, where names correspond to names of attributes and values correspond to... their values!

An expression object gives us easy access to condition, names, and values. As you can see I am using them as parameters to QueryInput.

At the end, I am just checking whether we really have 4 items which are in finances directory.

Bonus - Query #3 Give me reports before 2019

It turns out that I constructed filenames in a way that makes them sortable. I figured - let's try to use it to our advantage and get all the reports created before 2019.

Query stays exactly the same. The only thing we need to change is the key condition expression.

expr, err := expression.NewBuilder().
WithKeyCondition(
  expression.KeyAnd(
    expression.KeyEqual(expression.Key("directory"), expression.Value("finances")),
    expression.KeyLessThan(expression.Key("filename"), expression.Value("report2019")))).
Build()
assert.NoError(t, err)

We have two conditions that we combine with the AND clause. The first one specifies what is our Partition Key, second one, the Sort Key. KeyLessThan makes sure that we will only get report2018.pdf and report2017.pdf. Let's have a look at the results of the query.

var items []item
err = dynamodbattribute.UnmarshalListOfMaps(out.Items, &items)
assert.NoError(t, err)
if assert.Len(t, items, 2) {
  assert.Equal(t, "report2017.pdf", items[0].Filename)
  assert.Equal(t, "report2018.pdf", items[1].Filename)
}

In the first query we used dynamodbattribute.UnmarshalMap for unmarshaling single DynamoDB item into the struct. We knew we will get single item. Here we know that there will be one item or more - thus we use dynamodbattribute.UnmarshalListOfMaps - which unmarshals the query results into the slice of items.

Note that I assert that first item is the report from 2017 and second one is from 2018. How am I so sure that items will go back from the DynamoDB in that order? If not told otherwise - DynamoDB will scan items from given Partition in ascending order. Since 2017 comes before 2018 - I know that first item should be from 2017.

Summary

We learned today how to use composite primary keys. Moreover we know how to take advantage of them with Go! That's great! You know what is even better? Playing with the code! Make sure to clone this repository and tinker with it!

Also we used expression builder to create the DynamoDB expression. Get used to them - we will use them a lot in the future episode! It takes some time to build intuition around using expression builder API, but it's totally worth it!

DEV Community: Jędrzej Szczepaniak

You should want to write unit tests and here is why

Serverless and microservices are... pretty much the same?

How serverless fits in?

How this tech fits in the monolith/microservices battle?

With microservices - each team can be responsible for their microservice(s) end to end.

With microservices - you can deploy and scale different pieces of the application separately.

With microservices - your application can be more fault tolerant. Ideally, breaking one microservice doesn't mean that your whole application is down.

With microservices - When different pieces of code become services that are separately developed and deployed, you need to be extra careful when dividing your domain so that each service has responsibility clearly set.

With microservices - network calls are slower than function calls and they can randomly fail.

With microservices - Debugging becomes a problem, you need to have things like tracing.

Let's go back to the original question.

Let's go back to the title.

DynamoDB with Go #12 - Condition on other item from item collection

DynamoDB with Go #11 - Expressions API

Example #1 - Get item collection

Example #2 - "update A and unset B but only if B is set to baz"

Example #3 - Put if doesn't exist

Summary

DynamoDB with Go #10 - Gotcha with empty slices

Different flavours of empty slices

1. Zero value slice

2. Short declaration

3. Make

Empty slices in the DynamoDB

Summary

DynamoDB with Go #9 - Switching the toggle, toggling the switch

Test suite

A solution

Coding time

Summary

DynamoDB with Go #8 - Implement hierarchical data with Single Table Design

Registering a sensor

Inability to register the same sensor twice

Recording a sensor reading

Retrieving sensor and its last readings

Get sensors by location

Implementation - version 1

Registration

Sensor retrieval

Saving a reading

Retrieving the latest readings and the sensor

Get sensors by location

Implementation - version 2

Registration

Get sensors by location

Summary

DynamoDB with Go #7 - Modelling hierarchical data with Single Table Design

Access patterns

Table design

Single Table Design

Modeling hierarchical data

Sensors Table

Summary

DynamoDB with Go #6 - Legacy IDs mapping with transactions

Previous approach

ReturnValues parameter

Transactions

Code

Summary

DynamoDB with Go #5 - Legacy IDs mapping

Table definition

Failing test first

Structs setup

Solution

Summary

DynamoDB with Go #4 - Indices

You cannot really expect from our clients to put creation date of the file in the name

Indices

Local Secondary Index

Global Secondary Index

Database layout

Query #1 - Photos taken from 2019

Query #2 - Photos taken from 2017 to 2018

Query #3 - Newest photo

Summary

DynamoDB with Go #3 - Composite Primary Keys

Use this 4 tricks to build filesystem in DynamoDB under 15 minutes!

Let's build filesystem

Composite Primary Key

Example #2 - "update A and unset B but only if B is set to `baz`"