DynamoDB - dynamic method to insert or edit an item

It might seem pointless to write a post about Updating an Item on DynamoDB when there is plenty of documentation available on AWS and on the internet in general, but I must say that I struggled quite a bit to have it working properly due to the differences of AWS SDK v2 and v3, the DynamoDbClient and DynamoDBDocumentClient and various issues due to marshalling/unmarshalling and condition expressions.

Therefore I decided to share ( and keep it as a reference for myself in the future) the outcome of my struggle.

Edit an item or create a new one

According to the docs the UpdateItem method:

Edits an existing item's attributes, or adds a new item to the table if it does not already exist. You can put, delete, or add attribute values. You can also perform a conditional update on an existing item (insert a new attribute name-value pair if it doesn't exist, or replace an existing name-value pair if it has certain expected attribute values).

This is exactly what I needed. I receive some data from an API and want to stuff it in DynamoDB. If there was already an element with the same ID I want to update all the attributes I received, otherwise I will simply insert a new row.

It's good that such a method exists, otherwise we would have to Search for an Item, and do a Put if it wasn't found or an Edit if it was. Not so handy, right?

Client or DocumentClient?

One of the most confusing thing I noticed since I started working with DynamoDB is the existence, in AWS SDK for Javascript, of two ways of doing things: via the DynamoDB Client and the DynamoDBDocumentClient - which is what you should be using all the time, since it simplifies any method by abstracting away the marshalling/unmarshaling of the attributes by using native Javascript types):

Compare the DynamoDBClient Put

// you must specify attributes 
const dynamodb = new AWS.DynamoDB({apiVersion: '2012-08-10'});
const params = {
    Item: {
        "Artist": {
            S: "No One You Know"
        },
        "SongTitle": {
            S: "Call Me Today"
        },
        "Year": {
            N: 2001
        }
    },
    TableName: "Music"
};
const response = await dynamodb.putItem(params).promise() 
// Don't use this method!

with the DocumentClient:

const documentClient = new AWS.DynamoDB.DocumentClient();
const params = {
    Item: {
        "Artist": "No One You Know",
        "SongTitle": "Call Me Today",
        "Year": 2001
        }
    },
    TableName: "Music"
};
const response = await documentClient.put(params).promise() 
// pay attention to the method name, it is slightly different

Quite handy isn't it? Absolutely, because that means that you can receive your data and validate it and then pass it straight away to a generic function in charge of the put, no need to find out the props and types and then verbosely specify in the params!

AWS SDK Version 3

Now let's add the require changes to have this working with AWS SDK Version 3 (I wrote about the major differences in this post):

import {DynamoDBClient} from "@aws-sdk/client-dynamodb";
import {DynamoDBDocumentClient, PutCommand} from "@aws-sdk/lib-dynamodb";
const dynamoClient = new DynamoDBClient()
const documentClient = DynamoDBDocumentClient.from(dynamoClient)
const params = {
    Item: {
        "Artist": "No One You Know",
        "SongTitle": "Call Me Today",
        "Year": 2001
        }
    },
    TableName: "Music"
};
 const response = await documentClient.send(new PutCommand(params))

But let's go back to the object of this post: how to edit an item.

Put or Update, what's the difference?

Put inserts data in a row, updated edits an existing row or adds a new one.

Therefor don't even think to use the Put to update only some of the attributes. If you do that, DynamoDB will overwrite your current row and delete all the other attributes you did not pass to the put method (unless you added a ConditionExpression to prevent it).
If on the other hand you are always sure you have the entire object, with all the properties you need and that you have in the row, and you don't care that the data is entirely overwritten ( imagine if you have some inserted_timestamp, or versionNr ) then you can as well just resort to use the Put.

Normally though, it makes more sense to use the UpdateItem.

Express your updates

I find the Update method a bit more complex due to the UpdateExpressions.
Contrary to the put you can't just pass an object with just the couple of props that have changed, but you have to specify ( with a kinda awkward syntax ) the expression, the values, and the attribute names that were changed:

const params = {
    TableName: "Music",
    Key: {
        "Artist": "No One You Know",
    },
    UpdateExpression:
        'set #title = :v_songTitle, #year = :v_year',
    ExpressionAttributeNames: {
        '#title': 'SongTitle',
        '#year': 'Year'
    },
    ExpressionAttributeValues: {
        ':v_songTitle': "Call me tomorrow",
        ':v_year': 1998
    },
    ReturnValues: "ALL_NEW"
}
const response = await documentClient.update(params).promise() 

Not super clear, right? What is that #title, that :v_songTitle ?!?

In this specific example the ExpressionAttributeNames could in fact be omitted and the real property name could be used, but I wanted to show the complexity you can get if you have properties that conflict with some Dynamo reserved keys (see full list here
They are way more than you can imagine:

• Name? reserved!
• Counter? reserved!
• Comment? reserved
• Day? reserved!
• Status? reserved
• Language? reserved!

As you can see, lots of property names your average database object might have could be reserved. So if you don't want to see your Update function fail, get used to using ExpressionAttributeNames.

That means,

• mention all prop names you are going to edit prefixing them with #. ('#title': 'SongTitle')
• list all the values that are changing giving them a propname that starts with : (':v_songTitle': "Call me tomorrow")
• specify which values are being set in the Update Expression ('set #title = :v_songTitle')

Make it dynamic

All good when it comes with an actual update, where only some properties are changing, but what if the object is new and I have to list them all, what if I what that to be dynamic: Given an object, just give me all those expressions for all the props it has?

A quick search on StackOverflow gave me an interesting snippet of code, which I immediately tried, but due to the way my table was built, given the object I was passing around, and given the marshalling/unmarshalling of attributes I struggled a while to have it properly working.

// solution from https://stackoverflow.com/a/66036730 
const {
  DynamoDBClient, UpdateItemCommand,
} = require('@aws-sdk/client-dynamodb');
const { marshall, unmarshall } = require('@aws-sdk/util-dynamodb');

const client = new DynamoDBClient({});

/**
 * Update item in DynamoDB table
 * @param {string} tableName // Name of the target table
 * @param {object} key // Object containing target item key(s)
 * @param {object} item // Object containing updates for target item
 */
const update = async (tableName, key, item) => {
  const itemKeys = Object.keys(item);

  // When we do updates we need to tell DynamoDB what fields we want updated.
  // If that's not annoying enough, we also need to be careful as some field names
  // are reserved - so DynamoDB won't like them in the UpdateExpressions list.
  // To avoid passing reserved words we prefix each field with "#field" and provide the correct
  // field mapping in ExpressionAttributeNames. The same has to be done with the actual
  // value as well. They are prefixed with ":value" and mapped in ExpressionAttributeValues
  // along witht heir actual value
  const { Attributes } = await client.send(new UpdateItemCommand({
    TableName: tableName,
    Key: marshall(key),
    ReturnValues: 'ALL_NEW',
    UpdateExpression: `SET ${itemKeys.map((k, index) => `#field${index} = :value${index}`).join(', ')}`,
    ExpressionAttributeNames: itemKeys.reduce((accumulator, k, index) => ({ ...accumulator, [`#field${index}`]: k }), {}),
    ExpressionAttributeValues: marshall(itemKeys.reduce((accumulator, k, index) => ({ ...accumulator, [`:value${index}`]: item[k] }), {})),
  }));

  return unmarshall(Attributes);
};

First, I got some weird errors related to the Key and its value, depending on the various iterations that I tried I got:

ValidationException: Value null at 'key' failed to satisfy constraint: Member must not be null

or

ValidationException: The provided key element does not match the schema

Then, when I finally got it right I was stuck at:

ValidationException: One or more parameter values were invalid: Cannot update attribute "my-key". This attribute is part of the key

Of course it is! Since I don't have any object yet, this is in practice similar to a PUT ( an insert not an edit! ), therefore I need to specify what data goes into the Partition Key! But if Update method is supposed to do exactly that (edit an item or create a new one) what am am I doing wrong?

The solution

It turned out that the problem was that ( due to the dynamic expression/attributes) I was telling dynamo to SET the values for my primary key, which is not allowed.

As soon as I filtered out the primary key property from that method returning all attribute names and values for every object property, everything was working as expected!

In the end, it also seemed that Marshalling and Unmarshalling suggested in the answer is not even necessary ( isn't that exactly what DocumentClient takes care of? - if you know more, please write it in the comments).

So this is my final dynamic PutOrEdit method:

/**
 * Edit item in DynamoDB table or inserts new if not existing
 * @param {string} tableName // Name of the target table
 * @param {string} pk // partition key of the item ( necessary for new inserts but not modifiable by the update/edit)
 * @param {object} item // Object containing all the props for new item or updates for already existing item
**/
const update = async (tableName, item, pk) => {
const itemKeys = Object.keys(item).filter(k => k !== pk);
    const params = {
        TableName: tableName,
        UpdateExpression: `SET ${itemKeys.map((k, index) => `#field${index} = :value${index}`).join(', ')}`,
        ExpressionAttributeNames: itemKeys.reduce((accumulator, k, index) => ({
            ...accumulator,
            [`#field${index}`]: k
        }), {}),
        ExpressionAttributeValues: itemKeys.reduce((accumulator, k, index) => ({
            ...accumulator,
            [`:value${index}`]: item[k]
        }), {}),
        Key: {
            [pk]: item[pk]
        },
        ReturnValues: 'ALL_NEW'
    };
return await dynamoDocClient.send(new UpdateCommand(params))

Hope it helps

---

Photo by Max Langelott on Unsplash

Comments

[Flaxmagic]

Great solution, thanks for the post :+1

[Leonardo Gargioni]

Thanks for sharing. I'm new to javascript and DynamoDB, but since my table PK had a Sort Key, it didn't work. So I made some modifications:

/**
 * Edit item in DynamoDB table or inserts new if not existing
 * @param {string} tableName // Name of the target table
 * @param {object} item // Object containing all the props for new item or updates for already existing item
 * @param {string} pk // partition key of the item ( necessary for new inserts but not modifiable by the update/edit)
 * @param {string} sk // (optional) sort key of the item ( necessary for new inserts but not modifiable by the update/edit)
 * 
 * Adapted from https://dev.to/dvddpl/dynamodb-dynamic-method-to-insert-or-edit-an-item-5fnh
**/

import { ddbDocClient } from "./ddbDocClient.js";
import { UpdateCommand } from "@aws-sdk/lib-dynamodb";

export async function update(tableName, item, pk, sk) {
    const itemKeys = Object.keys(item).filter(k => k !== pk && k !== sk);
    const params = {
        TableName: tableName,
        UpdateExpression: `SET ${itemKeys.map((k, index) => `#field${index} = :value${index}`).join(', ')}`,
        ExpressionAttributeNames: itemKeys.reduce((accumulator, k, index) => ({
            ...accumulator,
            [`#field${index}`]: k
        }), {}),
        ExpressionAttributeValues: itemKeys.reduce((accumulator, k, index) => ({
            ...accumulator,
            [`:value${index}`]: item[k]
        }), {}),
        Key: {
            [pk]: item[pk],
            ...(arguments.length === 4 && { [sk]: item[sk] }),
        },
        ReturnValues: 'ALL_NEW'
    };
    return await ddbDocClient.send(new UpdateCommand(params))
}

[Ryan Klein]

Very useful. Thank you!

This line can be used in Typescript.

const itemKeys = (Object.keys(item) as Array<keyof typeof item>).filter(k => k !== 'pk')

Source stackoverflow.com/a/52856805/2201533