🦄 Making great presentations more accessible.
This project enhances multilingual accessibility and discoverability while preserving the original content. Detailed transcriptions and keyframes capture the nuances and technical insights that convey the full value of each session.
Note: A comprehensive list of re:Invent 2025 transcribed articles is available in this Spreadsheet!
Overview
📖 AWS re:Invent 2025 - Simplifying orchestrations with JSONata and AWS Step Functions (API323)
In this video, Ishita Chakraborty, Senior Technical Account Manager at AWS, demonstrates how JSONata simplifies AWS Step Functions orchestrations by eliminating unnecessary Lambda functions. She presents three key strategies: using JSONata for complex data transformations, leveraging direct SDK and HTTPS API calls, and reducing Lambda dependency. Through a stock trading workflow example, she shows how JSONata handles data aggregation, filtering, and DynamoDB payload formatting natively within Step Functions, using Amazon Q to generate JSONata queries. The session covers practical benefits including reduced latency, lower costs, simplified maintenance, and elimination of Lambda runtime deprecation issues. She also previews JSONata's application in generative AI agentic workflows using Bedrock runtime calls, emphasizing that complex transformations can be performed directly in Step Functions without creating multiple Lambda functions for every small task.
; This article is entirely auto-generated while preserving the original presentation content as much as possible. Please note that there may be typos or inaccuracies.
Main Part
Introduction to AWS Step Functions and the Challenge of Lambda Function Proliferation
Hello everyone. How are you doing? How is your event going so far? All good, having fun. All right. Welcome to the session on simplifying orchestrations with JSONata and AWS Step Functions. My name is Ishita Chakraborty. I'm a Senior Technical Account Manager at AWS. I've been working with orchestration use cases with customers for over 18 years, and as part of this session, we are going to be talking about how to simplify orchestrations and why it matters, why we need that simplification.
All right, so quick audience poll: how many of you are familiar with Step Functions? Raise your hands. All right, I see a couple of hands over here, that's great, but just to level set, we'll do a quick introduction as well. So what are Step Functions? Step Functions is a low-code visual workflow service. It provides you with orchestration for AWS services as well as your third-party APIs or your own application APIs. It supports common workflow patterns like parallel workflows or iterative workflows. You want to create some maps, you want to do human workflows, you want to have long-running workflows, you want to do some error handling, retries, end-to-end monitoring—all that comes with Step Functions by default.
These are the workflow patterns that we talked about, and Step Functions simplifies automation and orchestration use cases. Think of automating ETL scenarios or automating security and IT automations, or orchestrating microservices or agentic workflows. We won't get into too much of that, but we do have a slide for that as well at the end. Let's start off with the traditional approach in terms of what Step Functions workflows you would normally see. Here you have a very traditional approach for Step Functions where you have a Lambda function which you invoke to check stock price, and then you have a human approval to decide what to do with the stock—buy or sell. Then you have a Lambda invoked to buy the stock or to sell the stock, you make a decision, and then you report the results. This is a very traditional serverless stock trading workflow that you might have seen before.
What do we see over here? Some of the things we notice are that we have a lot of Lambda functions. This is a valid architecture, but what are some of the pain points that we see with this kind of architecture? Some of the common pain points are Lambda function proliferation. We have so many Lambda functions that we have to maintain all these Lambda functions now. There's limited data transformation capabilities. Previously with Step Functions, you had JSONPath, and then you had to juggle between input path and output path. That was not very convenient. And then you have maintenance challenges with all these Lambda functions. You have to maintain these Lambda functions, you have to maintain the CI/CD, as well as the deprecation. Let's not forget about that. What it leads to is an increase in latency and cost. You're hopping from Step Functions to Lambda functions, and now you've got to pay for both of them.
Now, how many of you are familiar with JSONata? Have you heard of it? All right, nobody. All right, let JSONata enter. So JSONata is a lightweight query and transformation language in JSON. It's open source and inspired by XPath. A lot of people might not be familiar with it, but I'm probably from the dinosaur age, so I am familiar with XPath as well. You're able to navigate JSON objects and arrays. You can flatten out arrays very easily, and you have all these built-in operators and functions that you can use for manipulating data. You have numeric operators, boolean operators, string functions, and a host of functions that are there. I'll have the references at the end, so you can look at the documentation as well.
Out of these, you can create user-defined functions. You can have variable definitions and then combine all these functions together to create your own unique user-defined functions that you can use. And the best part of it all is that this has native integration with AWS Step Functions. You're not having to do anything additional on top of it. It's built in and available within Step Functions itself, and you can use it. All right, so now let's get back to that same slide that we were looking at, the traditional approach. We had these Lambda functions, and for a workflow this small, three Lambda functions was a little too much.
Three Strategies for Simplifying Orchestrations with JSONata
So what do we do? Let's get rid of the boxes. Should we flip it? Let me try flipping. This is what I call the simplified approach. As part of this simplified approach, the first strategy is that I have removed the Lambda functions and I'm now making direct HTTPS API calls. We are using the transformation power of JSONata as we are calling, and as a result there's no Lambda proliferation now. We are changing the payload using JSONata to fit the end system. You can get input in any structure that you want, and you're formulating that to send it to the end system using JSONata.
Now let's add a little more complexity to it. What if we want to use all these transactions that we are getting and aggregate all these results and send them to the consumer as a report? It's a stock trading application, so you might want a report at the end of the day showing what all you sold, how much did you earn, and how much did you lose. This becomes a data analysis scenario. If you look at the input, you have transaction IDs, whether you want to buy or sell, the stock symbol, the sale price, and the quantity associated with that. In terms of what you want to send to the consumer as a report, you have the transaction type, whether it's a long-term or a short-term transaction.
Long-term transactions are transactions that are held for greater than 365 days, and often they have different tax implications depending on the countries that you are from. Let's look at the Step Functions ASL snippet that we have, and this is in YAML so that we can read it better. As part of this, we are defining variables for long-term and short-term using JSONata, where we are filtering the input depending on when the purchase was made. We are doing that filtration at the top and defining these variables. Then we are mapping these into long-term and short-term, counting the transactions in terms of how many are long-term and how many are short-term, and then calculating the sell value or the gain or loss associated with it.
You don't have to limit yourself to this data analysis scenario for stock trading. Think about and extrapolate this to all other automation use cases where you might need to do numerical calculations or string manipulations. Think of those scenarios where you would be needing these capabilities but do not want to invest in a Lambda function. You could do it straight away as part of the Step Functions. Now you have this output where you have the transaction type, the total transactions, the sell value, and the gain or loss associated with that. This brings us to the simplified approach of strategy number two, where we are using optimized service integrations. In this case we are using SNS Publish for the optimized service integration, and we are again performing all the complex transformations using JSONata. You saw we did a lot of data aggregation as well as part of this.
Now does the story end here? That would be too simple. Let's complicate it a little more. Let's say we want to persist this data now into DynamoDB. Maybe at the end of the year the customer wants all the transactions that they made and they want it to be persisted to DynamoDB for that. Let's look at the input and the outputs associated with DynamoDB. A common problem that we have had when we have used DynamoDB in the past with Step Functions is that DynamoDB needs this exact structure in terms of how it is formatted for DynamoDB to be inserting the items. How do we write this JSONata function? Some of that sounds a little bit complicated. Let's look at a quick demo in terms of how we would create it and how I actually created it. As part of this demo, we see that we have the DynamoDB order input, and this is just the input from the last step that we have.
We'll open up the input structure that DynamoDB expects, which would be the task output but also an input to DynamoDB. In this case, we are going to be using Amazon Q, and this is my Visual Studio Code IDE. We're going to use Amazon Q embedded in it, and we'll write a simple natural language query saying that we should use this as the input, this is the output, and write a JSONata query.
It's simple, right? I just wanted to use single quotes instead of double quotes so that I can directly copy and paste the content without having to worry about replacing all of that. Amazon Q thinks about it, ponders about it, and then generates a JSONata query. Let's look at that. It generates this JSONata query. Now, how do we know it did it right? Do we know that? We don't know that. Let's take it and see how we could test it.
So we're going to take this JSONata query and then go back to our state machine that we have defined. In this case, we have a sample state machine. We want to edit it, and that will bring us to the sample screen that we have. In this case, we are going to use the BatchWriteItem and the test state button. We're going to use that test state button, and in the arguments we are going to pass in this JSONata that we had created, because that's what DynamoDB expects. So we'll put that in, and then we have the input that we were already showcasing. We use the start test, and that creates this structure, exactly the structure that DynamoDB expects to use.
So you saw that, in essence, I did nothing. It was all created by Amazon Q, and we were easily able to do this without using any Lambda function, without using anything else, and all of this within Step Functions itself. Now let's look at this right definition in a little more detail so that we have some of the things that you might have noticed that we do not want to miss out on.
One is the stock orders. The stock orders is essentially the DynamoDB table that we are using for the insertion. Then we have the dollar states dot input, which is the default input that the Step Function expects. Ideally, a Q developer creates it using dollar states. You can put in a prompt. When you put in the prompt, you can say use dollar states dot input. You can also use any variable if you have put the input into an input variable, you can use that variable instead of dollar states dot input.
The next item is inside the item. We have this unique ID, which is the UUID, the unique identifier that we can use as part of this DynamoDB insertions because we want every transaction to have that unique ID. It's a defined function that you already have. Then you map the rest of the stuff as string or number and you map it out. You can use functions for the present date or time, and there are lots of other user-defined functions.
So essentially, you get this input to DynamoDB that is created within that Step Functions task itself and then processed within that single step. Everything within Step Functions you're doing, and you're able to process all that information. Which brings us to the simplified approach of strategy number three. You're going to be making direct SDK calls and using JSONata to manipulate the payloads.
Benefits of Simplified Orchestrations and Applications in Generative AI Workflows
So three steps to simplify orchestrations: use JSONata to perform complex data manipulation, use SDK optimized integrations and HTTPS API calls where you can, and reduce the Lambda dependency. JSONata can eliminate a lot of transformation Lambda functions. You're able to eliminate Lambda. You can use your numerical calculations, payload manipulation, everything within Step Functions itself. And that brings us to the simple orchestration world as we would call it, and here we are revisiting the pain points that we discussed at the very beginning.
We are able to simplify maintenance because Lambda comes with ongoing runtime maintenance. You have to maintain the CI/CD pipelines. You have to take care of deprecation, which is very important. I think with most of the customers that I've worked with, Lambda deprecation is a serious pain point. So you're able to build your IT automation use cases using this without having to worry that you would have to go back and revisit Lambda deprecation or the language getting deprecated after a couple of years.
You also get lower latency because direct calls within Step Functions means that you have lower latency, and then lower cost because there's no additional Lambda costs that you have to pay. It's all within the same Step Functions. It's 2025. How can we not talk about GenAI, right?
How many of you have attended any GenAI session or agentic workflow session as part of this re:Invent? I see some hands. All right, so let's do a quick sneak peek. We won't go into too much detail, but this is a generative AI agentic workflow that we are building using Step Functions. Couple of things that I want you to gather from it. Firstly, this is definitely possible. You can build these workflows using Step Functions. These are autonomous workflows that you can build using Step Functions.
For this example particularly, this is a tool use architecture where we are using multiple tools to make decisions and the LLM makes decisions associated with what it needs to do. So this example is for a social media agent that repurposes long form content across multiple social media channels. Maybe LinkedIn needs it in a different format versus Facebook or maybe some other social media platform. It's able to summarize the content, post the content, and adapt the tone associated with what social media you are posting it to.
Now, what's the common point that we see across all of this based on what we have seen across all the GenAI lessons that we have learned so far? We are able to replace all these Lambda functions that we saw using the Bedrock runtime calls for the Converse API. You can still continue to use Lambda in a supervisor role, and that's a valid architecture. Using Lambda is a valid architecture if the scope gets too complicated. But at least you should not have to go back to Lambda for every single small transformation that you are thinking of doing.
You should be able to do pretty complex transformations within Step Functions itself. So the microservice architecture pattern would still apply, but at least you don't have to create a Lambda function out of everything. Let's look at what the ASL for this would look like. You have the arguments over here. You have the model ID, the messages, the system prompt that you're defining, everything within Step Functions, and you can define the output that you would want the tool to get the output for as a tool result as well with the JSON content and exactly what you want for the agentic workflow to work seamlessly.
So what do I want you to take away from this session? The only thing that I want you to take away from this session is this coffee pot that we are seeing on the screen. Well, no. Imagine what making coffee in the 1920s would have been. You would have to use a stovetop and a pot to boil the coffee, and before that you would have to grind the coffee and then make the coffee, decant it, and then put it in. You have all these separate pieces that are associated with it.
Think of this as simplifying orchestrations in a very similar manner. JSONata can eliminate a lot of transformations that Lambda functions can do. You have reduced latency and cost for data manipulation. You have improved workflow readability and maintainability, and all of this within native Step Functions itself. So what I want you to think of is a modern coffee machine which can simplify your orchestration route and ease out your automation very easily using JSONata.
And that marks the end of this talk today. I do have some resources for you. We have the JSONata documentation that you would definitely be needing if you want to make any changes with JSONata and also to see how much scope and how much variety it actually has in terms of functions that are already available. I have a blog link that also describes how you can go about using it and some ServerlessLand resources as well. And you can check out some of these other sessions as well. If you have not forayed into the world of AI agents, then these are some of the sessions as well as some developer productivity sessions that you can take a quick look at.
All right, and thank you. Please do complete the session survey in the mobile app, and I hope this was helpful. Thank you.
; This article is entirely auto-generated using Amazon Bedrock.
Top comments (0)