<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Ian</title>
    <description>The latest articles on DEV Community by Ian (@ianbrumby).</description>
    <link>https://dev.to/ianbrumby</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F910966%2F6cadb92c-e0dc-4298-b150-5440fd6d9237.jpg</url>
      <title>DEV Community: Ian</title>
      <link>https://dev.to/ianbrumby</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/ianbrumby"/>
    <language>en</language>
    <item>
      <title>4D Geospatial Search with DynamoDB Vectors</title>
      <dc:creator>Ian</dc:creator>
      <pubDate>Fri, 20 Nov 2026 22:26:46 +0000</pubDate>
      <link>https://dev.to/ianbrumby/4d-geospatial-search-with-dynamodb-vectors-2jid</link>
      <guid>https://dev.to/ianbrumby/4d-geospatial-search-with-dynamodb-vectors-2jid</guid>
      <description>&lt;p&gt;I’m a big fan of DynamoDB vector indexes. They fit a really good space when much of my data is already in DynamoDB.&lt;br&gt;
See the &lt;a href="https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/VectorSearch.html" rel="noopener noreferrer"&gt;Official Docs&lt;/a&gt; for more information.&lt;/p&gt;

&lt;p&gt;Most of the time, when we talk about vector stores, we are talking about high-dimensional (e.g. 1,024) stores, used in AI for embeddings for use cases like RAG semantic search. This is a great use case.&lt;/p&gt;

&lt;p&gt;I was keen to explore the idea of using a smaller number of dimensions (e.g. 4) to store geospatial and temporal data. This feels like a great use case for vector indexes where geospatial data in DynamoDB does not always feel like a natural fit. No AI involved.&lt;/p&gt;

&lt;p&gt;I'm starting this blog under the assumption that you know DynamoDB basics and have a basic understanding of vector stores. Everything else I'll build on from there.&lt;/p&gt;

&lt;p&gt;I’ve written before about &lt;a href="https://dev.to/ianbrumby/effective-handling-of-geospatial-data-in-dynamodb-1hmn"&gt;Effective Handling of Geospatial Data in DynamoDB&lt;/a&gt;, using a geospatial hash. This approach is still absolutely valid, but I want to try and take it to the next level with vector indexes, allowing me to support truly high-scale geospatial (and temporal) searches within my DynamoDB table.&lt;/p&gt;

&lt;p&gt;DynamoDB is a fantastic database, but as we know, you need to define your access patterns well to make the most of it. Seemingly unstructured queries such as "tell me things that are near this position" are not well supported. Vector queries have absolutely changed that.&lt;/p&gt;

&lt;p&gt;What I’d like to prove:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Are DynamoDB vector indexes suitable for geospatial / temporal data?&lt;/li&gt;
&lt;li&gt;Are they accurate?&lt;/li&gt;
&lt;li&gt;Are they fast!?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The easiest way to prove this: a demo project. As always, everything is built with CDK, TypeScript and is fully serverless.&lt;/p&gt;


&lt;h2&gt;
  
  
  The Demo: Flight Proximity
&lt;/h2&gt;

&lt;p&gt;I wanted to try that idea on a question where the dimensions really matter. Two flight paths can cross on a map even if the aircraft passed hours apart or at entirely different altitudes. Given one flight, which other aircraft passed close to it, at a similar altitude, at the same time? True four-dimensional queries.&lt;/p&gt;

&lt;p&gt;The code for this article can be found here, with a fully deployable project:&lt;/p&gt;

&lt;p&gt;👉 &lt;a href="https://github.com/Crockwell-Solutions/cdk-geospatial-dynamodb-vectors" rel="noopener noreferrer"&gt;GitHub Repository – DynamoDB Geospatial Vectors&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The example project uses 14,444 flights and 431,356 position reports over France on 1 October 2026. I've pre-processed the flight data from &lt;a href="https://github.com/adsblol" rel="noopener noreferrer"&gt;adsb.lol&lt;/a&gt; and included it in the example repo. ADSB.lol is a great open-source project for this type of data. Their data is made available under the &lt;a href="https://opendatacommons.org/licenses/odbl/1-0/" rel="noopener noreferrer"&gt;Open Database License (ODbL)&lt;/a&gt;, and so is the extract in the repo.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F0a0wt8wqibaj0bcaqbc1.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F0a0wt8wqibaj0bcaqbc1.png" alt="Flight Proximity App" width="799" height="417"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;What I built:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;A CDK project&lt;/li&gt;
&lt;li&gt;Backend: DynamoDB database, API Gateway and Lambda functions. No authentication&lt;/li&gt;
&lt;li&gt;Loading script: A script to process the flight data, loading the DynamoDB records&lt;/li&gt;
&lt;li&gt;Frontend: A React project, hosted on S3 and CloudFront, used to demo the functionality&lt;/li&gt;
&lt;/ul&gt;
&lt;h2&gt;
  
  
  The Architecture
&lt;/h2&gt;

&lt;p&gt;Everything is pretty much standard CDK, with the exception of adding the vector index itself. The L2 &lt;code&gt;Table&lt;/code&gt; construct does not expose vector indexes yet, so they are set on the L1 &lt;code&gt;CfnTable&lt;/code&gt; underneath:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;cfnTable&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;node&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;defaultChild&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="nc"&gt;CfnTable&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="n"&gt;cfnTable&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;vectorIndexes&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
  &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;indexName&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'position-4d'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;vectorAttribute&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;attributeName&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'vec'&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;dimensions&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;distanceFunction&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'EUCLIDEAN'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;projection&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;projectionType&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'ALL'&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;EUCLIDEAN&lt;/code&gt; is the only one of the three distance functions that means separation. &lt;code&gt;COSINE&lt;/code&gt; compares direction from the origin and ignores how far out a point is. The choice is permanent, so get it right the first time.&lt;/p&gt;

&lt;p&gt;And there is a minor issue with the permissions. The intent-based permission &lt;code&gt;grantReadData&lt;/code&gt; is not enough. Searching has its own IAM action, so this has to be added. I'm sure all this will get resolved in newer releases of CDK.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="nc"&gt;Grant&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;addToPrincipal&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;grantee&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;findNearbyFlights&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;actions&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;'dynamodb:SearchVectors'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;resourceArns&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sb"&gt;`${table.tableArn}/index/position-4d`&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The architecture looks like:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F0kkvy8ilu7omf7eh77jd.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F0kkvy8ilu7omf7eh77jd.png" alt="Architecture" width="799" height="373"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Turning the position reports into vectors
&lt;/h2&gt;

&lt;p&gt;Aviation is old-skool, measuring distances in nautical miles (NM) and altitude in feet. I'm oversimplifying here and standards vary, but in normal airspace, flights must be 5 nautical miles or 1,000 ft apart at any given time. Air Traffic Control (ATC) ensure this is the case, making sure everyone stays safe.&lt;/p&gt;

&lt;p&gt;Aircraft do not report x, y and z. They report longitude and latitude in degrees, altitude in feet and a time. The first job is to turn this into something that I can use with a vector search.&lt;/p&gt;

&lt;p&gt;I start with a series of position reports like this one:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;[ 2.82609,   49.29648,  37025,         1790866322.2 ]
  longitude  latitude   altitude (ft)  time (epoch seconds)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;I want to use &lt;strong&gt;Euclidean distance&lt;/strong&gt;, the familiar straight-line distance between two points. Giving the index that raw list would be a poor choice. A degree of longitude covers less ground than a degree of latitude over France, and changing altitude from feet to metres would change the ranking. I need to choose units that make differences across the four dimensions meaningful.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Component&lt;/th&gt;
&lt;th&gt;Calculation for this report&lt;/th&gt;
&lt;th&gt;Indexed value&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;x&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Project the coordinates onto the Lambert-93 grid. Measure kilometres east of the projection’s origin at 3° E, 46.5° N.&lt;/td&gt;
&lt;td&gt;&lt;code&gt;-12.653&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;y&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Take the northward component of the same projection, in kilometres.&lt;/td&gt;
&lt;td&gt;&lt;code&gt;310.777&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;z&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Multiply &lt;code&gt;37,025 ft × 0.00926&lt;/code&gt;, so a 1,000 ft difference carries the same weight as 5 nautical miles horizontally.&lt;/td&gt;
&lt;td&gt;&lt;code&gt;342.852&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;t&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Multiply &lt;code&gt;53,522.2&lt;/code&gt; seconds since midnight UTC by &lt;code&gt;0.25&lt;/code&gt;, roughly the distance in kilometres a jet travels in a second.&lt;/td&gt;
&lt;td&gt;&lt;code&gt;13380.55&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The altitude and time scales are design choices. Two reports from the same location and altitude, one minute apart, are &lt;strong&gt;15 units apart&lt;/strong&gt; in this index. They rank about as far apart as two simultaneous reports at the same altitude taken 15 km apart on the map. For other use cases, this would be different.&lt;/p&gt;

&lt;h2&gt;
  
  
  Storing the data
&lt;/h2&gt;

&lt;p&gt;Every position report is stored as one item, so a flight is about 30 items that share a partition key, with the report time as the sort key. The vector is an ordinary list attribute:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;"PK"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"FLIGHT#IBE08FA#2026-10-01"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"SK"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"POS#2026-10-01T14:52:02.200Z"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"lon"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;2.82609&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"lat"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;49.29648&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"alt"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;37025&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"ts"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;1790866322.2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"vec"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mf"&gt;-12.653&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;310.777&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;342.852&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;13380.55&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;There is no separate API for writing vectors. This goes in with a normal &lt;code&gt;BatchWriteItem&lt;/code&gt; and DynamoDB updates the index in the background. Items with no &lt;code&gt;vec&lt;/code&gt; attribute, such as the one summary item per flight, are simply left out of the index.&lt;/p&gt;

&lt;p&gt;There is a script that is used to load the data, converting and processing the data.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;npm run load-data
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Querying the data
&lt;/h2&gt;

&lt;p&gt;The whole encoding is one function:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;export&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;toVector&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;space&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;VectorSpace&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;point&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;FourDPoint&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;Vector4&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;y&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;projectToKm&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;point&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;lon&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;point&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;lat&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
    &lt;span class="nf"&gt;round3&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nf"&gt;round3&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;y&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nf"&gt;round3&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;point&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;alt&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;space&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;kmPerFoot&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="c1"&gt;// 9.26 km per 1,000 ft&lt;/span&gt;
    &lt;span class="nf"&gt;round3&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;point&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ts&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;space&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;epochSeconds&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;space&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;kmPerSecond&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="c1"&gt;// 0.25 km per second&lt;/span&gt;
  &lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A query uses &lt;code&gt;SearchVectorsCommand&lt;/code&gt; and takes four numbers for the four dimensions, converts them to the vector and returns the nearest stored reports:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;response&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;send&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;SearchVectorsCommand&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nc"&gt;TableName&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;tableName&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nc"&gt;IndexName&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'position-4d'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nc"&gt;SearchVector&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;toVector&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;space&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;point&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="c1"&gt;// [x, y, z, t]&lt;/span&gt;
    &lt;span class="nc"&gt;TopK&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nc"&gt;ProjectionExpression&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'#pk, #lon, #lat, #alt, #ts'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nc"&gt;ExpressionAttributeNames&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="s1"&gt;'#pk'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'PK'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'#lon'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'lon'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'#lat'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'lat'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'#alt'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'alt'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'#ts'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'ts'&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;TopK&lt;/code&gt; is how many results to return, and 100 is the most DynamoDB allows. That covers one point. A flight is a line, so the Lambda runs one search for each point along it, all in parallel, and notes every other flight that turns up.&lt;/p&gt;

&lt;h3&gt;
  
  
  The Limitations
&lt;/h3&gt;

&lt;p&gt;The flight's own reports turn up too, since nothing is nearer to a flight than itself. Filters on a vector index only support equality, so there is no way to say "not this flight". They are dropped in code.&lt;/p&gt;

&lt;p&gt;For my example use case, there is one big limitation. The aircraft are moving. And moving pretty fast. So with a report only every 75 seconds or so, it is not going to catch the exact closest approach.&lt;/p&gt;

&lt;p&gt;So the search is widened and given a second step. It also runs from points interpolated between the reports, at least one every 30 seconds, and uses a generous radius to produce a shortlist of flights. For each one, a single &lt;code&gt;Query&lt;/code&gt; loads its reports, both tracks are joined up as straight lines, and the exact closest approach is calculated. The index does the narrowing and a little geometry gives the answer.&lt;/p&gt;




&lt;h2&gt;
  
  
  Try it out
&lt;/h2&gt;

&lt;p&gt;The &lt;a href="https://github.com/Crockwell-Solutions/cdk-geospatial-dynamodb-vectors" rel="noopener noreferrer"&gt;project&lt;/a&gt; has the CDK stacks, the day of flight data, a loader, the evaluation script and the map application from the screenshot, which exists so that you can pick a flight and see the answer.&lt;/p&gt;

&lt;p&gt;You need Node.js 22.13 or later:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;git clone https://github.com/Crockwell-Solutions/cdk-geospatial-dynamodb-vectors.git
&lt;span class="nb"&gt;cd&lt;/span&gt; cdk-geospatial-dynamodb-vectors
npm ci &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; npm ci --prefix frontend
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;To deploy it, you need credentials for an account that has been &lt;a href="https://docs.aws.amazon.com/cdk/v2/guide/bootstrapping.html" rel="noopener noreferrer"&gt;bootstrapped for CDK&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;1. Deploy&lt;/strong&gt; the three stacks and note the &lt;code&gt;SiteUrl&lt;/code&gt; output:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;npm run deploy
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;2. Wait for the index.&lt;/strong&gt; A table can be &lt;code&gt;ACTIVE&lt;/code&gt; while its vector index is still being created, so the API has a health endpoint. Wait until &lt;code&gt;vectorIndex.status&lt;/code&gt; is &lt;code&gt;ACTIVE&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl https://YOUR_SITE_URL/api/health
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;3. Load the data.&lt;/strong&gt; This writes 445,801 items and takes a few minutes:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;npm run load-data
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;4. Verify.&lt;/strong&gt; Ask for the flight in the screenshot. You should get five encounters back, the closest being &lt;code&gt;JAF6LD&lt;/code&gt; at 0.19 NM and 986 ft:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl &lt;span class="s2"&gt;"https://YOUR_SITE_URL/api/flights/EWG9GB%232026-10-01/nearby?horizontalNm=5&amp;amp;verticalFt=1000"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then open the &lt;code&gt;SiteUrl&lt;/code&gt;, search for a callsign or click a track, and press &lt;strong&gt;Analyse&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;5. Clean up.&lt;/strong&gt; This removes everything, including the table:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;npm run destroy
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It costs very little. Vector indexes are billed per byte written and per byte searched, with a 1 KB minimum for each. At us-east-1 prices, loading the data is about 446,000 write request units ($0.28) plus 0.41 GB of vector writes at $0.52 per GB ($0.21), so around fifty cents. An analysis is a couple of hundred searches at $0.002 per GB, which is less than the Lambda invocation that runs them. Check the &lt;a href="https://aws.amazon.com/dynamodb/pricing/" rel="noopener noreferrer"&gt;pricing page&lt;/a&gt; for your region.&lt;/p&gt;




&lt;h2&gt;
  
  
  Accuracy
&lt;/h2&gt;

&lt;p&gt;A vector index is approximate, and each search only returns 100 results, so it is fair to ask whether anything gets missed. I measured it. For 1,000 flights I worked out the true answer by brute force, comparing each one against all 14,443 others, and then checked how many of those encounters the search found.&lt;/p&gt;

&lt;p&gt;At 5 NM and 1,000 ft there were 1,002 real encounters, and the search found every one of them. It only starts to miss when I cut &lt;code&gt;TopK&lt;/code&gt; right down and make "close" very wide. At 20 NM and 5,000 ft with just 10 results per search, it finds 82%. With 100, it is back to finding everything.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;npm run evaluate -- --flights 1000            &lt;span class="c"&gt;# the result above, about a minute&lt;/span&gt;
npm run evaluate -- --flights 1000 --sweep    &lt;span class="c"&gt;# also tries lower TopK values, about ten minutes&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;






&lt;h2&gt;
  
  
  Performance
&lt;/h2&gt;

&lt;p&gt;The performance is lightning fast. Querying for a flight makes parallel vector searches for each point along the flight, then returns and processes the results in the Lambda. All of this takes ~500ms.&lt;/p&gt;

&lt;p&gt;With DynamoDB, given the way we have structured the queries and by not using scans, this will scale up with almost zero impact. This is the real value.&lt;/p&gt;




&lt;h2&gt;
  
  
  Other Options
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;PostGIS.&lt;/strong&gt; A GIS engineer would reach for this, with good reason. Store each flight as a single 4D line with time as the fourth ordinate, and &lt;code&gt;ST_ClosestPointOfApproach&lt;/code&gt; and &lt;code&gt;ST_DistanceCPA&lt;/code&gt; return when and how close two tracks came. It is exact and it works on paths, so the sparse-report problem disappears. The price is a database server to run and the associated overhead.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;No index at all.&lt;/strong&gt; One day over France is 431,356 points. That fits in memory, where comparing one flight against all the others takes milliseconds. The index earns its place when the data does not fit in memory, is being written continuously, or already lives in DynamoDB for other reasons.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Geohashes.&lt;/strong&gt; My &lt;a href="https://dev.to/ianbrumby/effective-handling-of-geospatial-data-in-dynamodb-1hmn"&gt;earlier approach&lt;/a&gt; is still the a good tool for "what is in this area?". It struggles here because every extra dimension multiplies the work: finding what was near a single point means querying 9 neighbouring cells × 3 altitude bands × 3 time buckets, so 81 queries, and then throwing most of the results away.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;DynamoDB vector index&lt;/th&gt;
&lt;th&gt;PostGIS&lt;/th&gt;
&lt;th&gt;DynamoDB geohash keys&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;"What is nearest to this point?"&lt;/td&gt;
&lt;td&gt;Built in, approximate&lt;/td&gt;
&lt;td&gt;Built in, exact&lt;/td&gt;
&lt;td&gt;Many queries, then filter&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;"How close did two tracks come?"&lt;/td&gt;
&lt;td&gt;Your own code&lt;/td&gt;
&lt;td&gt;Built in&lt;/td&gt;
&lt;td&gt;Your own code&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;"What is inside this area?"&lt;/td&gt;
&lt;td&gt;Not supported&lt;/td&gt;
&lt;td&gt;Built in&lt;/td&gt;
&lt;td&gt;What it is designed for&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Filtering on a range, such as a time window&lt;/td&gt;
&lt;td&gt;No, exact matches only&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;td&gt;Yes, on the sort key&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;What you have to operate&lt;/td&gt;
&lt;td&gt;Nothing: no servers, and it scales on demand&lt;/td&gt;
&lt;td&gt;A PostgreSQL server, or Aurora Serverless&lt;/td&gt;
&lt;td&gt;Nothing: no servers, and it scales on demand&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Cost when nobody is using it&lt;/td&gt;
&lt;td&gt;Storage only&lt;/td&gt;
&lt;td&gt;The server, unless it can pause&lt;/td&gt;
&lt;td&gt;Storage only&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Use it when&lt;/td&gt;
&lt;td&gt;The data is already in DynamoDB and you need nearest neighbours&lt;/td&gt;
&lt;td&gt;You need real spatial analysis&lt;/td&gt;
&lt;td&gt;You need area queries at scale&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;




&lt;h2&gt;
  
  
  Conclusion
&lt;/h2&gt;

&lt;p&gt;So, back to the three questions.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Are DynamoDB vector indexes suitable for geospatial / temporal data?&lt;/strong&gt; 100%. It is four numbers on an ordinary attribute, with nothing extra to run. The thinking goes into choosing the scales, and into remembering that the index sees points, not paths.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Are they accurate?&lt;/strong&gt; 100%. Every one of the 1,002 real encounters was found.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Are they fast?&lt;/strong&gt; 100%. Around 500ms for a whole flight, and it will scale.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Given most of my workloads are serverless, this adds a significant capability to DynamoDB for my geospatial projects.&lt;/p&gt;




&lt;h2&gt;
  
  
  About Me
&lt;/h2&gt;

&lt;p&gt;I’m Ian, an AWS Serverless Specialist, AWS Community Builder and AWS Certified Cloud Architect based in the UK. I work as an independent consultant, having worked across multiple sectors, with a passion for Aviation.&lt;/p&gt;

&lt;p&gt;Let's connect on &lt;a href="https://www.linkedin.com/in/ibrumby" rel="noopener noreferrer"&gt;LinkedIn&lt;/a&gt;, or find out more about my work at &lt;a href="https://crockwell.com" rel="noopener noreferrer"&gt;Crockwell Solutions&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>aws</category>
      <category>serverless</category>
      <category>dynamodb</category>
      <category>vectordatabase</category>
    </item>
    <item>
      <title>Serverless Vector Stores on AWS</title>
      <dc:creator>Ian</dc:creator>
      <pubDate>Thu, 15 Oct 2026 09:55:39 +0000</pubDate>
      <link>https://dev.to/ianbrumby/serverless-vector-stores-on-aws-5800</link>
      <guid>https://dev.to/ianbrumby/serverless-vector-stores-on-aws-5800</guid>
      <description>&lt;p&gt;Even with the massive context windows of frontier models, vector stores still have their place. I recently needed to make a relatively small collection of private documents searchable within an existing serverless application. I wanted to retain control over that data and retrieve only the relevant document sections. Perfect use case for vector.&lt;/p&gt;

&lt;p&gt;Prior to 2026, it was still very expensive to build vector search on AWS, OpenSearch "Serverless" was not serverless at all and required a minimum spend of around &lt;strong&gt;$200 per month&lt;/strong&gt;, even in their "dev-test" mode. @aws, this is not serverless!&lt;/p&gt;

&lt;p&gt;Then, in true AWS style, we went from zero serverless vector stores to &lt;strong&gt;THREE&lt;/strong&gt; in the space of a year.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;OpenSearch Serverless &lt;strong&gt;NextGen&lt;/strong&gt; can now scale to zero.&lt;/li&gt;
&lt;li&gt;S3 Vectors&lt;/li&gt;
&lt;li&gt;Vector indexing in DynamoDB&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;All now are true serverless, scale to zero services. I'm a massive DynamoDB advocate, so the last one is the option I'm most excited about. I've tried all three out, and the TLDR is that actually, it is very hard to look past OpenSearch Serverless NextGen.&lt;/p&gt;

&lt;p&gt;This article walks through the differences, the costs, the tradeoffs, and a &lt;strong&gt;CDK TypeScript starter project&lt;/strong&gt; that loads the same document chunks into all three services and tests retrieval.&lt;/p&gt;

&lt;p&gt;The end goal:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Three serverless vector stores&lt;/strong&gt;, deployed from one project.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The same embeddings and source data&lt;/strong&gt; in each store, so we can compare retrieval sensibly.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Answers with source references&lt;/strong&gt;, using a small aviation knowledge base.&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  Introduction
&lt;/h2&gt;

&lt;p&gt;The code for this article can be found here:&lt;/p&gt;

&lt;p&gt;👉 &lt;a href="https://github.com/Crockwell-Solutions/cdk-serverless-vector-stores" rel="noopener noreferrer"&gt;GitHub Repository – Serverless Vector Stores&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The project demonstrates how to:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Deploy OpenSearch Serverless NextGen, S3 Vectors and a DynamoDB on-demand table with a native vector index.&lt;/li&gt;
&lt;li&gt;Extract and chunk a PDF, preserving references to the original pages.&lt;/li&gt;
&lt;li&gt;Generate embeddings with Amazon Bedrock and load them into all three stores.&lt;/li&gt;
&lt;li&gt;Retrieve relevant text, tables and chart descriptions.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;I'm comparing the technical options here, rather than presenting a formal benchmark. The repo includes tools for measuring retrieval, but in reality, this is more about the feasibility of these vector stores.&lt;/p&gt;




&lt;h2&gt;
  
  
  The Use Case — An Aviation Knowledge Base
&lt;/h2&gt;

&lt;p&gt;My starting point is an aviation knowledge base: being able to ask a question and find the relevant part of a document, with a reference I can follow back to the source.&lt;/p&gt;

&lt;p&gt;For this starter, I've kept things smaller: a &lt;strong&gt;nine-page extract from the Public UK Aeronautical Information Publication&lt;/strong&gt;, covering measurement conventions, altimeter procedures and Heathrow airport information.&lt;/p&gt;

&lt;p&gt;There is a useful mix of ordinary text, runway tables and airport charts. For example:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;em&gt;“Which units are used for runway lengths and altitudes?”&lt;/em&gt;&lt;/li&gt;
&lt;li&gt;&lt;em&gt;“What dimensions and surface are published for Heathrow runway 09L/27R?”&lt;/em&gt;&lt;/li&gt;
&lt;li&gt;&lt;em&gt;“Where is Terminal 4 relative to the southern runway on the airport chart?”&lt;/em&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The last question is particularly useful. Extracting text from a PDF doesn't necessarily preserve the relationship between a terminal, a runway and the rest of the diagram.&lt;/p&gt;

&lt;p&gt;I appreciate that nine pages isn't a lot. You could put the whole document into context window if you wanted. The point of this example is the retrieval pattern: finding the right evidence when your application grows to hundreds or thousands of documents, including private material the model hasn't seen.&lt;/p&gt;

&lt;p&gt;The current repo prepares one PDF per corpus. A larger document library would need an ingestion workflow around it. The same approach applies to engineering manuals, internal policies, product documentation or customer support material.&lt;/p&gt;




&lt;h2&gt;
  
  
  Comparing the Three Options
&lt;/h2&gt;

&lt;p&gt;My starting point would be:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;OpenSearch&lt;/strong&gt; when the application needs a broad search engine, including keyword and hybrid search.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;S3 Vectors&lt;/strong&gt; when storage cost matters most and queries are relatively infrequent.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;DynamoDB&lt;/strong&gt; when I want low-latency semantic retrieval alongside application data, particularly data already stored in DynamoDB.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Here's how AWS positions the three services:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Service&lt;/th&gt;
&lt;th&gt;Published latency guidance, P95&lt;/th&gt;
&lt;th&gt;Vector scale&lt;/th&gt;
&lt;th&gt;Query throughput guidance&lt;/th&gt;
&lt;th&gt;Main reason to consider it&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Amazon OpenSearch Service&lt;/td&gt;
&lt;td&gt;10–100 ms&lt;/td&gt;
&lt;td&gt;Hundreds of billions&lt;/td&gt;
&lt;td&gt;Thousands of queries per second&lt;/td&gt;
&lt;td&gt;Broad search capabilities: keywords, vectors and hybrid retrieval&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Amazon S3 Vectors&lt;/td&gt;
&lt;td&gt;More than 100 ms&lt;/td&gt;
&lt;td&gt;Billions across indexes&lt;/td&gt;
&lt;td&gt;Low hundreds of queries per second per index&lt;/td&gt;
&lt;td&gt;Economical storage for large collections with less frequent searches&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Amazon DynamoDB vector search&lt;/td&gt;
&lt;td&gt;Below 10 ms&lt;/td&gt;
&lt;td&gt;Trillions across partitions&lt;/td&gt;
&lt;td&gt;Thousands of queries per second per vector index partition; scales across partitions&lt;/td&gt;
&lt;td&gt;Fast retrieval alongside operational data&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Adapted from &lt;a href="https://aws.amazon.com/blogs/machine-learning/aws-vector-solutions-build-agentic-ai-where-your-data-lives/" rel="noopener noreferrer"&gt;AWS's vector solutions comparison&lt;/a&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  OpenSearch Serverless
&lt;/h3&gt;

&lt;p&gt;OpenSearch is the option I'd look at when semantic similarity is only part of the requirement.&lt;/p&gt;

&lt;p&gt;In an aviation document, an exact reference such as &lt;code&gt;EGLL AD 2.12&lt;/code&gt; can matter just as much as the meaning of the surrounding text. &lt;a href="https://docs.aws.amazon.com/opensearch-service/latest/developerguide/serverless-configure-neural-search.html" rel="noopener noreferrer"&gt;Hybrid search&lt;/a&gt; lets us combine keyword matching with vector similarity.&lt;/p&gt;

&lt;p&gt;The significant recent change is &lt;strong&gt;NextGen scale to zero&lt;/strong&gt;. With both minimum capacity settings at zero, the collection group can pause after ten minutes without requests. AWS documents &lt;strong&gt;10–30 seconds&lt;/strong&gt; to cold start on first search or indexing request, so the first query after an idle period needs to be considered separately from steady-state latency. See &lt;a href="https://docs.aws.amazon.com/opensearch-service/latest/developerguide/serverless-scale-to-zero.html" rel="noopener noreferrer"&gt;the scale-to-zero behaviour&lt;/a&gt;. In my experience, even with this small example, the cold start was around &lt;strong&gt;15 seconds&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;For this project, I'm just using vector search with cosine similarity. It doesn't exercise hybrid search.&lt;/p&gt;

&lt;h3&gt;
  
  
  S3 Vectors
&lt;/h3&gt;

&lt;p&gt;S3 Vectors is appealing when I have a lot of information to retain, but don't need to search every part of it continuously.&lt;/p&gt;

&lt;p&gt;You create a vector bucket and index, write embeddings with metadata, and query for similar vectors. There is no search cluster to size. You pay for storage and the operations you perform.&lt;/p&gt;

&lt;p&gt;For a large archive of manuals or historical documents, an extra fraction of a second in retrieval may be a perfectly reasonable trade-off. Particularly when generating the final answer takes much longer anyway.&lt;/p&gt;

&lt;p&gt;S3 Vectors also supports metadata filtering, including numeric comparisons and logical conditions. That gives us more flexibility than the equality filters currently documented for DynamoDB vector indexes. See &lt;a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-vectors-metadata-filtering.html" rel="noopener noreferrer"&gt;S3 Vectors metadata filtering&lt;/a&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  DynamoDB Vectors
&lt;/h3&gt;

&lt;p&gt;If you're already using DynamoDB, being able to keep embeddings with your application data is really quite unique. I'm really excited about this possibility, although for the purposes of this test use case, it is not actually relevant.&lt;/p&gt;

&lt;p&gt;The project writes ordinary DynamoDB items containing the chunk text, its source information and an embedding. A native vector index handles similarity search through &lt;code&gt;SearchVectors&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;The index uses the corpus identifier as its &lt;strong&gt;HASH scope&lt;/strong&gt;, with page and content type declared as inline filters. For a SaaS application, a tenant identifier could serve a similar search-scoping role.&lt;/p&gt;

&lt;p&gt;This makes DynamoDB an interesting fit for:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;RAG over existing DynamoDB data&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Per-tenant semantic search&lt;/strong&gt; in a SaaS application.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Agent memory and context retrieval&lt;/strong&gt;.&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  How the Example Works
&lt;/h2&gt;

&lt;p&gt;The retrieval pipeline is the same for all three stores:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Extract the PDF text&lt;/strong&gt;, retaining the physical page number.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Split it into chunks&lt;/strong&gt;, using roughly 1,800 characters with a 200-character overlap.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Generate embeddings&lt;/strong&gt; using Titan Text Embeddings V2 through Bedrock.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Cache and load the same 1,024-dimensional vectors&lt;/strong&gt; into each backend.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Embed the question&lt;/strong&gt; and retrieve similar chunks using cosine similarity.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F46i2iny9ayy72kykwrlj.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F46i2iny9ayy72kykwrlj.png" alt="AWS Serverless Vector Stores Architecture" width="799" height="407"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  Handling Charts and Tables
&lt;/h3&gt;

&lt;p&gt;For the visual examples, the project renders PDF pages as images and asks Amazon Nova Lite to describe them. Those descriptions are embedded using the same text embedding model.&lt;/p&gt;

&lt;p&gt;This is &lt;strong&gt;retrieval using image descriptions&lt;/strong&gt;. The vector stores aren't interpreting the chart themselves.&lt;/p&gt;

&lt;p&gt;When generating an answer, the application can also send the original retrieved page images to the model. That lets it inspect the table or diagram directly, although a poor description can still cause the retrieval stage to miss the right page.&lt;/p&gt;

&lt;h3&gt;
  
  
  Architecture
&lt;/h3&gt;

&lt;p&gt;The project consists of a &lt;strong&gt;CDK stack and a local TypeScript CLI&lt;/strong&gt;. Bedrock provides embeddings. The user interface is through a CLI that interacts directly to each vector store. There is no Bedrock Knowledge Base or frontend in this example.&lt;/p&gt;

&lt;p&gt;CDK provisions the OpenSearch collection, the DynamoDB table and its native vector index, the S3 Vectors bucket and index, and an ordinary S3 bucket for page images.&lt;/p&gt;

&lt;p&gt;After deployment, &lt;code&gt;init&lt;/code&gt; creates or validates the OpenSearch index mapping and validates the existing DynamoDB and S3 vector indexes. For DynamoDB, it also waits for &lt;code&gt;ACTIVE&lt;/code&gt; status with backfill complete and confirms that vector search succeeds.&lt;/p&gt;




&lt;h2&gt;
  
  
  Cost Comparison
&lt;/h2&gt;

&lt;p&gt;The first improvement is that &lt;strong&gt;all three options can avoid an idle compute floor&lt;/strong&gt; in the configurations used here. Storage remains billable. With OpenSearch NextGen, we also need to allow for the ten-minute idle window before compute stops.&lt;/p&gt;

&lt;p&gt;The figures below use &lt;strong&gt;us-east-1, in USD&lt;/strong&gt;.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Service&lt;/th&gt;
&lt;th&gt;Stored data&lt;/th&gt;
&lt;th&gt;Writes and searches&lt;/th&gt;
&lt;th&gt;What matters for the bill&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;S3 Vectors&lt;/td&gt;
&lt;td&gt;$0.06 per GB-month&lt;/td&gt;
&lt;td&gt;PUTs: $0.20 per GB, with a 128 KB minimum per request. Queries: $2.50 per million, plus tiered logical-index processing and applicable returned-data charges.&lt;/td&gt;
&lt;td&gt;Corpus size, index layout, query frequency and write batching&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;DynamoDB vectors, Standard&lt;/td&gt;
&lt;td&gt;$0.25 per GB-month for the index, plus base-table storage&lt;/td&gt;
&lt;td&gt;Vector index writes: $0.52 per GB. Search: $0.002 per GB processed/returned. Vector writes and searches each have a 1 KB minimum. Base-table writes are additional.&lt;/td&gt;
&lt;td&gt;Item and index size, updates, partitioning and actual search bytes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;OpenSearch Serverless NextGen&lt;/td&gt;
&lt;td&gt;Persisted index storage at the regional rate&lt;/td&gt;
&lt;td&gt;Indexing and search OCU time, metered per second; applicable GPU indexing is separate&lt;/td&gt;
&lt;td&gt;Capacity consumed while active, including time before the group sleeps&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The query charges need a little care. S3's processing calculation uses the logical index size and pricing tiers; non-filterable metadata is excluded. DynamoDB charges for the data its search processes and returns. Those are different calculations, so comparing a headline “price per GB processed” doesn't tell us which query will cost less.&lt;/p&gt;

&lt;h3&gt;
  
  
  A Simple Storage Example
&lt;/h3&gt;

&lt;p&gt;Assume &lt;strong&gt;10 GB of billable storage in each relevant layer&lt;/strong&gt;, retained for a full month:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;S3 Vectors:&lt;/strong&gt; 10 × $0.06 = &lt;strong&gt;$0.60 per month&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;DynamoDB Standard:&lt;/strong&gt; 10 GB in the vector index costs &lt;strong&gt;$2.50&lt;/strong&gt;. If the base table also bills 10 GB at $0.25 per GB-month, that adds &lt;strong&gt;$2.50&lt;/strong&gt;, giving &lt;strong&gt;$5.00 per month&lt;/strong&gt; before requests.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;OpenSearch NextGen:&lt;/strong&gt; price the stored index, then add the indexing and search capacity consumed while active. A corpus size alone doesn't determine that total.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This illustrates storage rates, not the billed size of identical embeddings in each service. Text, metadata and index representation change the amount stored. If the data already lives in DynamoDB, some base-table cost may already exist; adding embeddings and an index still adds usage.&lt;/p&gt;

&lt;p&gt;For a complete monthly example, &lt;a href="https://aws.amazon.com/s3/pricing/" rel="noopener noreferrer"&gt;AWS's S3 pricing page&lt;/a&gt; models 10 million vectors split across 40 indexes, one million monthly queries and replacement of one-sixth of the corpus each month. Its estimate is &lt;strong&gt;$11.38&lt;/strong&gt; for vector storage, PUTs and queries. That is AWS's example, not the bill for this project, and changing how those vectors are split across indexes changes the query calculation.&lt;/p&gt;

&lt;p&gt;Overall, the idle storage costs are low, unless your corpus is particularly huge, so it doesn't feel like cost should be the driving factor when selecting the vector store for a use case like this one. Worth noting that the vector store is only part of the cost. This example also uses Bedrock for embeddings and image descriptions. Model charges depend on the model and tokens used; see &lt;a href="https://aws.amazon.com/bedrock/pricing/" rel="noopener noreferrer"&gt;Bedrock pricing&lt;/a&gt;.&lt;/p&gt;




&lt;h2&gt;
  
  
  Getting Started
&lt;/h2&gt;

&lt;p&gt;Get started by cloning the repo and following the &lt;a href="https://github.com/Crockwell-Solutions/cdk-serverless-vector-stores#prerequisites" rel="noopener noreferrer"&gt;README&lt;/a&gt; covering permissions, Bedrock model access and CDK bootstrapping.&lt;/p&gt;

&lt;p&gt;Once deployed, use the CLI to interact wih the vector stores.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;npm run lab
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;E.g. embed the content and load all three backends:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;npm run lab &lt;span class="nt"&gt;--&lt;/span&gt; embed &lt;span class="nt"&gt;--vision&lt;/span&gt;
npm run lab &lt;span class="nt"&gt;--&lt;/span&gt; ingest
npm run lab &lt;span class="nt"&gt;--&lt;/span&gt; verify
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Example Queries
&lt;/h3&gt;

&lt;p&gt;Retrieve the same question from all three stores:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;npm run lab &lt;span class="nt"&gt;--&lt;/span&gt; query &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--question&lt;/span&gt; &lt;span class="s1"&gt;'What dimensions and surface are published for Heathrow runway 09L/27R?'&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--modality&lt;/span&gt; text
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Or select one backend and generate an answer using a chart:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;npm run lab &lt;span class="nt"&gt;--&lt;/span&gt; query &lt;span class="nt"&gt;--stores&lt;/span&gt; dynamodb &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--question&lt;/span&gt; &lt;span class="s1"&gt;'Where is Terminal 4 relative to the southern runway on the Heathrow aerodrome chart?'&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--modality&lt;/span&gt; image &lt;span class="nt"&gt;--answer&lt;/span&gt; &lt;span class="nt"&gt;--images&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Results include the retrieved content and physical page references within the nine-page demo PDF.&lt;/p&gt;

&lt;h3&gt;
  
  
  Comparing Retrieval
&lt;/h3&gt;

&lt;p&gt;The repo includes a benchmark command if you want to take the comparison further:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;npm run lab &lt;span class="nt"&gt;--&lt;/span&gt; benchmark &lt;span class="nt"&gt;--rounds&lt;/span&gt; 10
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It records retrieval latency, errors, recall against an exact local vector search, and whether the expected source pages were found. Query embedding and answer generation sit outside the retrieval timing.&lt;/p&gt;

&lt;p&gt;There are two useful questions here: &lt;strong&gt;did the store return the nearest vectors, and did those vectors lead us to the right evidence?&lt;/strong&gt; A good score on the first doesn't automatically answer the second.&lt;/p&gt;

&lt;p&gt;A small PDF also won't establish production throughput, so don't read too much into the benchmarks in this example repo.&lt;/p&gt;




&lt;h2&gt;
  
  
  Which Would I Choose?
&lt;/h2&gt;

&lt;p&gt;Although I started this exercise with a bias towards &lt;strong&gt;DynamoDB Vector Indexes&lt;/strong&gt;, I wouldn’t automatically choose them for every vector workload. My main hesitation is the additional design work. Designing access patterns for DynamoDB can already take some thought, particularly with single-table design, and vector search adds decisions around partitioning, filtering and which data to index. I’d want to be confident those requirements fit comfortably alongside the existing application. The lack of native full-text search also makes DynamoDB less attractive when I need hybrid search combining keyword and semantic results. For me, the strongest appeal is adding semantic retrieval to an existing DynamoDB application where the data is already well structured and managed. Keeping the content and its vectors together is a useful simplification.&lt;/p&gt;

&lt;p&gt;For a large document archive with occasional queries, I’d start by looking at &lt;strong&gt;S3 Vectors&lt;/strong&gt;, particularly if storage cost is a priority. The trade-off is retrieval latency, so I’d want to be comfortable that the response times suit the application.&lt;/p&gt;

&lt;p&gt;For a new application with broader search requirements, though, &lt;strong&gt;OpenSearch Serverless NextGen&lt;/strong&gt; is difficult to overlook. Its support for keyword, vector and hybrid search makes it a flexible starting point. Provided the cold starts are acceptable, it would probably be my default unless storage economics or data already held elsewhere gave me a reason to choose differently.&lt;/p&gt;




&lt;h2&gt;
  
  
  Next Steps
&lt;/h2&gt;

&lt;p&gt;This starter provides a foundation for experimenting with serverless retrieval. The next things I'd explore are:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;A larger document library&lt;/strong&gt;, with versioning and ingestion across multiple PDFs.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Hybrid retrieval&lt;/strong&gt;, particularly for exact document references. Natively supported for OpenSearch, but also investigating some form of non-native hybrid retrieval options in the other vector store options.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Richer relevance tests&lt;/strong&gt;, including questions the documents cannot answer.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Tenant authorisation&lt;/strong&gt;, applied before any retrieval request.&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  About Me
&lt;/h2&gt;

&lt;p&gt;I'm Ian, an AWS Serverless Specialist, AWS Community Builder and AWS Certified Cloud Architect based in the UK. I work as an independent consultant, having worked across multiple sectors, with a passion for Aviation.&lt;/p&gt;

&lt;p&gt;Let's connect on &lt;a href="https://www.linkedin.com/in/ibrumby" rel="noopener noreferrer"&gt;LinkedIn&lt;/a&gt;, or find out more about my work at &lt;a href="https://crockwell.com?utm_source=devto&amp;amp;utm_medium=devto" rel="noopener noreferrer"&gt;Crockwell Solutions&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>aws</category>
      <category>vectordatabase</category>
      <category>serverless</category>
    </item>
    <item>
      <title>Realtime Event-Driven Applications with AppSync Events and EventBridge Pipes</title>
      <dc:creator>Ian</dc:creator>
      <pubDate>Wed, 22 Oct 2025 14:09:31 +0000</pubDate>
      <link>https://dev.to/ianbrumby/-49io</link>
      <guid>https://dev.to/ianbrumby/-49io</guid>
      <description>&lt;p&gt;

&lt;/p&gt;
&lt;div class="ltag__link--embedded"&gt;
  &lt;div class="crayons-story "&gt;
  &lt;a href="https://dev.to/ianbrumby/realtime-event-driven-applications-with-appsync-events-and-eventbridge-pipes-1b27" class="crayons-story__hidden-navigation-link"&gt;Realtime Event-Driven Applications with AppSync Events and EventBridge Pipes&lt;/a&gt;


  &lt;div class="crayons-story__body crayons-story__body-full_post"&gt;
    &lt;div class="crayons-story__top"&gt;
      &lt;div class="crayons-story__meta"&gt;
        &lt;div class="crayons-story__author-pic"&gt;

          &lt;a href="/ianbrumby" class="crayons-avatar  crayons-avatar--l  "&gt;
            &lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F910966%2F6cadb92c-e0dc-4298-b150-5440fd6d9237.jpg" alt="ianbrumby profile" class="crayons-avatar__image"&gt;
          &lt;/a&gt;
        &lt;/div&gt;
        &lt;div&gt;
          &lt;div&gt;
            &lt;a href="/ianbrumby" class="crayons-story__secondary fw-medium m:hidden"&gt;
              Ian
            &lt;/a&gt;
            &lt;div class="profile-preview-card relative mb-4 s:mb-0 fw-medium hidden m:inline-block"&gt;
              
                Ian
                
              
              &lt;div id="story-author-preview-content-2944961" class="profile-preview-card__content crayons-dropdown branded-7 p-4 pt-0"&gt;
                &lt;div class="gap-4 grid"&gt;
                  &lt;div class="-mt-4"&gt;
                    &lt;a href="/ianbrumby" class="flex"&gt;
                      &lt;span class="crayons-avatar crayons-avatar--xl mr-2 shrink-0"&gt;
                        &lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F910966%2F6cadb92c-e0dc-4298-b150-5440fd6d9237.jpg" class="crayons-avatar__image" alt=""&gt;
                      &lt;/span&gt;
                      &lt;span class="crayons-link crayons-subtitle-2 mt-5"&gt;Ian&lt;/span&gt;
                    &lt;/a&gt;
                  &lt;/div&gt;
                  &lt;div class="print-hidden"&gt;
                    
                      Follow
                    
                  &lt;/div&gt;
                  &lt;div class="author-preview-metadata-container"&gt;&lt;/div&gt;
                &lt;/div&gt;
              &lt;/div&gt;
            &lt;/div&gt;

          &lt;/div&gt;
          &lt;a href="https://dev.to/ianbrumby/realtime-event-driven-applications-with-appsync-events-and-eventbridge-pipes-1b27" class="crayons-story__tertiary fs-xs"&gt;&lt;time&gt;Oct 21 '25&lt;/time&gt;&lt;span class="time-ago-indicator-initial-placeholder"&gt;&lt;/span&gt;&lt;/a&gt;
        &lt;/div&gt;
      &lt;/div&gt;

    &lt;/div&gt;

    &lt;div class="crayons-story__indention"&gt;
      &lt;h2 class="crayons-story__title crayons-story__title-full_post"&gt;
        &lt;a href="https://dev.to/ianbrumby/realtime-event-driven-applications-with-appsync-events-and-eventbridge-pipes-1b27" id="article-link-2944961"&gt;
          Realtime Event-Driven Applications with AppSync Events and EventBridge Pipes
        &lt;/a&gt;
      &lt;/h2&gt;
        &lt;div class="crayons-story__tags"&gt;
            &lt;a class="crayons-tag  crayons-tag--monochrome " href="/t/serverless"&gt;&lt;span class="crayons-tag__prefix"&gt;#&lt;/span&gt;serverless&lt;/a&gt;
            &lt;a class="crayons-tag  crayons-tag--monochrome " href="/t/aws"&gt;&lt;span class="crayons-tag__prefix"&gt;#&lt;/span&gt;aws&lt;/a&gt;
            &lt;a class="crayons-tag  crayons-tag--monochrome " href="/t/eventdriven"&gt;&lt;span class="crayons-tag__prefix"&gt;#&lt;/span&gt;eventdriven&lt;/a&gt;
            &lt;a class="crayons-tag  crayons-tag--monochrome " href="/t/appsync"&gt;&lt;span class="crayons-tag__prefix"&gt;#&lt;/span&gt;appsync&lt;/a&gt;
        &lt;/div&gt;
      &lt;div class="crayons-story__bottom"&gt;
        &lt;div class="crayons-story__details"&gt;
          &lt;a href="https://dev.to/ianbrumby/realtime-event-driven-applications-with-appsync-events-and-eventbridge-pipes-1b27" class="crayons-btn crayons-btn--s crayons-btn--ghost crayons-btn--icon-left"&gt;
            &lt;div class="multiple_reactions_aggregate"&gt;
              &lt;span class="multiple_reactions_icons_container"&gt;
                  &lt;span class="crayons_icon_container"&gt;
                    &lt;img src="https://assets.dev.to/assets/fire-f60e7a582391810302117f987b22a8ef04a2fe0df7e3258a5f49332df1cec71e.svg" width="18" height="18"&gt;
                  &lt;/span&gt;
                  &lt;span class="crayons_icon_container"&gt;
                    &lt;img src="https://assets.dev.to/assets/sparkle-heart-5f9bee3767e18deb1bb725290cb151c25234768a0e9a2bd39370c382d02920cf.svg" width="18" height="18"&gt;
                  &lt;/span&gt;
              &lt;/span&gt;
              &lt;span class="aggregate_reactions_counter"&gt;5&lt;span class="hidden s:inline"&gt;&amp;nbsp;reactions&lt;/span&gt;&lt;/span&gt;
            &lt;/div&gt;
          &lt;/a&gt;
            &lt;a href="https://dev.to/ianbrumby/realtime-event-driven-applications-with-appsync-events-and-eventbridge-pipes-1b27#comments" class="crayons-btn crayons-btn--s crayons-btn--ghost crayons-btn--icon-left flex items-center"&gt;
              

              &lt;span class="hidden s:inline"&gt;Add&amp;nbsp;Comment&lt;/span&gt;
            &lt;/a&gt;
        &lt;/div&gt;
        &lt;div class="crayons-story__save"&gt;
          &lt;small class="crayons-story__tertiary fs-xs mr-2"&gt;
            6 min read
          &lt;/small&gt;
            
              &lt;span class="bm-initial"&gt;
                

              &lt;/span&gt;
              &lt;span class="bm-success"&gt;
                

              &lt;/span&gt;
            
        &lt;/div&gt;
      &lt;/div&gt;
    &lt;/div&gt;
  &lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;




</description>
      <category>serverless</category>
      <category>aws</category>
      <category>eventdriven</category>
      <category>appsync</category>
    </item>
    <item>
      <title>Realtime Event-Driven Applications with AppSync Events and EventBridge Pipes</title>
      <dc:creator>Ian</dc:creator>
      <pubDate>Tue, 21 Oct 2025 19:33:49 +0000</pubDate>
      <link>https://dev.to/ianbrumby/realtime-event-driven-applications-with-appsync-events-and-eventbridge-pipes-1b27</link>
      <guid>https://dev.to/ianbrumby/realtime-event-driven-applications-with-appsync-events-and-eventbridge-pipes-1b27</guid>
      <description>&lt;p&gt;I want to share with you a powerful AWS serverless pattern that uses two of my favourite AWS services: &lt;strong&gt;AppSync Events&lt;/strong&gt; and &lt;strong&gt;EventBridge Pipes&lt;/strong&gt;. Using these services we can combine the power of event-driven architecture with realtime user notifications.&lt;/p&gt;

&lt;p&gt;This article walks through how to build a demo application called &lt;strong&gt;Airspace Alerter&lt;/strong&gt;, designed to showcase how these two services work together to deliver realtime user updates in a fully event-driven workflow.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fheumbs0x9k82rfqbx34b.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fheumbs0x9k82rfqbx34b.png" alt="Airspace Alerter" width="800" height="483"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Airspace Alerter is a map based web application, but this isn't a blog about geospatial data handling (although I'm passionate about that topic too!). This blog focuses on event management. If you’re interested in geospatial data in DynamoDB, check out my related post: &lt;a href="https://dev.to/ianbrumby/effective-handling-of-geospatial-data-in-dynamodb-1hmn"&gt;Effective Handling of Geospatial Data in DynamoDB&lt;/a&gt;&lt;/p&gt;




&lt;h2&gt;
  
  
  Introduction
&lt;/h2&gt;

&lt;p&gt;The full source code for this article can be found here:&lt;/p&gt;

&lt;p&gt;👉 &lt;a href="https://github.com/Crockwell-Solutions/cdk-appsync-events" rel="noopener noreferrer"&gt;GitHub Repository – Airspace Alerter - CDK AppSync Events&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;In the demo, users can:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Submit and store their planned flight routes.&lt;/li&gt;
&lt;li&gt;Simulate random airspace hazards (thunderstorms, drone sightings, restrictions).&lt;/li&gt;
&lt;li&gt;See realtime alerts whenever a hazard impacts one of their routes.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;These alerts are streamed back to the user through an open WebSocket connection using &lt;strong&gt;AppSync Events&lt;/strong&gt;.&lt;/p&gt;




&lt;h2&gt;
  
  
  The Core Services
&lt;/h2&gt;

&lt;p&gt;At the heart of this architecture are these two core AWS services:&lt;/p&gt;

&lt;h3&gt;
  
  
  1. EventBridge Pipes for Event Filtering and Enrichment
&lt;/h3&gt;

&lt;p&gt;EventBridge Pipes act as a simple, fully-managed bridge between sources and targets, allowing filtering, transformation, and enrichment of events as they flow through. They are an alternative and more observable way to process events in your application, rather than using an EventBridge Bus with rules.&lt;/p&gt;

&lt;p&gt;You can have multiple different sources for the pipes, but one of the most powerful is a DynamoDB stream. Streams is one the greatest features of DynamoDB, allowing for full Change Data Capture (CDC). This can be configured for all new, modified or deleted items to be passed to the pipes. The pipes allow a 'fan out' pattern where you can have multiple pipes, each with a specific filter. You only ever pay for a pipe when the filter matches and the pipe is invoked.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. AppSync Events for Realtime Event Delivery
&lt;/h3&gt;

&lt;p&gt;AppSync Events are AWS capability that provide realtime WebSocket event delivery. This is &lt;strong&gt;not&lt;/strong&gt; AppSync as you might know it. They come under the same service name, but AppSync GraphQL API (a great GraphQL API service) is completely separate from AppSync Events API (a great JSON based event driven websocket).&lt;/p&gt;

&lt;p&gt;AWS has two WebSocket services. API Gateway WebSocket API and AppSync Events. Think about AppSync events as API Gateway WebSocket APIs v2. WebSockets might not always be right for your application (see this &lt;a href="https://www.readysetcloud.io/blog/allen.helton/appsync-events/" rel="noopener noreferrer"&gt;great article by Allen Helton&lt;/a&gt;), but if they are, I would encourage you to use AppSync Events.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;As a side note, I'm not quite sure why AWS chose to put AppSync Events under the AppSync service. It doesn't naturally fit. I understand the need to consolidate services, but I tend to think of AppSync as purely AWS's GraphQL offering.&lt;/p&gt;
&lt;/blockquote&gt;




&lt;h2&gt;
  
  
  Architecture Overview
&lt;/h2&gt;

&lt;p&gt;The architecture looks like this:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fu261dbenqlwpbcsdsrs1.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fu261dbenqlwpbcsdsrs1.png" alt="Airspace Alerter Architecture" width="800" height="422"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;User submits a flight route via the web application.&lt;/li&gt;
&lt;li&gt;The route is stored in &lt;strong&gt;DynamoDB&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;The user can trigger random hazard events (e.g., a thunderstorm, drone sighting) through &lt;strong&gt;API Gateway&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;A &lt;strong&gt;DynamoDB Stream&lt;/strong&gt; captures the new hazard record and sends it through an &lt;strong&gt;EventBridge Pipe&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;The Pipe filters and then enriches the event: Fetching planned routes and checking if the hazard intersects with planned routes. If it does, it create an alert event.&lt;/li&gt;
&lt;li&gt;Both alert events and hazard events are passed to the target of the EventBridge pipe. The target is an &lt;strong&gt;EventBridge API Destination&lt;/strong&gt; which is configured to POST to the &lt;strong&gt;AppSync Events&lt;/strong&gt; endpoint.&lt;/li&gt;
&lt;li&gt;The connected user receives the hazards &amp;amp; alert in real time and sees it displayed on the map.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;&lt;strong&gt;EventBridge Pipes&lt;/strong&gt; handle filtering and enrichment. In our case, whenever a &lt;strong&gt;new&lt;/strong&gt; hazard is inserted into the table, the event is picked up by the Pipe. Before the event is sent downstream, the Pipe performs an &lt;strong&gt;enrichment step&lt;/strong&gt;, fetching flight routes from DynamoDB and determining whether any of them intersect with the hazard area.&lt;/p&gt;

&lt;p&gt;If a hazard impacts a route, the event is marked as an &lt;strong&gt;alert&lt;/strong&gt;. Both hazards and alerts are passed on to the target the pipe.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;AppSync Events&lt;/strong&gt; handle delivery to the web application. Clients simply subscribe once and wait for updates. AppSync Events handles some of the hidden complexity such as authentication, but more complex workflows requiring retries, specific authorisation will require additional logic that you build. Having said that, for a simple setup, it very easy to get working and the AWS Console provides some great tools to test and debug. Kudos to the AWS team here.&lt;/p&gt;




&lt;h2&gt;
  
  
  Building It with CDK
&lt;/h2&gt;

&lt;p&gt;The demo is built entirely with the &lt;strong&gt;AWS CDK&lt;/strong&gt;, which makes defining and connecting these event-driven components simple.&lt;/p&gt;

&lt;p&gt;The key constructs include:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;Pipe&lt;/code&gt; – EventBridge Pipe that defines the source (DynamoDB stream), enrichment (Lambda), and target (An API Destinations that publishes to AppSync Events). This is defined with an Alpha higher level construct &lt;a href="https://docs.aws.amazon.com/cdk/api/v2/docs/aws-pipes-alpha-readme.html" rel="noopener noreferrer"&gt;@aws-cdk/aws-pipes-alpha&lt;/a&gt;. Alpha constructs are subject to breaking changes, so please be aware of this as you build. If you need stability, you can instead use the lower level &lt;code&gt;cfnPipe&lt;/code&gt; construct.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;EventApi&lt;/code&gt; – defines the AppSync Events WebSocket API and connection management. This is a really great, simple construct for deploying an AppSync Event API.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;DynamoDB&lt;/code&gt; - Using a simple single-table design pattern for both the flight routes and the hazards. We can filter the stream in the EventBridge pipe if we are only interested in only one type of item.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;Lambda&lt;/code&gt; – to handle route submission, hazard simulation, and enrichment logic.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Here’s an example of the enrichment pipe in CDK:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Pipe&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;this&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;HazardEventsPipe&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="na"&gt;source&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;DynamoDBSource&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;props&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;airspaceAlerterTable&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="na"&gt;startingPosition&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;DynamoDBStartingPosition&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;LATEST&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="na"&gt;batchSize&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;}),&lt;/span&gt;
  &lt;span class="na"&gt;target&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;ApiDestinationTarget&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;airspaceAlertsAppsyncDestination&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
  &lt;span class="na"&gt;enrichment&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;LambdaEnrichment&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;props&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;hazardsEnrichmentFunction&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
  &lt;span class="na"&gt;logLevel&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;LogLevel&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;TRACE&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="na"&gt;logIncludeExecutionData&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nx"&gt;IncludeExecutionData&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;ALL&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
  &lt;span class="na"&gt;logDestinations&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nx"&gt;cwlLogDestination&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
  &lt;span class="na"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;sourceFilter&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;});&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This simple construct connects the DynamoDB stream, invokes the enrichment function, and publishes to the AppSync Events endpoint using a configured 'API Destination'. This is how the pipe looks once it is deployed:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Foym1i3x2idmdvr6pv2rw.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Foym1i3x2idmdvr6pv2rw.png" alt="EventBridge Pipe" width="800" height="476"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;For testing, you can log all the pipe actions and the execution data which allows for good observability of the pipe invocations.&lt;/p&gt;

&lt;p&gt;The AppSync Events API is deployed with just a few lines:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;apiKeyProvider&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;authorizationType&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;AppSyncAuthorizationType&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;API_KEY&lt;/span&gt; &lt;span class="p"&gt;};&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;eventsApi&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;EventApi&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;this&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;AirspaceAlerterEventsApi&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="na"&gt;apiName&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;AirspaceAlertsEvents&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="na"&gt;authorizationConfig&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;authProviders&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nx"&gt;apiKeyProvider&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="p"&gt;},&lt;/span&gt;
&lt;span class="p"&gt;});&lt;/span&gt;

&lt;span class="c1"&gt;// add a channel namespace called `alerts`&lt;/span&gt;
&lt;span class="nx"&gt;eventsApi&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;addChannelNamespace&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;alerts&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This gets you up and running with an API key for authentication. This is great for a demo, but in the real world, it also supports IAM, Cognito, OpenID Connect and custom authentication with Lambda.&lt;/p&gt;




&lt;h2&gt;
  
  
  Publishing Events
&lt;/h2&gt;

&lt;p&gt;In this demo application, we are using the HTTP endpoint to publish events, and this has been integrated into the application using &lt;code&gt;API Destinations&lt;/code&gt;, a feature of EventBridge.&lt;/p&gt;

&lt;p&gt;In Airspace Alerter, the Pipe target is the API Destinations, with an array of events output from the enrichment phase posted to the endpoint.&lt;/p&gt;

&lt;p&gt;However, you could also use a Lambda function as the destination from the pipe, for example, using IAM permissions to invoke the AppSync API rather than managing secrets.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;In the current way to create an API Destination with CDK, you must provide a secret from AWS Secrets Manager which contains the API key. API Destinations then creates an additional secret it uses to store a copy of this key and the other header information. If like me, this annoys you ($0.40 a month for each secret), the Lambda destination pattern avoids this for some minor overhead in additional code maintenance.&lt;/p&gt;
&lt;/blockquote&gt;




&lt;h2&gt;
  
  
  Getting Started
&lt;/h2&gt;

&lt;p&gt;To get started with the example project, clone the repo and deploy following the steps in the README.&lt;/p&gt;




&lt;h2&gt;
  
  
  Costs and Considerations
&lt;/h2&gt;

&lt;p&gt;Each component scales independently and costs only when used:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;DynamoDB Streams&lt;/strong&gt; are low-cost and event-based.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;EventBridge Pipes&lt;/strong&gt; are charged per event processed.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;AppSync Events&lt;/strong&gt; have lightweight connection-based pricing.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Lambda&lt;/strong&gt; only runs when invoked for enrichment or simulation.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The only caveat is the current AppSync Event destination pattern — as noted earlier, it relies on a &lt;strong&gt;Secret&lt;/strong&gt; to store the API key connection.&lt;/p&gt;




&lt;h2&gt;
  
  
  Next Steps
&lt;/h2&gt;

&lt;p&gt;Future enhancements might include:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Alternative, more secure authentication mechanisms (e.g. IAM, OIDC)&lt;/li&gt;
&lt;li&gt;Multi-user subscription models with fine-grained filters&lt;/li&gt;
&lt;li&gt;Use of IoT Core as a managed MQTT message broker instead of AppSync Events.&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  About Me
&lt;/h2&gt;

&lt;p&gt;I’m Ian, an AWS Serverless Specialist, AWS Community Builder, and AWS Certified Cloud Architect based in the UK.&lt;/p&gt;

&lt;p&gt;I work as an independent consultant with a passion for aviation, helping teams build cloud-native and event-driven solutions.&lt;/p&gt;

&lt;p&gt;Connect with me on &lt;a href="https://www.linkedin.com/in/ibrumby" rel="noopener noreferrer"&gt;LinkedIn&lt;/a&gt; or find out more at &lt;a href="https://crockwell.com" rel="noopener noreferrer"&gt;Crockwell Solutions&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>serverless</category>
      <category>aws</category>
      <category>eventdriven</category>
      <category>appsync</category>
    </item>
    <item>
      <title>Effective Handling of Geospatial Data in DynamoDB</title>
      <dc:creator>Ian</dc:creator>
      <pubDate>Mon, 08 Sep 2025 10:49:14 +0000</pubDate>
      <link>https://dev.to/ianbrumby/effective-handling-of-geospatial-data-in-dynamodb-1hmn</link>
      <guid>https://dev.to/ianbrumby/effective-handling-of-geospatial-data-in-dynamodb-1hmn</guid>
      <description>&lt;p&gt;Handling Geospatial data in DynamoDB doesn't feel like a natural fit, and can appear complex. With the right approach and some upfront work, you can leverage the superpowers of DynamoDB for your geospatial application.&lt;/p&gt;

&lt;p&gt;There are some really good existing patterns for handling geospatial data, in particular the &lt;a href="https://www.npmjs.com/package/dynamodb-geo" rel="noopener noreferrer"&gt;DynamoDB Geo Package&lt;/a&gt;. However, there are some limitations I wanted to avoid for my application. In particular, I wanted more control over the structure of my data, and I wanted to be able to return a geographically distributed set of results within a large result set without paginating. (woah, that's a mouthful... I'll explain what I mean further down)&lt;/p&gt;

&lt;p&gt;This article walks through how I achieved this using DynamoDB, with an example dataset of weather data from ~5,000 global airports. The  approach for structuring and querying the geospatial data includes making use of multiple geohash indexes and some de-clustering.&lt;/p&gt;

&lt;p&gt;The end goal:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Real-time queries&lt;/strong&gt; working at lightning speed at any precision level
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Uniform response time&lt;/strong&gt;, regardless of the bounding box size
&lt;/li&gt;
&lt;li&gt;Queries &lt;strong&gt;not impacted by the total dataset size&lt;/strong&gt;: No scan operations, only optimised queries.&lt;/li&gt;
&lt;li&gt;A &lt;strong&gt;random, well-distributed set of results&lt;/strong&gt;, rather than clustered data from a single corner of the bounding box&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  Introduction
&lt;/h2&gt;

&lt;p&gt;The code for this article can be found here:&lt;/p&gt;

&lt;p&gt;👉 &lt;a href="https://github.com/Crockwell-Solutions/cdk-dynamodb-geospatial" rel="noopener noreferrer"&gt;GitHub Repository – DynamoDB Geospatial Example&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The project demonstrates how to:  &lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Load real-world weather data into DynamoDB
&lt;/li&gt;
&lt;li&gt;Structure indexes for &lt;strong&gt;multi-level geohash precision&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;Support both large (worldwide) and small (town-level) bounding box queries&lt;/li&gt;
&lt;li&gt;Perform "points of interest along a route" type queries&lt;/li&gt;
&lt;li&gt;Return randomised results evenly distributed across space
&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  The Challenge with Geospatial in DynamoDB
&lt;/h2&gt;

&lt;p&gt;At its core, DynamoDB does not provide geospatial queries natively. You cannot simply say: &lt;em&gt;"Give me all points within this bounding box."&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;We need to break the problem down:  &lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Geohashing&lt;/strong&gt;: Encode latitude and longitude into a compact, ordered string.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Precision levels&lt;/strong&gt;: Wider bounding boxes use low-precision geohashes, smaller bounding boxes use higher precision.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Indexes&lt;/strong&gt;: Leverage DynamoDB Primary Partition and GSIs to query data at the right level of precision.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Partition prefixes&lt;/strong&gt;: Avoid hot partitions and enable randomised selection.&lt;/li&gt;
&lt;/ol&gt;




&lt;h2&gt;
  
  
  Table Design
&lt;/h2&gt;

&lt;p&gt;Here’s the design I used that allows for multi-precision level queries:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Partition Key&lt;/th&gt;
&lt;th&gt;Sort Key&lt;/th&gt;
&lt;th&gt;GSI1 Partition Key&lt;/th&gt;
&lt;th&gt;GSI1 Sort Key&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;PK&lt;/td&gt;
&lt;td&gt;SK&lt;/td&gt;
&lt;td&gt;GSI1PK&lt;/td&gt;
&lt;td&gt;GSI1SK&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;&amp;lt;ShardID&amp;gt;#&amp;lt;GeoHash Precision 1&amp;gt;&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;&amp;lt;GeoHash Precision 8&amp;gt;&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;&amp;lt;GeoHash Precision 4&amp;gt;&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;&amp;lt;GeoHash Precision 8&amp;gt;&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;And importantly:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The &lt;code&gt;ShardID&lt;/code&gt; is a random number (1–10 in the default case) that distributes data across partitions.&lt;/li&gt;
&lt;li&gt;We fan out to parallel queries, returning a random spread of results.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The precision levels configurable based on your use case, but under test, I found this structure to work well from very low to very high precision queries.&lt;/p&gt;

&lt;p&gt;The &lt;code&gt;ShardID&lt;/code&gt; is used on the main partition key, while the GSI1 allows for mid-level precision queries without the shard. This provides two benefits:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;We don't create a hot partition when using a very low precision geohash (e.g. world-level queries) on the main partition key.&lt;/li&gt;
&lt;li&gt;We are able to make higher precision queries (e.g. city-level) without needing to include the shard, which would require multiple queries to cover all shard values.&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  Query Types
&lt;/h2&gt;

&lt;p&gt;Depending on the zoom level of the bounding box, we dynamically adjust which keys/indexes are used:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;PK only&lt;/strong&gt; → World-scale queries&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;PK + SK&lt;/strong&gt; → Country or region queries&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;GSI1PK only&lt;/strong&gt; → City-level queries&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;GSI1PK + GSI1SK&lt;/strong&gt; → Town-level queries&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;This tiered approach ensures we always query at the right precision without over-fetching.&lt;/p&gt;




&lt;h2&gt;
  
  
  Randomised Result Distribution
&lt;/h2&gt;

&lt;p&gt;Without a partition prefix, DynamoDB queries naturally return clustered results (e.g. all from the "top-left" corner). By introducing a &lt;strong&gt;random ShardID prefix&lt;/strong&gt;, we ensure queries fan out across partitions and return a &lt;strong&gt;spread of points&lt;/strong&gt; across the bounding box.&lt;/p&gt;

&lt;p&gt;Sharding alone doesn't completely solve the clustering issue, especially for large bounding boxes with low precision geohashes. Therefore we further randomise results by gridding the bounding box into smaller sections and selecting results evenly from these grids. &lt;/p&gt;

&lt;p&gt;⚠️ &lt;strong&gt;Before&lt;/strong&gt; Querying world-scale without gridding → clustered data:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Funup76of63nlki3thjri.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Funup76of63nlki3thjri.png" alt=" " width="800" height="449"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;✅ &lt;strong&gt;After&lt;/strong&gt; Querying world-scale with partition prefixes and gridding → evenly spread data:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F7iluljsv50ld48gegpv6.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F7iluljsv50ld48gegpv6.png" alt=" " width="800" height="484"&gt;&lt;/a&gt;&lt;/p&gt;




&lt;h2&gt;
  
  
  Performance Tuning with Lambda
&lt;/h2&gt;

&lt;p&gt;Response times were tested response times using the truly excellent &lt;a href="https://github.com/alexcasalboni/aws-lambda-power-tuning" rel="noopener noreferrer"&gt;Lambda Power Tuning&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;At 1024MB, with large (county level) bounding box queries averaged ~124ms. Scaling memory up or down adjusted response time linearly without any unpredictable spikes.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fsurjdt2s04etpk255yf0.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fsurjdt2s04etpk255yf0.png" alt=" " width="800" height="419"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Example: Bounding Box Queries
&lt;/h2&gt;

&lt;p&gt;Here’s a comparison of bounding box sizes, the index used, and the performance on my ~5,000 record dataset:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Level&lt;/th&gt;
&lt;th&gt;Bounding Box&lt;/th&gt;
&lt;th&gt;Diagonal Distance (Km)&lt;/th&gt;
&lt;th&gt;Geohash Precision&lt;/th&gt;
&lt;th&gt;Index Used&lt;/th&gt;
&lt;th&gt;DDB Queries Made&lt;/th&gt;
&lt;th&gt;Avg Response Time (1024MB)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;World&lt;/td&gt;
&lt;td&gt;(80, -170), (-80, 170)&lt;/td&gt;
&lt;td&gt;17,845 km&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;PK&lt;/td&gt;
&lt;td&gt;320&lt;/td&gt;
&lt;td&gt;880 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Continent&lt;/td&gt;
&lt;td&gt;(70, -25), (40, 30)&lt;/td&gt;
&lt;td&gt;4,568 km&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;PK&lt;/td&gt;
&lt;td&gt;40&lt;/td&gt;
&lt;td&gt;150 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Country&lt;/td&gt;
&lt;td&gt;(43, -10), (37, 2)&lt;/td&gt;
&lt;td&gt;1,220 km&lt;/td&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;PK + SK&lt;/td&gt;
&lt;td&gt;40&lt;/td&gt;
&lt;td&gt;124 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Region&lt;/td&gt;
&lt;td&gt;(50, 8), (47, 13)&lt;/td&gt;
&lt;td&gt;497 km&lt;/td&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;td&gt;PK + SK&lt;/td&gt;
&lt;td&gt;150&lt;/td&gt;
&lt;td&gt;324 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;City&lt;/td&gt;
&lt;td&gt;(51.6, -0.45), (51.3, 0.2)&lt;/td&gt;
&lt;td&gt;56 km&lt;/td&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;GSI1&lt;/td&gt;
&lt;td&gt;9&lt;/td&gt;
&lt;td&gt;23 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Town&lt;/td&gt;
&lt;td&gt;(37.041, -7.995), (37.006, -7.901)&lt;/td&gt;
&lt;td&gt;9 km&lt;/td&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;GSI1PK + GSI1SK&lt;/td&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;td&gt;11 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Even world-scale queries return results in under a second, with evenly distributed points. There is also much more we could do here to optimise the Lambda function performance.&lt;/p&gt;

&lt;p&gt;This shows the approach delivers &lt;strong&gt;uniform response times regardless of bounding box size&lt;/strong&gt;.&lt;/p&gt;




&lt;h2&gt;
  
  
  Getting Started
&lt;/h2&gt;

&lt;p&gt;To get started with the example project:  &lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Clone the repo
&lt;/li&gt;
&lt;li&gt;Deploy the CDK stack (see README) &lt;/li&gt;
&lt;li&gt;Load the sample dataset (5,000 airports with METAR weather data)
&lt;/li&gt;
&lt;li&gt;Load up the frontend and try different bounding box queries&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;You’ll see that &lt;strong&gt;whether querying the whole world or a small town, response times remain fast and results are evenly distributed.&lt;/strong&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  Configuration
&lt;/h3&gt;

&lt;p&gt;The hashes and indexes used as configured within the environment configuration: &lt;code&gt;config/environment-config.ts&lt;/code&gt; and queries configuration are configured within the &lt;code&gt;config/geospatial-config.ts&lt;/code&gt; which includes the indexes and precision that should be used based on the size of the bounding box.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fpefno1hx1bzef9t83bu5.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fpefno1hx1bzef9t83bu5.png" alt=" " width="800" height="282"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  Architecture
&lt;/h3&gt;

&lt;p&gt;The demo project consists of two main components:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Frontend&lt;/strong&gt;: React-based web application served via CloudFront. Deployed with AWS CDK.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Backend&lt;/strong&gt;: Serverless API built with AWS Lambda and API Gateway, using DynamoDB for data storage and event driven architecture for real-time updates. Deployed with AWS CDK.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;AWS services consist of:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Amazon DynamoDB for geospatial data store&lt;/li&gt;
&lt;li&gt;AWS Lambda for serverless compute&lt;/li&gt;
&lt;li&gt;AWS API Gateway for REST API management&lt;/li&gt;
&lt;li&gt;Amazon S3 for static hosting of the frontend&lt;/li&gt;
&lt;li&gt;Amazon CloudFront for content delivery&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  API
&lt;/h3&gt;

&lt;p&gt;The API also includes a route endpoint that uses similar logic to the bounding box query and will return points of interest along a route. The payload response will including the total distance of the route:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Frcnfmxu1y5oxvbl1itt5.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Frcnfmxu1y5oxvbl1itt5.png" alt=" " width="800" height="1215"&gt;&lt;/a&gt;&lt;/p&gt;




&lt;h2&gt;
  
  
  Next Steps
&lt;/h2&gt;

&lt;p&gt;This approach provides a solid foundation for handling geospatial data in DynamoDB:  &lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Scalable to millions of records&lt;/li&gt;
&lt;li&gt;Uniform response times&lt;/li&gt;
&lt;li&gt;Randomised and well-distributed query results&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Future work could include:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Additional GSIs for intermediate precision levels&lt;/li&gt;
&lt;li&gt;Support for more complex geospatial queries (e.g. radius, nearest neighbor)&lt;/li&gt;
&lt;li&gt;Improved efficiency of the de-clustering. This is currently a very basic approach, and could be improved with more advanced algorithms.&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  About Me
&lt;/h2&gt;

&lt;p&gt;I’m Ian, an AWS Serverless Specialist, AWS Community Builder and AWS Certified Cloud Architect based in the UK. I work as an independent consultant, having worked across multiple sectors, with a passion for Aviation.&lt;/p&gt;

&lt;p&gt;Let's connect on &lt;a href="https://www.linkedin.com/in/ibrumby" rel="noopener noreferrer"&gt;LinkedIn&lt;/a&gt;, or find out more about my work at &lt;a href="https://crockwell.com" rel="noopener noreferrer"&gt;Crockwell Solutions&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>aws</category>
      <category>serverless</category>
      <category>dynamodb</category>
      <category>geospatial</category>
    </item>
    <item>
      <title>Finally got some time to play with the new JSONata and Variables support for Step Functions, and I have to say, it is massive improvement. Check out my latest blog post, where I walk through a simple example of how easy it is to handle pagination now</title>
      <dc:creator>Ian</dc:creator>
      <pubDate>Mon, 16 Dec 2024 19:07:55 +0000</pubDate>
      <link>https://dev.to/ianbrumby/finally-got-some-time-to-play-with-the-new-jsonata-and-variables-support-for-step-functions-and-i-3086</link>
      <guid>https://dev.to/ianbrumby/finally-got-some-time-to-play-with-the-new-jsonata-and-variables-support-for-step-functions-and-i-3086</guid>
      <description>&lt;div class="ltag__link"&gt;
  &lt;a href="/ianbrumby" class="ltag__link__link"&gt;
    &lt;div class="ltag__link__pic"&gt;
      &lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F910966%2F6cadb92c-e0dc-4298-b150-5440fd6d9237.jpg" alt="ianbrumby"&gt;
    &lt;/div&gt;
  &lt;/a&gt;
  &lt;a href="/ianbrumby/handling-paginated-results-seamlessly-with-aws-step-functions-42po" class="ltag__link__link"&gt;
    &lt;div class="ltag__link__content"&gt;
      &lt;h2&gt;Handling Paginated Results Seamlessly with AWS Step Functions&lt;/h2&gt;
      &lt;h3&gt;Ian ・ Dec 16&lt;/h3&gt;
      &lt;div class="ltag__link__taglist"&gt;
        &lt;span class="ltag__link__tag"&gt;#aws&lt;/span&gt;
        &lt;span class="ltag__link__tag"&gt;#serverless&lt;/span&gt;
        &lt;span class="ltag__link__tag"&gt;#stepfunctions&lt;/span&gt;
        &lt;span class="ltag__link__tag"&gt;#jsonata&lt;/span&gt;
      &lt;/div&gt;
    &lt;/div&gt;
  &lt;/a&gt;
&lt;/div&gt;


</description>
      <category>aws</category>
      <category>stepfunctions</category>
      <category>tutorial</category>
      <category>programming</category>
    </item>
    <item>
      <title>Handling Paginated Results Seamlessly with AWS Step Functions</title>
      <dc:creator>Ian</dc:creator>
      <pubDate>Mon, 16 Dec 2024 18:09:08 +0000</pubDate>
      <link>https://dev.to/ianbrumby/handling-paginated-results-seamlessly-with-aws-step-functions-42po</link>
      <guid>https://dev.to/ianbrumby/handling-paginated-results-seamlessly-with-aws-step-functions-42po</guid>
      <description>&lt;h2&gt;
  
  
  ...using Step Function Variables and JSONata
&lt;/h2&gt;

&lt;p&gt;Paginated results exist everywhere. Whether you are fetching data from an API, or iterating through database queries. It has been traditionally very difficult to deal with paginated results natively within your AWS Step Function workflows. The primary reason is that there was no out-of-the-box way to keep the state of the results and then append to it. That all changed when &lt;a href="https://aws.amazon.com/blogs/compute/simplifying-developer-experience-with-variables-and-jsonata-in-aws-step-functions/" rel="noopener noreferrer"&gt;AWS announced support for Step Function variables and JSONata&lt;/a&gt;. This made many people (myself included) very happy.&lt;/p&gt;




&lt;h2&gt;
  
  
  Introduction
&lt;/h2&gt;

&lt;p&gt;The code for this article can be found here:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://github.com/Crockwell-Solutions/cdk-stepfunctions-iterator" rel="noopener noreferrer"&gt;https://github.com/Crockwell-Solutions/cdk-stepfunctions-iterator&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;This CDK / TypeScript project gets you up and running with all the resources you need to iterate through some example records in DynamoDb using Step Functions. Also included is a seeding function to populate your DynamoDb table with 100 sample records to iterate through.&lt;/p&gt;

&lt;p&gt;It is worth noting that at the time of writing this blog, the CDK does not support JSONata based Step Functions using the native &lt;code&gt;fromChainable&lt;/code&gt; method. JSONata is supported using CDK when using ASL and the &lt;code&gt;fromFile&lt;/code&gt; method.&lt;/p&gt;




&lt;h2&gt;
  
  
  Step Function Variables and JSONata
&lt;/h2&gt;

&lt;p&gt;No more "Wait, where did my data go" &lt;code&gt;ResultPath&lt;/code&gt; or hours spent manipulating data in the Data Flow Simulator to conclude it is just not possible to add data to the top level of the JSON object with &lt;code&gt;OutputPath&lt;/code&gt;. The support for JSONata has completely eliminated the need. It is just &lt;code&gt;Arguments&lt;/code&gt; (input), &lt;code&gt;Output&lt;/code&gt; and optionally &lt;code&gt;Variables&lt;/code&gt; (see below). JSONata is a query and transformation language for JSON data. More information is available at the &lt;a href="https://jsonata.org" rel="noopener noreferrer"&gt;official guide&lt;/a&gt; and a cool &lt;a href="http://try.jsonata.org/" rel="noopener noreferrer"&gt;simulator&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;When a string in the value of an Step Function field, a JSON object field, or a JSON array element is surrounded by &lt;code&gt;{%&lt;/code&gt; &lt;code&gt;%}&lt;/code&gt; characters, that string will be evaluated as JSONata. It is that simple. JSONata supports basic operations such as selecting attributes and complex operations such as sorting and grouping.&lt;/p&gt;

&lt;p&gt;Step Function variables has complemented the support of JSONata by allowing assignment of variables for the Step Function rather than just for a task. This allows you to fetch and manipulate Step Function Variables through tasks.&lt;/p&gt;

&lt;p&gt;We will use both of these new bits of functionality within our Step Function Iterator.&lt;/p&gt;




&lt;h2&gt;
  
  
  The Architecture
&lt;/h2&gt;

&lt;p&gt;We are starting with a simple Step Function that will query DynamoDb. If there is a &lt;code&gt;nextToken&lt;/code&gt; provided in the response, it will continue to query and append to the Step Function Variable &lt;code&gt;$results&lt;/code&gt;. Before Step Functions Variables, this simple workflow was not possible like this.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F6jehpyzcnzz7a79n8kz2.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F6jehpyzcnzz7a79n8kz2.png" alt="Step Function Workflow" width="685" height="486"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Although we are focusing on DynamoDb as an example, this pattern equally applies to other paginated systems such as APIs that have a &lt;code&gt;limit&lt;/code&gt; and &lt;code&gt;page&lt;/code&gt; option.&lt;/p&gt;

&lt;p&gt;The key elements of the Step Function are the instantiation of the Step Function Variable in the &lt;code&gt;Query DynamoDb Table&lt;/code&gt; task:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="nl"&gt;"Assign"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"results"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"{% $states.result.Items %}"&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;$states.result&lt;/code&gt; is the output from the task, used within an JSONata expression indicated but the &lt;code&gt;{% %}&lt;/code&gt; encapsulation. And then we can now append to this array when we fetch the next page of results like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="nl"&gt;"Assign"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"results"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"{% $append($results, $states.result.Items) %}"&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Hooray to JSONata support 🎉&lt;/p&gt;




&lt;h2&gt;
  
  
  Getting Started
&lt;/h2&gt;

&lt;p&gt;To get started with the example project, clone the repo and deploy following the steps in the README. When you invoke the &lt;code&gt;SeedDynamoDbFunction&lt;/code&gt;, it will create 100 records in the DynamoDb table that we can iterate through.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fgql3r989rmoc6fcni596.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fgql3r989rmoc6fcni596.png" alt="Seeded DynamoDb Table" width="800" height="432"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;If we wanted to return a query a set of results where the PK is &lt;code&gt;ItemGroup&lt;/code&gt;, this could lead to paginated results (depending on the total size of the response).&lt;/p&gt;

&lt;p&gt;You can see from the output that the Step Function Variable &lt;code&gt;results&lt;/code&gt; now contains an array with the 100 DynamoDb results. The results were  fetched across 4 queries, where the limit of 30 was applied to each query. Winner!&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F9m5ru9rkewz55mld0yve.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F9m5ru9rkewz55mld0yve.png" alt="Completed Step Function" width="800" height="300"&gt;&lt;/a&gt;&lt;/p&gt;




&lt;h2&gt;
  
  
  Next Steps
&lt;/h2&gt;

&lt;p&gt;This simple pattern can be used to iterate results from alternative data sources. Prior to Step Function Variables and JSONata support, there was either the need for very complex task management, or more likely, you simply invoked a Lambda function to handle the logic. Now with native support, this is an effective way to get a complete result set in your Step Function workflow.&lt;/p&gt;




&lt;h2&gt;
  
  
  About Me
&lt;/h2&gt;

&lt;p&gt;I’m Ian, an AWS Serverless Specialist and AWS Certified Cloud Architect based in the UK. I work as an independent consultant, having worked across multiple sectors, with a passion for Aviation.&lt;/p&gt;

&lt;p&gt;Let's connect on &lt;a href="https://www.linkedin.com/in/ibrumby" rel="noopener noreferrer"&gt;linkedIn&lt;/a&gt;, or find out more about my work at &lt;a href="https://crockwell.com" rel="noopener noreferrer"&gt;Crockwell Solutions&lt;/a&gt;&lt;/p&gt;

</description>
      <category>aws</category>
      <category>serverless</category>
      <category>stepfunctions</category>
      <category>jsonata</category>
    </item>
    <item>
      <title>Query your EventBridge Scheduled Events in DynamoDB</title>
      <dc:creator>Ian</dc:creator>
      <pubDate>Sun, 24 Nov 2024 11:22:35 +0000</pubDate>
      <link>https://dev.to/ianbrumby/query-your-eventbridge-scheduled-events-in-dynamodb-2859</link>
      <guid>https://dev.to/ianbrumby/query-your-eventbridge-scheduled-events-in-dynamodb-2859</guid>
      <description>&lt;h1&gt;
  
  
  Querying Your EventBridge Scheduled Events
&lt;/h1&gt;

&lt;p&gt;EventBridge Scheduled Events are a fantastic addition to the AWS event driven ecosystem. However, they do have an issue... &lt;strong&gt;observability&lt;/strong&gt;. If you’re looking for a reliable way to track these events without the issues of the EventBridge API, I’ve got a practical solution for you.&lt;/p&gt;

&lt;h2&gt;
  
  
  Introduction
&lt;/h2&gt;

&lt;p&gt;The code for this article can be found here:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://github.com/Crockwell-Solutions/cdk-eventbridge-scheduler" rel="noopener noreferrer"&gt;https://github.com/Crockwell-Solutions/cdk-eventbridge-scheduler&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;This CDK / Typescript project gets you up and running with all the resources you need to observe your schedules in DynamoDB. Also included is a seeding function to demonstrate the end-to-end workflow.&lt;/p&gt;




&lt;h2&gt;
  
  
  What Are EventBridge Scheduled Events?
&lt;/h2&gt;

&lt;p&gt;Amazon EventBridge offers two types of schedules:  &lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Recurring (cron-like)&lt;/strong&gt;: These have been around for a while, letting you define periodic event triggers. Previously these were part of CloudWatch. AWS rightly moved this all under the EventBridge services.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;One-off schedules&lt;/strong&gt;: Introduced in 2022, these fire once and optionally delete themselves. They were the missing piece in event-driven systems.&lt;/li&gt;
&lt;/ol&gt;

&lt;h3&gt;
  
  
  Why One-off Schedules?
&lt;/h3&gt;

&lt;p&gt;One-off schedules are perfect for situations requiring time-sensitive actions:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;User notifications&lt;/strong&gt;: Trigger reminders at specific times based on user activity.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Time-bound processes&lt;/strong&gt;: Schedule tasks like price changes or temporary discounts to occur at a predefined time.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Temporary API Keys&lt;/strong&gt;: Generate API keys with a time-bound expiration, using a one-off schedule to trigger their deactivation at the exact expiration time.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For the purposes of this blog, I'm specifically referring to these one-off schedules.&lt;/p&gt;




&lt;h2&gt;
  
  
  The Problem with Tracking Scheduled Events
&lt;/h2&gt;

&lt;p&gt;EventBridge Scheduler allows you to manage thousands of future events, but it is not easy to get an overview of these events: To know what is planned to fire, or even what has fired in the past. There are some options.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;ListSchedules&lt;/code&gt; API Limitation&lt;/strong&gt;: The &lt;code&gt;ListSchedules&lt;/code&gt; API returns a maximum of 100 schedules per call. If you’re dealing with millions of events, this becomes impractical. There is also no out-of-the-box way to query these by firing time. You could incorporate the firing time in the schedule name allowing partial filtering using &lt;code&gt;NamePrefix&lt;/code&gt;. However, renaming schedules isn’t possible, so updating firing times means deleting and recreating schedules, adding unnecessary complexity for observability. Not ideal.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Custom Solutions&lt;/strong&gt;: You could just write back event metadata to the event origin. For example, you might want to schedule events for your customers based on their last activity in your platform. Say, schedule an event for 24 hours after they added an item to their basket to send them a reminder. You could just write back information about the scheduled event to the customer record, but what if you don’t want to keep updating customer records. You want to manage this schedule independently?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A completely different approach could be to DynamoDB TTL with Streams instead of EventBridge Scheduled events. You can configure your DynamoDB items with a Time-To-Live (TTL) attribute as the event firing time, allowing you to: &lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Store schedule metadata in DynamoDB.
&lt;/li&gt;
&lt;li&gt;Trigger downstream workflows using DynamoDB Streams when TTLs expire.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;However, TTL expiration isn’t precise enough for some strict timing requirements. AWS documentation states:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;“Items with valid, expired TTL attributes may be deleted by the system at any time, typically within a few days of their expiration.”  &lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;In my experience, it performs much better than that, but the point being, you can’t rely on the firing time.&lt;/p&gt;




&lt;h2&gt;
  
  
  A Robust Solution: CloudTrail and DynamoDB
&lt;/h2&gt;

&lt;p&gt;For a scalable and reliable way to track schedules, we can combine &lt;strong&gt;CloudTrail&lt;/strong&gt; and &lt;strong&gt;DynamoDB&lt;/strong&gt;:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;Monitor Scheduler Events with CloudTrail&lt;/strong&gt;&lt;br&gt;&lt;br&gt;
Enable CloudTrail logging for your AWS environment. It will capture events such as:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Schedule creation
&lt;/li&gt;
&lt;li&gt;Updates (e.g. firing time changes)
&lt;/li&gt;
&lt;li&gt;Deletion&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Capture Events in a DynamoDB Table&lt;/strong&gt;&lt;br&gt;&lt;br&gt;
Use EventBridge Rules to filter CloudTrail logs and write relevant events into a DynamoDB table.&lt;/p&gt;&lt;/li&gt;
&lt;/ol&gt;




&lt;h2&gt;
  
  
  Benefits of This Approach
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Comprehensive Tracking&lt;/strong&gt;: Easily query schedules and their firing times, even for millions of events.
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Historical Data&lt;/strong&gt;: Maintain a record of schedules, even after they’ve fired and have deleted themselves in EventBridge.
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Resilience&lt;/strong&gt;: With some further work, you could also store the payloads of the EventBridge Scheduled Events which could allow you to build a disaster recovery system in the event that AWS were to lose your Schedules!&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  The Architecture
&lt;/h2&gt;

&lt;p&gt;At the core of this architecture is CloudTrail. We leverage monitoring of the EventBridge Schedule API calls:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fhff6dafl4o5597p3djw9.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fhff6dafl4o5597p3djw9.png" alt="EventBridge Scheduled Events Architecture" width="632" height="423"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Independently to the core logic of the schedule creation, our service does the following steps:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;CloudTrail picks up the EventBridge API events and they are published on the main Event Bus&lt;/li&gt;
&lt;li&gt;An EventBridge rule filters for these events and pushes them to an SQS queue&lt;/li&gt;
&lt;li&gt;The queue will batch the messages and send them to the &lt;code&gt;ScheduleMonitorFunction&lt;/code&gt;. This Lambda will process the messages and create/update relevant records in DynamoDb&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;A queue is used during Step 2 because CloudTrail will not guarantee the order of events that are delivered. We need to ensure that changes are processed in order, so SQS will re-batch the messages, which are then sorted by the &lt;code&gt;eventTime&lt;/code&gt; as they are processed in the &lt;code&gt;ScheduleMonitorFunction&lt;/code&gt; Lambda Function.&lt;/p&gt;

&lt;p&gt;To see how this works, deploy the demo repo and run the &lt;code&gt;SeedSchedulesFunction&lt;/code&gt;. This creates three scheduled events, waits a few seconds then modifies one of them and deleted one of them. This demonstrate the functionality of the solution.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;aws lambda invoke &lt;span class="nt"&gt;--function-name&lt;/span&gt; SeedSchedulesFunction outfile.txt
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F9e4xhsf70yeao452mdmu.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F9e4xhsf70yeao452mdmu.png" alt="EventBridge Scheduled Events" width="800" height="324"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Event metadata is stored in DynamoDB using the following structure:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Primary Key&lt;/strong&gt;: &lt;code&gt;PK = SCHEDULEGROUP#SCHEDULENAME&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Global Secondary Index (GSI)&lt;/strong&gt;: &lt;code&gt;GSI1 = groupName, fireTime&lt;/code&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This design lets you query events by group and firing time, enabling efficient lookups for upcoming schedules within specific time windows:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fb7ey092mji6khuasn9j2.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fb7ey092mji6khuasn9j2.png" alt="DynamoDB Query" width="800" height="449"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Items are stored with a Time to Live (TTL) so they will expire and be removed from DynamoDB 30 days after the firing time.&lt;/p&gt;




&lt;h2&gt;
  
  
  Limitations and Considerations
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Fired Events&lt;/strong&gt;: This solution won’t explicitly track fired events, but since AWS guarantees “at least once delivery,” it’s safe to assume fired schedules have actually been fired.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Storage Costs&lt;/strong&gt;: DynamoDB storage and CloudTrail logs will incur additional costs, so optimise your queries and retention policies.&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  Next Steps
&lt;/h2&gt;

&lt;p&gt;This pattern is extensible for further uses. Options that could be considered include:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Analytics&lt;/strong&gt;: Push data to Redshift, S3, or Athena for advanced querying and analytics.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Payload Recovery&lt;/strong&gt;: Store payloads of scheduled events for disaster recovery or debugging.&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  About Me
&lt;/h2&gt;

&lt;p&gt;I’m Ian, an AWS Serverless Specialist and AWS Certified Cloud Architect based in the UK. I work as an independent consultant, having worked across multiple sectors, with a passion for Aviation.&lt;/p&gt;

&lt;p&gt;Let's connect on &lt;a href="https://www.linkedin.com/in/ibrumby" rel="noopener noreferrer"&gt;linkedIn&lt;/a&gt;, or find out more about my work at &lt;a href="https://crockwell.com" rel="noopener noreferrer"&gt;Crockwell Solutions&lt;/a&gt;&lt;/p&gt;

</description>
      <category>aws</category>
      <category>serverless</category>
      <category>eventdriven</category>
      <category>cdk</category>
    </item>
    <item>
      <title>Effortless Debugging: AWS CDK TypeScript Projects in VSCode</title>
      <dc:creator>Ian</dc:creator>
      <pubDate>Sun, 27 Oct 2024 23:29:15 +0000</pubDate>
      <link>https://dev.to/ianbrumby/effortless-debugging-aws-cdk-typescript-projects-in-vscode-5hj4</link>
      <guid>https://dev.to/ianbrumby/effortless-debugging-aws-cdk-typescript-projects-in-vscode-5hj4</guid>
      <description>&lt;p&gt;Debugging Lambda functions &lt;em&gt;should&lt;/em&gt; be easy, right? You want to debug as you develop, using a standard editor (VSCode), a popular IaC framework (AWS CDK), and a widely adopted language (TypeScript).&lt;/p&gt;

&lt;p&gt;But it is never quite &lt;em&gt;that&lt;/em&gt; easy.&lt;/p&gt;

&lt;p&gt;One of the main challenges to wider serverless adoption is the developer experience itself. Setting up an effective debugging environment for Lambda functions can be challenging, especially with the variety of tools, frameworks, and configurations available.&lt;/p&gt;

&lt;p&gt;If you're using the same stack as I am, this guide will help you set up a smooth, flexible debugging environment so you can iterate through Lambda functions like a pro.&lt;/p&gt;

&lt;h2&gt;
  
  
  Introduction
&lt;/h2&gt;

&lt;p&gt;The code for this article can be found here:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://github.com/Crockwell-Solutions/cdk-lambda-debugger" rel="noopener noreferrer"&gt;https://github.com/Crockwell-Solutions/cdk-lambda-debugger&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The starter project provides a CDK configuration with an example resource stack (API Gateway -&amp;gt; Lambda -&amp;gt; SQS). The Lambda function is ready for debugging using both AWS SAM and Lambda Live Debugger. With this setup, you can iterate quickly through Lambda changes without any deployment delays.&lt;/p&gt;

&lt;h2&gt;
  
  
  Debugging Goals
&lt;/h2&gt;

&lt;p&gt;When I built this, I wanted to simplify Lambda function debugging for a better developer experience. My main requirements were:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Quick debugging (&amp;lt;5 seconds)&lt;/strong&gt;: Debugging has got to be quick. I don’t want to run cdk synth every time I debug: Within a large project with multiple stacks and functions, this takes too long.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;No deployment or hot-swap requirement&lt;/strong&gt;: I want to debug without deploying each time.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Support for ephemeral environment&lt;/strong&gt;: Within a team, I need the setup to support multiple developer environments. I want it to work with SSO and I don't want to commit local environment variables.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Local debugging with remote resources&lt;/strong&gt;: I don't want to emulate resources locally, just the Lambda runtime.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This makes things a little trickier. There are plenty of options out there, but I really just needed a starter project with everything configured. I couldn't find one, so I've made one.&lt;/p&gt;

&lt;h2&gt;
  
  
  How it works
&lt;/h2&gt;

&lt;p&gt;I want the most flexible developer experience, so the sample project uses two different debugging options.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Run function locally, debug locally&lt;/strong&gt;: Using &lt;a href="https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-using-debugging.html" rel="noopener noreferrer"&gt;AWS SAM&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Run function remotely, debug locally&lt;/strong&gt;: Using &lt;a href="https://www.lldebugger.com" rel="noopener noreferrer"&gt;Lambda Live Debugger&lt;/a&gt;
&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;There is value in having both options available within a project as the SAM setup provides a bit more flexibility during early development for payloads and environment variables. Lambda Live Debugger provides a more representative runtime environment once you have your Lambda function is slightly more mature.&lt;/p&gt;

&lt;h4&gt;
  
  
  Getting Started
&lt;/h4&gt;

&lt;p&gt;To get started with the sample project, clone the &lt;a href="https://github.com/Crockwell-Solutions/cdk-lambda-debugger" rel="noopener noreferrer"&gt;sample project&lt;/a&gt; and follow the installation instruction in the README.&lt;/p&gt;

&lt;p&gt;After deployment, configure the required environment variables in a &lt;code&gt;local.env.json file&lt;/code&gt;. Here’s an example of what that file might look like:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;{
  "SQS_QUEUE_URL": "https://sqs.region.amazonaws.com/123456789012/LambdaDebuggerQueue"
}
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This file isn’t committed to version control, so each developer can set their own local environment variables. The &lt;code&gt;local.env.json&lt;/code&gt; file is referenced from &lt;code&gt;launch.json&lt;/code&gt; in VSCode using the SAM CLI argument &lt;code&gt;--container-env-vars&lt;/code&gt;, which does then allow for committing and sharing of the &lt;code&gt;launch.json&lt;/code&gt;.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;{
  "type": "aws-sam",
  "request": "direct-invoke",
  "name": "Lambda Function - SAM Debugger",
  "preLaunchTask": "Build",
  "invokeTarget": {
    "target": "code",
    "architecture": "arm64",
    "projectRoot": "dist",
    "lambdaHandler": "lambda-debugging-function.handler"
  },
  "aws": {
    "region": "region"
  },
  "lambda": {
    "runtime": "nodejs20.x",
    "payload": {
      "path": "${workspaceFolder}/test/test-payload.json"
    }
  },
  "sam": {
    "localArguments": [
      "--container-env-vars",
      "${workspaceFolder}/local.env.json"
    ]
  }
}
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The project includes a reusable &lt;code&gt;CustomLambda&lt;/code&gt; construct based on the CDK &lt;code&gt;NodejsFunction&lt;/code&gt;, bundling with ESBuild and including Lambda Powertools. This bundling is handled as a preLaunchTask in &lt;code&gt;launch.json&lt;/code&gt;, avoiding the need to perform a &lt;code&gt;cdk synth&lt;/code&gt;. This saves crucial seconds when initiating the debug run.&lt;/p&gt;

&lt;h4&gt;
  
  
  AWS SAM Debugging
&lt;/h4&gt;

&lt;p&gt;This uses AWS SAM to invoke the Lambda function locally. Select &lt;code&gt;Lambda Function - SAM Debugger&lt;/code&gt; from the Run and Debug menu in VSCode and hit F5. You should see the logging output.&lt;/p&gt;

&lt;p&gt;The debugger launch configuration triggers a build phase to bundle the code with ESBuild. Bundled code is saved to the &lt;code&gt;dist&lt;/code&gt; folder and run locally, removing the need to synth the CDK project. You can set breakpoints in &lt;code&gt;src/lambda-debugging-function.ts&lt;/code&gt;, and the debugger will respect source maps to link to the original code.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fys7btmoojydd8an9xn59.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fys7btmoojydd8an9xn59.png" alt="Debugging with AWS SAM" width="800" height="420"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h4&gt;
  
  
  Lambda Live Debugging
&lt;/h4&gt;

&lt;p&gt;Lambda Live Debugger is an alternative debugging option and offers the added benefit of validating permissions set up in CDK by running the Lambda functions remotely. This uses the &lt;a href="https://www.lldebugger.com" rel="noopener noreferrer"&gt;Lambda Live Debugger&lt;/a&gt; project led by the industry hero Marko at &lt;a href="https://www.serverlesslife.com" rel="noopener noreferrer"&gt;Serverless Life&lt;/a&gt;.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Deploy Lambda Live Debugger resources to your AWS account with &lt;code&gt;npx lld&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Set any custom debugger options in &lt;code&gt;lldebugger.config.ts&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;In VSCode, select &lt;code&gt;Lambda Function - Lambda Live Debugger&lt;/code&gt; from the Run and Debug menu, then press F5.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Lambda Live Debugger will work its magic and deploy out the required resources to your account. Now, to initiate the debug environment, you need to invoke the remote Lambda function. For example, you can add your breakpoint to your local code in &lt;code&gt;src/lambda-debugging-function.ts&lt;/code&gt; and then invoke the Lambda function through the API with a call such as:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;curl --request POST --header "Content-Type: application/json" --data '{"message": "This is a test message"}' https://xxxxxxxx.execute-api.eu-west-1.amazonaws.com/api/data
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fukc40vdydf4xvjqxptb5.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fukc40vdydf4xvjqxptb5.png" alt="Debugging with Lambda Live Debugger" width="800" height="439"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;In case of code changes, Lambda Live Debugger automatically reloads the updated code without the need to redeploy or restart the debugger. Magic!&lt;/p&gt;

&lt;p&gt;When you’re done, remove the debugger resources with &lt;code&gt;npx lld -r&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Wrapping Up
&lt;/h2&gt;

&lt;p&gt;This is still an evolving area, new tools and best practice changes all the time.&lt;/p&gt;

&lt;p&gt;I initially used Serverless Framework (v3) for debugging, but with its deprecation and the new signup requirement in v4, I’ve moved to the setup in this article. Running Lambda functions locally always risks slight discrepancies with deployed environments (especially in terms of permissions), but this approach with AWS SAM and Lambda Live Debugger balances speed with reliability.&lt;/p&gt;

&lt;h2&gt;
  
  
  About Me
&lt;/h2&gt;

&lt;p&gt;I’m Ian, an AWS Serverless Specialist and AWS Certified Cloud Architect based in the UK. I work as an independent consultant, having worked across multiple sectors, with a passion for Aviation.&lt;/p&gt;

&lt;p&gt;Let's connect on &lt;a href="https://www.linkedin.com/in/ibrumby" rel="noopener noreferrer"&gt;linkedIn&lt;/a&gt;, or find out more about my work at &lt;a href="https://crockwell.com" rel="noopener noreferrer"&gt;Crockwell Solutions&lt;/a&gt;&lt;/p&gt;

</description>
      <category>aws</category>
      <category>serverless</category>
      <category>cdk</category>
      <category>vscode</category>
    </item>
  </channel>
</rss>
