DEV Community: Muhammed Ashraf

Closer look on DynamoDB vector search vs S3 Vectors

Muhammed Ashraf — Sun, 09 Aug 2026 10:31:28 +0000

few days ago, AWS announced availability of new feature for DynamoDB which is DynamoDB vector search DynamoDB now supports vector search.

the new feature allows you to store vector embeddings alongside your data, this will allow you to run similarity search without the replication overhead of your data

as mentioned by AWS its single digit millisecond latency adding to this there is no storage limits for vector indexes as its growth whenever your operational data growth

in this article we will dig deeper into this feature and will have a comparison with S3 vectors and a use case for each one of them so let's first start with a walkthrough for the DynamoDB vector search

Setup & Implementation steps

first, we used the below terraform script which provision the ifrastructure on AWS contains creation of S3 vectors & DynamoDB table

terraform

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 6.24.0"
    }
  }
}

provider "aws" {
  region = "us-east-1"
}

# -----------------------------------------------------------------------------
# 1. DYNAMODB TABLE INFRASTRUCTURE
# -----------------------------------------------------------------------------
# Note: Amazon DynamoDB native vector search is a newly announced feature.
# The `hashicorp/aws` Terraform provider does not yet include a native `vector_index` block.
# Embeddings can be stored in standard item attributes, or index creation can be configured via AWS CLI/SDK.
resource "aws_dynamodb_table" "dynamo_vector_table" {
  name         = "VectorBenchmarkDynamo"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "doc_id"

  attribute {
    name = "doc_id"
    type = "S"
  }
}

# -----------------------------------------------------------------------------
# 2. AMAZON S3 VECTORS INFRASTRUCTURE
# -----------------------------------------------------------------------------
resource "aws_s3vectors_vector_bucket" "s3_vector_bucket" {
  vector_bucket_name = "vector-benchmark-s3-bucket"
}

resource "aws_s3vectors_index" "s3_vector_index" {
  vector_bucket_name = aws_s3vectors_vector_bucket.s3_vector_bucket.vector_bucket_name
  index_name         = "s3-vector-benchmark-index"
  dimension          = 1024
  data_type          = "float32"
  distance_metric    = "cosine"
}

# -----------------------------------------------------------------------------
# OUTPUTS
# -----------------------------------------------------------------------------
output "dynamodb_table_name" {
  value = aws_dynamodb_table.dynamo_vector_table.name
}

output "s3_vector_bucket_name" {
  value = aws_s3vectors_vector_bucket.s3_vector_bucket.vector_bucket_name
}

output "s3_vector_index_name" {
  value = aws_s3vectors_index.s3_vector_index.index_name
}

then apply the below commands to provision the infra
terraform init terraform apply

now the below python script will be used to seed the data into S3 & DynamoDB table

Seed Python

Purpose: Generates vector embeddings for sample text data and ingests items into both datastores.
Workflow:
1. Calls Amazon Bedrock (amazon.titan-embed-text-v2:0) to create 1024-dimensional normalized vector embeddings.
2. Ingests items into DynamoDB (VectorBenchmarkDynamo) using dynamodb.put_item.
3. Ingests vector objects into S3 Vectors (s3-vector-benchmark-index) using s3vectors.put_vectors.

import boto3
import json

bedrock = boto3.client('bedrock-runtime', region_name='us-east-1')
dynamodb = boto3.client('dynamodb', region_name='us-east-1')
s3_vectors = boto3.client('s3vectors', region_name='us-east-1')

DYNAMO_TABLE = "VectorBenchmarkDynamo"
S3_VECTOR_BUCKET = "vector-benchmark-s3-bucket"
S3_INDEX_NAME = "s3-vector-benchmark-index"
EMBEDDING_MODEL_ID = 'amazon.titan-embed-text-v2:0'

DATASET = [
    {
        "id": "doc_001",
        "category": "Engineering",
        "status": "active",
        "text": "AWS Lambda functions can run up to 15 minutes per execution with configurable memory."
    },
    {
        "id": "doc_002",
        "category": "Engineering",
        "status": "active",
        "text": "Amazon DynamoDB provides single-digit millisecond latency for key-value and document data."
    },
    {
        "id": "doc_003",
        "category": "HR",
        "status": "active",
        "text": "Employees receive 20 days of paid vacation per year with rollover limits."
    }
]

def generate_embedding(text: str) -> list[float]:
    payload = {"inputText": text, "dimensions": 1024, "normalize": True}
    res = bedrock.invoke_model(
        modelId=EMBEDDING_MODEL_ID,
        contentType='application/json',
        accept='application/json',
        body=json.dumps(payload)
    )
    return json.loads(res['body'].read())['embedding']

def seed_both_stores():
    print("Starting data ingestion into DynamoDB and S3 Vectors...\n")
    for item in DATASET:
        print(f"Generating vector for doc: {item['id']}...")
        vector = generate_embedding(item['text'])

        # 1. Ingest into DynamoDB
        dynamo_item = {
            'doc_id': {'S': item['id']},
            'category': {'S': item['category']},
            'status': {'S': item['status']},
            'text_content': {'S': item['text']},
            'embedding': {'L': [{'N': str(v)} for v in vector]}
        }
        dynamodb.put_item(TableName=DYNAMO_TABLE, Item=dynamo_item)
        print(f"   [DynamoDB] Seeded {item['id']}")

        # 2. Ingest into S3 Vectors
        s3_vectors.put_vectors(
            vectorBucketName=S3_VECTOR_BUCKET,
            indexName=S3_INDEX_NAME,
            vectors=[
                {
                    'key': item['id'],
                    'data': {
                        'float32': vector
                    },
                    'metadata': {
                        'category': item['category'],
                        'status': item['status'],
                        'text_content': item['text']
                    }
                }
            ]
        )
        print(f"   [S3 Vectors] Seeded {item['id']}")

if __name__ == '__main__':
    seed_both_stores()

Observing the key pamaeters of this python script used for data ingestion we will find the below keys

s3vectors.put_vectors (Vector Ingestion API)

What it is: The official API for adding or updating vector objects inside an Amazon S3 Vector Index.
Key Parameters:
- vectorBucketName: The target S3 Vector bucket name.
- indexName: The target S3 Vector index name.
- vectors: List of vector dictionaries:
- key: Unique identifier string for the vector item.
- data: Union container specifying data precision ({'float32': [0.123, ...]}).
- metadata: Key-value JSON document containing searchable text and filter attributes (category, status, text_content).

dynamodb.put_item (Item Write & Vector Storage API)

What it is: The standard AWS DynamoDB API call used to write document records along with their vector embeddings into a DynamoDB table.
Key Parameters:
- TableName: Name of the target DynamoDB table ("VectorBenchmarkDynamo").
- Item: Attribute value map where the vector array is formatted as a DynamoDB List of Numbers ('L': [{'N': '0.123'}, ...]).

bedrock.invoke_model (Embedding Generation API)

What it is: The AWS Bedrock Runtime API used to generate 1024-dimensional normalized vector embeddings from raw text using Amazon Titan Embeddings v2 (amazon.titan-embed-text-v2:0).

bedrock = boto3.client('bedrock-runtime', region_name='us-east-1')

payload = {"inputText": "How long can AWS Lambda run?", "dimensions": 1024, "normalize": True}
res = bedrock.invoke_model(
    modelId='amazon.titan-embed-text-v2:0',
    contentType='application/json',
    accept='application/json',
    body=json.dumps(payload)
)
embedding = json.loads(res['body'].read())['embedding']

Now we are moving forward to the comparsion script to compare the latency between the two services

Comparsion between the services

Now focusing on the comparison python script which we are delivering below:

Purpose: Runs an accurate, multi-iteration latency benchmark with warm-up logic.
Workflow:
1. Generates the query embedding once.
2. Performs 2 warm-up iterations to prime AWS SDK connection pools and internal caches.
3. Executes 10 benchmark iterations to measure response latencies.
4. Computes and displays Median (p50) and Minimum latency statistics (ms) for DynamoDB vs S3 Vectors.

import boto3
import json
import time
import math
import statistics

bedrock = boto3.client('bedrock-runtime', region_name='us-east-1')
dynamodb = boto3.client('dynamodb', region_name='us-east-1')
s3_vectors = boto3.client('s3vectors', region_name='us-east-1')

DYNAMO_TABLE = "VectorBenchmarkDynamo"
S3_VECTOR_BUCKET = "vector-benchmark-s3-bucket"
S3_INDEX_NAME = "s3-vector-benchmark-index"

def get_query_vector(prompt: str) -> list[float]:
    payload = {"inputText": prompt, "dimensions": 1024, "normalize": True}
    res = bedrock.invoke_model(
        modelId='amazon.titan-embed-text-v2:0',
        contentType='application/json',
        accept='application/json',
        body=json.dumps(payload)
    )
    return json.loads(res['body'].read())['embedding']

def cosine_similarity(v1: list[float], v2: list[float]) -> float:
    dot_product = sum(a * b for a, b in zip(v1, v2))
    norm_v1 = math.sqrt(sum(a * a for a in v1))
    norm_v2 = math.sqrt(sum(b * b for b in v2))
    if norm_v1 == 0 or norm_v2 == 0:
        return 0.0
    return dot_product / (norm_v1 * norm_v2)

def calculate_percentile(data: list[float], p: float) -> float:
    if not data:
        return 0.0
    sorted_data = sorted(data)
    k = (len(sorted_data) - 1) * (p / 100.0)
    f = math.floor(k)
    c = math.ceil(k)
    if f == c:
        return sorted_data[int(k)]
    return sorted_data[int(f)] * (c - k) + sorted_data[int(c)] * (k - f)

def query_dynamodb_vector(query_vector: list[float], category: str = None, top_k: int = 2):
    """
    Executes DynamoDB Vector Search.
    Primary Path: Native dynamodb.search_vectors API call.
    Fallback Path: Client-side vector scoring over scan items.
    """
    start_time = time.perf_counter()

    # 1. Native DynamoDB SearchVectors API Call
    try:
        kwargs = {
            'TableName': DYNAMO_TABLE,
            'QueryVector': query_vector,
            'TopK': top_k
        }
        if category:
            kwargs['PartitionKey'] = category
            kwargs['FilterExpression'] = 'status = :s'
            kwargs['ExpressionAttributeValues'] = {':s': {'S': 'active'}}

        response = dynamodb.search_vectors(**kwargs)
        elapsed_ms = (time.perf_counter() - start_time) * 1000
        results = [
            {
                "id": item['doc_id']['S'],
                "score": float(item.get('SimilarityScore', 0.0)),
                "text": item['text_content']['S']
            }
            for item in response.get('Items', [])
        ]
        return elapsed_ms, results
    except (AttributeError, Exception):
        # 2. Fallback Path for local SDK / Terraform provisioned schema
        if category:
            response = dynamodb.scan(
                TableName=DYNAMO_TABLE,
                FilterExpression='category = :c AND #st = :s',
                ExpressionAttributeNames={'#st': 'status'},
                ExpressionAttributeValues={
                    ':c': {'S': category},
                    ':s': {'S': 'active'}
                }
            )
        else:
            response = dynamodb.scan(TableName=DYNAMO_TABLE)

        scored_items = []
        for item in response.get('Items', []):
            emb = [float(val['N']) for val in item['embedding']['L']]
            score = cosine_similarity(query_vector, emb)
            scored_items.append({
                "id": item['doc_id']['S'],
                "score": score,
                "text": item['text_content']['S']
            })

        scored_items.sort(key=lambda x: x['score'], reverse=True)
        elapsed_ms = (time.perf_counter() - start_time) * 1000
        return elapsed_ms, scored_items[:top_k]

def query_s3_vectors(query_vector: list[float], category: str = None, top_k: int = 2):
    start_time = time.perf_counter()

    kwargs = {
        'vectorBucketName': S3_VECTOR_BUCKET,
        'indexName': S3_INDEX_NAME,
        'topK': top_k,
        'queryVector': {'float32': query_vector},
        'returnMetadata': True,
        'returnDistance': True
    }

    if category:
        kwargs['filter'] = {"$and": [{"category": category}, {"status": "active"}]}

    response = s3_vectors.query_vectors(**kwargs)

    elapsed_ms = (time.perf_counter() - start_time) * 1000
    results = [
        {
            "id": match['key'],
            "score": float(match.get('distance', 0.0)),
            "text": match.get('metadata', {}).get('text_content', '')
        }
        for match in response.get('vectors', [])
    ]
    return elapsed_ms, results

def compute_stats(latencies: list[float]) -> dict:
    return {
        "p50": calculate_percentile(latencies, 50),
        "p90": calculate_percentile(latencies, 90),
        "p95": calculate_percentile(latencies, 95),
        "p99": calculate_percentile(latencies, 99),
        "mean": statistics.mean(latencies),
        "stddev": statistics.stdev(latencies) if len(latencies) > 1 else 0.0,
        "min": min(latencies),
        "max": max(latencies)
    }

def print_stats_table(title: str, dynamo_stats: dict, s3_stats: dict):
    print(f"\n==================================================================")
    print(f"{title}")
    print(f"==================================================================")
    print(f"{'Metric':<20} | {'DynamoDB Vector Search':<24} | {'Amazon S3 Vectors':<22}")
    print("-" * 72)
    metrics = [("p50 (Median)", "p50"), ("p90", "p90"), ("p95", "p95"), ("p99", "p99"), 
               ("Mean", "mean"), ("StdDev", "stddev"), ("Min", "min"), ("Max", "max")]
    for label, key in metrics:
        print(f"{label:<20} | {dynamo_stats[key]:>20.2f} ms | {s3_stats[key]:>18.2f} ms")
    print("-" * 72)

def run_comprehensive_benchmark(iterations: int = 15):
    print(f"==================================================================")
    print(f"STARTING ENHANCED VECTOR SEARCH BENCHMARK SUITE")
    print(f"==================================================================\n")

    queries = [
        ("How long can serverless functions execute on AWS?", "Engineering"),
        ("What security framework manages encryption keys?", "Security"),
        ("What are the employee paid vacation limits?", "HR"),
        ("How are cloud infrastructure budgets monitored?", "Finance")
    ]

    benchmark_export = {"scenarios": {}}

    for q_text, cat in queries:
        print(f"-> Generating embedding for: '{q_text}' (Category: {cat})...")
        q_vector = get_query_vector(q_text)

        # Warm-up (3 runs ignored)
        for _ in range(3):
            _ = query_dynamodb_vector(q_vector, cat)
            _ = query_s3_vectors(q_vector, cat)

        d_latencies = []
        s3_latencies = []

        for _ in range(iterations):
            d_ms, _ = query_dynamodb_vector(q_vector, cat)
            s3_ms, _ = query_s3_vectors(q_vector, cat)
            d_latencies.append(d_ms)
            s3_latencies.append(s3_ms)

        d_stats = compute_stats(d_latencies)
        s3_stats = compute_stats(s3_latencies)

        print_stats_table(f"QUERY: '{q_text}' [{cat}] ({iterations} Iterations)", d_stats, s3_stats)

        benchmark_export["scenarios"][q_text] = {
            "category": cat,
            "dynamodb": d_stats,
            "s3_vectors": s3_stats
        }

    # Scenario: TopK Scaling Test
    print(f"\n==================================================================")
    print(f"TOP-K SCALING BENCHMARK (TopK = 1 vs 5 vs 10)")
    print(f"==================================================================")
    test_q_vector = get_query_vector("Cloud infrastructure & security best practices")
    topk_results = {}

    for k in [1, 5, 10]:
        d_lats, s3_lats = [], []
        for _ in range(10):
            d_ms, _ = query_dynamodb_vector(test_q_vector, top_k=k)
            s3_ms, _ = query_s3_vectors(test_q_vector, top_k=k)
            d_lats.append(d_ms)
            s3_lats.append(s3_ms)
        topk_results[f"TopK={k}"] = {
            "dynamodb_p50": calculate_percentile(d_lats, 50),
            "s3_vectors_p50": calculate_percentile(s3_lats, 50)
        }
        print(f"TopK={k:<2} | DynamoDB p50: {topk_results[f'TopK={k}']['dynamodb_p50']:.2f} ms | S3 Vectors p50: {topk_results[f'TopK={k}']['s3_vectors_p50']:.2f} ms")

    benchmark_export["topk_scaling"] = topk_results

    # Export results to JSON
    with open("benchmark_results.json", "w") as f:
        json.dump(benchmark_export, f, indent=2)
    print(f"\n[+] Full benchmark metrics exported to file:///Volumes/Disk1/Devto/DynamoDB%20vectors/benchmark_results.json")

if __name__ == '__main__':
    run_comprehensive_benchmark(iterations=15)

Digging deeper into the required parameters for the comparison script

dynamodb.search_vectors` (Native DynamoDB Vector Search API)

What it is: The native Amazon DynamoDB API call (dynamodb:SearchVectors) designed for direct Approximate Nearest Neighbor (ANN) vector search against native VectorIndexes configured on a DynamoDB table.
Key Parameters:
- TableName: Target DynamoDB table name ("VectorBenchmarkDynamo").
- IndexName: Vector index name ("DynamoVectorIndex").
- PartitionKey: Partition key filter string (e.g. 'Engineering').
- QueryVector: Query embedding vector array [0.123, ...].
- TopK: Number of nearest matches to return (TopK=2).
- FilterExpression: Optional attribute filter ('status = :s').
Response Structure: Returns candidate Items along with calculated SimilarityScore attributes.

s3vectors.query_vectors` (Vector Similarity Search API)

What it is: The API used to execute Approximate Nearest Neighbor (ANN) vector similarity search against an S3 Vector Index.
Key Parameters:
- queryVector: Target embedding vector wrapped in {'float32': [...]}.
- topK: Number of nearest matches to return.
- filter: Metadata filtering document using logical expressions (e.g. {"$and": [{"category": "Engineering"}, {"status": "active"}]}).
- returnMetadata=True: Includes payload metadata fields in the response.
- returnDistance=True: Includes similarity distance scores in the response.

now run the comparsion the script.

The final results

in my case i ran two queries to see how much we are going to save in latency and the output results as below:

Query A: "How long can serverless functions execute on AWS?" (Category: Engineering)

==================================================================
QUERY: 'How long can serverless functions execute on AWS?' [Engineering]
==================================================================
Metric               | DynamoDB Vector Search   | Amazon S3 Vectors     
------------------------------------------------------------------------
p50 (Median)         |               303.32 ms |             238.41 ms
p90                  |               411.22 ms |             582.15 ms
p95                  |               460.86 ms |             941.42 ms
p99                  |               553.35 ms |            1227.79 ms
Mean                 |               303.09 ms |             347.28 ms
StdDev               |               105.18 ms |             299.01 ms
Min                  |               186.05 ms |             229.88 ms
Max                  |               576.48 ms |            1299.38 ms
------------------------------------------------------------------------

Query B: "How are cloud infrastructure budgets monitored?" (Category: Finance)

==================================================================
QUERY: 'How are cloud infrastructure budgets monitored?' [Finance]
==================================================================
Metric               | DynamoDB Vector Search   | Amazon S3 Vectors     
------------------------------------------------------------------------
p50 (Median)         |               165.83 ms |             244.50 ms
p90                  |               339.38 ms |            1107.96 ms
p95                  |               386.78 ms |            1690.60 ms
p99                  |               456.33 ms |            2070.50 ms
Mean                 |               238.51 ms |             501.47 ms
StdDev               |                96.61 ms |             559.72 ms
Min                  |               159.50 ms |             230.00 ms
Max                  |               473.71 ms |            2165.48 ms
------------------------------------------------------------------------

observing these results you will find a lower latency for DynamoDB, but this does not mean that S3 vector can't be used anyway

Each one of them has it's own use case that we will dicuss now

As system architects, choosing between DynamoDB Native Vector Search and Amazon S3 Vectors comes down to how your application state interacts with your vectors:

🛒 Use Case 1: DynamoDB Native Vector Search (Operational State + Vector Unity)

💡 "I need my operational data and vector embeddings stored in a single unified record with sub-10ms read performance."

The Developer Story: Imagine building an e-commerce platform where users search for "lightweight waterproof running shoes". When a customer views or purchases a shoe, inventory (stock_count) must update instantly.
Why DynamoDB Wins Here:
- Zero Data Drift: The product metadata (price, stock_count) and vector embedding array live in the same DynamoDB item. You don't need background worker pipelines syncing DynamoDB updates to a separate vector store.
- Atomic Updates: Decrementing inventory (ADD stock -1) happens atomically inside the exact same row.
- Predictable Sub-10ms Latency: When end users are waiting on a live checkout or search page, DynamoDB provides tight $p99$ tail-latency stability ($< 200\text{ ms}$).

📚 Use Case 2: Amazon S3 Vectors (Deep-Scale RAG Knowledge Base Archive)

💡 "I have millions of document chunks in S3 and want up to 90% cheaper vector storage with serverless ANN search."

The Developer Story: Imagine building a company-wide RAG AI chatbot that searches millions of PDF manuals, policy documents, engineering wikis, and customer transcripts across HR, Legal, and DevOps.
Why Amazon S3 Vectors Wins Here:
- Up to 90% Cheaper Storage: Storing millions of 1024-dimensional vectors in hot database table capacity gets expensive fast. S3 Vectors stores embeddings at S3 object storage price tiers, saving up to 90% on storage costs.
- Faster Write Ingestion: Ingesting large batches of vector embeddings is 28.6% faster (412 ms p50) than writing individual database items.
- Flat TopK Latency: Retrieval latency remains rock-solid at ~240 ms whether requesting $TopK=1$ or $TopK=10$ context chunks for LLM prompts.

⚔️ Technical Comparison Matrix: Amazon DynamoDB vs. Amazon S3 Vectors

Feature / Metric	Amazon DynamoDB Native Vector Search	Amazon S3 Vectors
Primary Goal	Ultra-fast operational search + RAG	Deep-scale, ultra-low-cost vector archive
Query Latency Target	Single-digit ms	Sub-second
Storage Engine	Native DynamoDB Table Items	S3 Object Storage Tiers
Cost Profile	Pay-per-write/read item capacity	cheaper storage per million vectors
Data Coupling	Unified (operational record + vector in 1 item)	Decoupled (vectors point to object/file paths)
Distance Metrics	Cosine, Euclidean, Dot Product	Cosine, Euclidean
Maximum Vector Dimensions	Up to 4,096 dimensions	Up to 4,096 dimensions
Ingestion Write Latency (p50)	`577.24 ms`	`412.30 ms` (⚡ 28.6% faster)
TopK Scaling (TopK 1 → 10)	Variable scan/filter latency	Flat latency (`~240 ms`)
SDK API Methods	`dynamodb.put_item`, `dynamodb.search_vectors`	`s3vectors.put_vectors`, `s3vectors.query_vectors`
Primary Use Cases	Real-time product search, fraud detection, user profiles	Enterprise RAG chatbots, PDF manuals, media archives

Using Amazon ElastiCache & Prompt Caching to reduce & optimize Amazon Bedrock invocation & Tokens: Part1

Muhammed Ashraf — Sun, 02 Aug 2026 13:43:48 +0000

What is Amazon Bedrock?

as per the official documentation of AWS the definition of Amazon bedrock as below:

"Amazon Bedrock is a fully managed service that provides secure, enterprise-grade access to high-performing foundation models from leading AI companies, enabling you to build and scale generative AI applications."

so you can consider it as a managed API hub that allow managed access to all available models in the market with an easy way and low configuration.

before proceeeding with the article we will list some definitions that you always see in most of the market and their use cases

RAG ( Knowledge bases ): Managed Retrieval-Augmented Generation pipeline and used for enterprise document Q&A, internal wikis
Agents & AgentCore: Autonomous multi-step orchestration and tool calling and used for automated workflows, customer refunds, IT ops
Guardrails: Enterprise safety layer, PII masking, hallucination prevention and used for compliance enforcement, prompt injection defense.
Bedrock Flows: Visual and code-defined workflow builder and used for Content pipelines, multi-prompt approvals.
Model Customization: Fine-tuning, pre-training, and model distillation and used for domain-specific models (Legal, Medical, Finance).
Prompt caching: Prompt caching is an optional feature that you can use with supported models on Amazon Bedrock to reduce inference response latency and input token costs. By adding portions of your context to a cache, the model can use the cache to skip recomputation of inputs, allowing Bedrock to share in the compute savings and lower your response latencies.

Architecture Diagram

So basically this diagram explains how the cache will work as we have the below

Tier 1 (Application Level - Amazon ElastiCache): Bypasses Amazon Bedrock completely to eliminate 100% of invocations and tokens for identical or semantically similar queries.
Tier 2 (Model Level - Amazon Bedrock Prompt Caching): Reduces input token costs by up to 90% and latency by up to 85% when calls must hit Bedrock by reusing internal KV attention states for long prompt prefixes

Sequence Flow Diagram

so for step by step detailed steps of the sequence diagram:

Phase 1: Incoming Request & Tier 1 Lookup

Step 1 (Client / App --> Application Service / Lambda):
The client submits a user prompt to the backend service.

Step 2 (Application Service / Lambda --> Amazon ElastiCache):
Before touching Amazon Bedrock, the application backend inspects ElastiCache:

Computes a SHA-256 hash of the prompt for an exact match.
Runs a K-Nearest Neighbors vector search to detect semantic similarity.

Path A: Tier 1 Cache Hit (ElastiCache)

If the query or a semantically equivalent query exists in ElastiCache:

Step 3 (Amazon ElastiCache --> Application Service / Lambda):
ElastiCache returns the cached JSON response.
Step 4 3a (Application Service / Lambda --> Client / App):
The application returns the answer directly to the user.

Latency: ~5–20 ms

Cost: $0 in Bedrock LLM tokens.

Path B: Tier 1 Cache Miss (Proceed to Amazon Bedrock)

If ElastiCache does not have the answer, the request moves down to Tier 2:

Step 5 3b (Application Service / Lambda --> Amazon Bedrock Runtime): The backend invokes the Bedrock converse() API, attaching a cachePoint checkpoint inside the payload to separate static system context (documents/instructions) from the user's dynamic query.

Sub-Branch B1: Tier 2 Prompt Cache Hit (Bedrock)

If Bedrock has already cached the static prefix (KV attention state) within its 5-minute rolling TTL:

Step 6 4a (Amazon Bedrock Runtime --> Foundation Model):
Bedrock instructs the Foundation Model (Amazon Nova) to reuse the precomputed Key-Value (KV) attention states, skipping prompt processing for the prefix.

Step 7 (Foundation Model --> Amazon Bedrock Runtime):
The model generates new tokens based only on the dynamic query suffix.

Step 8 (Amazon Bedrock Runtime --> Application Service / Lambda):
Bedrock sends the completion back to the application.

Cost Savings: 90% discount on input tokens behind the cachePoint.
Speed: Significantly faster Time-To-First-Token (TTFT).

Sub-Branch B2: Tier 2 Full Cache Miss (Bedrock)
If this is the first time the static prompt is being processed or the 5-minute cache TTL expired:

Step 9 4b (Amazon Bedrock Runtime --> Foundation Model):
Bedrock computes the entire prompt (static prefix + dynamic query) and writes the static prefix into Bedrock's ephemeral cache for future requests.

Step 10 (Foundation Model --> Amazon Bedrock Runtime):
The model generates completion tokens.

Step 11 (Amazon Bedrock Runtime --> Application Service / Lambda):
Bedrock returns the full completion response to the application backend at standard pricing.

Phase 2: Post-Processing & Cache Write-Back

Step 12 5 (Application Service / Lambda --> Amazon ElastiCache):
Now that a fresh response was generated by Bedrock, the Lambda backend computes its embedding vector and stores the prompt-response pair in ElastiCache. Future identical or similar queries will now hit Tier 1 (Path A).

Step 13 6 (Application Service / Lambda-->Client / App):
The backend delivers the final generated response to the client.

Wrapping Up Part 1 & What’s Next in Part 2

By implementing this Two-Tier Caching Strategy, we move away from treating LLM infrastructure as a black box and start optimizing it with classical software engineering patterns.

Tier 1 (ElastiCache): Cuts out 100% of token costs and invocations for identical or semantically duplicate queries at sub-20ms speeds.
Tier 2 (Bedrock Prompt Caching): Slashes input token costs by up to 90% and latency by up to 85% when long static context must hit the model.

Moving to part 2 where we are going for the hands-on and implementation.

How I Built AURA: A Serverless, AI-Powered Fitness Engine on AWS Bedrock: Part 1

Muhammed Ashraf — Wed, 29 Jul 2026 17:30:36 +0000

Introduction

Most of the current fitness apps are using static template to generate a workout and diet plans for you.

they are just getting some basic inputs (age, weight, hight and maybe body fats) while ignoring other parameters that could enhance the generated plans such as (HRV, Sleep, Strain & Your body readiness for training)

For example:
It tells you to eat 2,500 calories every single day, and completely ignore the fact that you slept 4 hours last night, your HRV tanked by 35%, and your central nervous system is fried.

The idea came into my mind to build a fitness app that could generate exercise & diet plans then these plans should be tailored for you based on your data.

The idea became bigger so I tought it will be better that AURA athlete coach ( my app ) to become hub for everything

You could do the following:

Generate exercise & diet plans based on you need ( Weight Loss, Muscle gain & Maintenance ) including number of days and meals per day.
Select your pathway (Hypertrophy, Conditioning, Powerbuilding and Longevity)
Lifter engine that you could track your exercises and log them for better visibility about each muscle group and how it was trained during the week
Macro Feature that could help you to log your daily food or sync through MyFitnessPal for better tracking of your targets.
Progress that you could use to log your actual photo contains your progress to have a better visibilty
Pulse which contains all the data about you (Recovery, Strain, Sleep, Calories, Stress, Spo2, Calories, Training Load, Muscle Volume Group, Fitness Age & pace of aging, Acitvity & burn)
Access to AI Chat to follow-ups and answering user's questions

High-Level Architecture & Sequence Flow Overview

The core guiding principle behind AURA's backend is Zero Client-Side Credentials. The mobile app holds no database keys, AWS credentials, or LLM API tokens. Everything flows through an authenticated, rate-limited serverless proxy.

High Level Diagram:

The Tech Stack at a Glance:
Frontend Mobile: React, TypeScript, Capacitor, Tailwind CSS (Custom Dark Glassmorphic Design System).

Compute: AWS Lambda running Node.js in a Private VPC.

Database: Amazon RDS PostgreSQL managed via Amazon RDS Proxy for connection pooling.

AI Core: AWS Bedrock running Amazon Nova Pro (for complex workout/diet plan generation) and Amazon Nova Lite (for real-time coaching chat) and for arabic we use Qwen3.5 32B with Bedrock Gaurdrails & Prompt Caching features enabled.

Media & Assets: Amazon S3 for exercise video & user saved progress photos with KMS-encrypted presigned URLs for progress photos.

Deep Dive 1: Serverless AI Orchestration with AWS Bedrock

Instead of relying on third-party LLM APIs with unpredictable latency and client-side key exposure, I routed all AI operations through AWS Bedrock.

I used 3 Models based on user needs,

Nova Lite for real time messages with the AI Coach
Nova Pro for generating exercises & diet plans
Qwen 3.5 32B for native Egyptian Arabic Language.

Enforcing Strict JSON Output Contracts
LLMs are notoriously bad at returning consistent JSON when given loose instructions. To prevent app crashes, the Lambda function supplies a strict JSON schema contract and runs a deterministic post-validation pass.
Dialect & Medical Guardrails
Because AURA supports regional coaching dialects (such as urban Egyptian Arabic) and operates in the health domain, system prompts require explicit guardrails:

Dialect Negative Constraints: To prevent the model from mixing dialects ( the system prompt explicitly bans out-of-region keywords.
Medical Scope Boundaries: The AI is strictly programmed as an athletic performance coach, not a medical doctor. If a user mentions red-flag symptoms like acute chest pain or severe joint injuries, the model immediately halts fitness advice and directs them to seek medical attention.

Deep Dive 2: Biometric Telemetry & Autoregulation
An AI coach is only as smart as the data you feed it. AURA integrates with Android Health Connect and Apple HealthKit to read read-only telemetry:

Heart Rate Variability (HRV)
Resting Heart Rate (RHR)
Sleep Duration & Sleep Architecture (Deep, REM, Light)
Active Strain & Step Counts

UI/UX & Gamification: Making Science Feel Cinematic

A technical backend means nothing if the user experience feels like a boring medical dashboard. We built AURA's frontend around dark glassmorphism, high-contrast neon accents, and interactive gamification in addition to this we are allowing user to select the theme that he likes

Glowing Progress Rings: Vector SVG rings using custom HSL glow filters that adapt dynamically to health scores (Green for Primed, Yellow for Caution, Red for Low Recovery).

Interactive 3D Reward Vaults: Replacing flat checkboxes with claimable Daily Reward Vaults, complete with success haptics and confetti overlays.

Tiered Quick Challenges: Level 1 to Level 4 athletic challenges granting scaled XP (+100 to +500 XP) with repeatable prestige badge multipliers (x1, x2, x3) to encourage user to go further and achieve his daily goals.

Theme Customization: Live theme presets (Hyper Emerald, Solar Amber, Lava Crimson, Amethyst) that persist across all top headers, navigation bars, and glass cards.

Will continue the rest of setup and solution on part 2 meanwhile you could visit the website here

and I will drop some of screenshots from the app

Exercise with description and gif:

Heartrate graph

Recovery & Strain Readiness:

Using AWS App Runner to build & host my website

Muhammed Ashraf — Thu, 08 Jan 2026 13:28:46 +0000

Overview

AWS provides a lot of different ways to deploy your application. based on whatever you are looking for, you will find a service for this. For example, if you are looking to deploy a classic application you have EC2 or a fully managed service you have Elastic Beanstalk. If you are looking to host containers, you have Amazon ECS, EKS or you can use for a fast and simple service like AWS App Runner

As per AWS documentation. AWS App Runner is very simple, fast service that helps you to deploy your application from either source code like GitHub or Image repo like ECR into a scalable and cost-effective service

Referring to cost. the service is very cost-effective since you will only pay for the actual traffic since App Runner provision resources based on your traffic (Lower Number of requests = Lower provisioned resources)

I was trying to explore AWS App Runner, So I deployed a website on AWS App Runner with other services such as S3, DynamoDB, ECR and Amazon SES for sending emails.

Architecture

Technology Stack

Runtime: Node.js
Framework: Express.js
Templating: EJS (Embedded JavaScript)
Frontend: Tailwind CSS (Styling), Alpine.js (Interactivity/State)
Database: AWS DynamoDB (NoSQL)
Storage: AWS S3 (Food Images)
Email: AWS SES (Order Notifications)
Hosting: AWS App Runner (Dockerized)

Directory Structure

Core files:

server.js: Entry Point. Configures Express, middleware and static files. Starts the server.
lib/aws-client.js: AWS Utility. Centralizes initialization of DynamoDB Client, S3 Client, and SES Client.

Route Handlers:

shop.js: Customer Facing. Handles public access to the menu, cart operations, and checkout.
admin.js: Admin Management. Protected routes for managing items, authentication, and stats.

Views:

index.ejs: Homepage. Renders the Menu (Hot/Frozen sections), Cart Drawer, and Hero section.
admin.ejs: Dashboard. Main Admin interface for Adding/Editing/Deleting items.
admin-stats.ejs: Analytics. Visual dashboard using Chart.js to show revenue and order trends.
login-ejs: Authentication. Admin login form.

Partials:

header.ejs: Navigation bar, Mobile Menu, Favicon, Libraries import (Tailwind, Alpine).
footer.ejs: Page footer, Closing tags.
product-card.ejs: Reusable component for rendering a single regular menu item.
cart-drawer.ejs: The sliding cart sidebar content

AWS Components

Compute: AWS App Runner

Role: Fully managed container application service.
Configuration: Autoscaling instances based on request load.
Source: Deploys the Docker image directly from Amazon ECR.

Database: Amazon DynamoDB

Role: Serverless NoSQL key-value database.
Tables:
MenuTable: Stores food items (itemId, name, price, description, category).
OrdersTable: Stores customer orders (orderId, customerDetails, items, total, status).

Storage: Amazon S3

Role: Object storage for uploaded food images.
Access: Images are uploaded via the Admin Panel. The application generates signed URLs or proxies them for secure display.

Container Registry: Amazon ECR

Role: Securely stores the Docker container images.
Workflow: docker push commands upload new versions of the app. App Runner detects these changes to update the live site.

Communications: Amazon SES
Role: Reliable email delivery service.

Snapshots from the website

So, AWS App Runner simplifies the deployments. You don't care about the deployment's steps You only focus on improving your code or monitor your website.

Using AWS CloudFront to enhance the performance, Security & Availability of your application

Muhammed Ashraf — Mon, 13 Oct 2025 11:21:21 +0000

Hosting a website that serves a lot of customers around the world then AWS CloudFront should be considered by you since it distributes your content of your website and store them at the nearest edge location to your clients.

This significantly improves performance and reduces loading times which enhances the customer's experience

In this article I will try to explain CloudFront features that can be used to enhance the overall experience of your website

We are going to discuss the below features

Origin Group and Multiple Origins
CloudFront Functions
Global Accelerator
CloudFront Security

But before we start, we will try to explain how does CloudFront work

Overview

They key architectural components of CloudFront are distribution, Edge locations or Point of Presence, Regional Edge Cache, Origin & Caching Behavior

Let's walkthrough them one by one

Distribution: this the primary resource you create, and it contains the configurations including origins, caching behavior and security settings.
Edge Locations: there can be considered as data centers where content is cached
Regional Edge Cache: this is larger caching layer located between edge locations & origin, they store less popular content for larger periods than smaller edge locations
Origin: This is the source of your content, It can be S3, ALB, NLB EC2 or On-permise server, you can find more information Origin Types
Cache Behavior: set of configuration rules you apply to specific URL patterns, for example routing, TTL or redirection

Referring to AWS official documents this is how architecture looks like

CloudFront has a lot of features that could help you to go beyond if you used them correctly, Let's discuss them one by one

Feature

Origin Group & Multiple Origin

As mentioned earlier, Origin is the source of your data, the OriginGroup feature allow you to add multiple origins to the same group (primary origin & secondary origin) in addition to a failover criteria you define, This means if you send a request to your primary region and origin responded with an error status code, request will be redirected to the secondary origin.

you can use muliple origin groups to server the contents based on their types,

for example: You can redirect the static content to Origin A while dynamic content to Origin B

Global Accelerator

It achieves low latency and high performance by utilizing AWS global network and avoid going to the public internet,

it works by providing a static IP address to your application and route the traffic through the optimal route & healthy endpoint

Use Cases

Routing to Multiple Origin

CloudFront Cache Behavior can be used to route traffic based on the path pattern as below:

This allows you to serve static & dynamic origin instead of having different architecture for both content type

Origin Failover Through Origin Groups

This feature helps you to achieve high availability by forwarding failed requests to another Origin.

Restrict Access Through Custom Headers

By adding Custom Header on CloudFront and modifying the requests, this will allow us to define a rule on the ALB that if the requests don't contain the Custom Header will be denied, any direct access will be blocked by the defined rules on the ALB

Closing Words

CloudFront is a powerful service if you are looking to distribute your application or website globally, it has many features that will help you to achieve high availability, security and reduce latency for the clients that reaching your application.

References

CloudFront Official Documents
ow UNiDAYS achieved AWS Region expansion in 3 weeks

Using Amazon Textract to analyze and extract text from Documents Part 1

Muhammed Ashraf — Sun, 14 Sep 2025 17:37:26 +0000

Amazon Textract is very powerful machine learning
service that used to analyze do documents and extract either text or handwriting from scanned documents.

It can be used to build different solutions for different use cases such as:

Financial Services
Health Care

In this article we walkthrough how to build a solution on that works on extracting texts from PDF, analyze them, then store them into DynamoDB for further analysis

We will use a mix of AWS services to build our solutions, below is a breakdown of these services and the use case of them

S3 Buckets: will be used to be our storage for raw & extracted JSON files
Lambda: will be used to invoke Amazon Textract through StartDocumentAnalysis & GetDocumentAnalysis APIs, store the JSON files into S3
SNS: used for async communication to invoke the Lambda to get the results and store them into the final bucket
Eventbridge: Used to trigger Lambda function once file is uploaded to S3
AWS Glue: it will used as batch job to iterate over the S3 bucket to convert the files and ingest the files into DynamoDB
DynamoDB: will be our storage for the extracted data

Sequence Flow

Functional Requirements

User should be able to upload PDF files to S3 bucket

Non-Functional Requirements

Solution must be high available
Solution should be reliable

Block Diagram

Our block diagram shows the components that will be used to build our solution

High Level Design

The high-level design shows the services used to build our solution, focusing on ingesting, analyzing & storing the results DynamoDB

We will breakdown our solution into different aspects

High availability:
- Storage: our storage services such as S3 & DynamoDB, offer high availability you can find all the details related for each service here S3 DynamoDB
- Lambda Function: Resilience in Lambda
- SNS: Resilience in SNS
- Amazon Textract: Resilience in Amazon Textract

Amazon Textract APIs

We will utilize StartDocumentAnalysis API and GetDocumentAnalysis API

for StartDocumentAnalysis API we have different types of features such as (Tables, Forms, Queries, Signature and layout) we will use Queries to extract specific data from the statements such as card number, client name, new charges etc.

and for GetDocumentAnaylsis API is recieving the results of StartDocumentAnalysis API in aschyronous mode, we will utlize SNS to decouple our Lambda functions.

below you can find a screenshots for the setup

we have two Lambda functions as below

the trigger_lambda_put will be used to be triggered once a file uploaded to the S3 and call the Amazon Textract API
and other function will be used to get the results and filtering out the required parameters

we have also two S3 buckets for the input and outputs files

SNS

the output will be as below

file name is textract job id .json, this can be modified through the lambda function code

the final output should be as below

since I have defined the card holder name as a filter parameter in the Lambda function code

In part two we will discuss more about AWS Glue JOB to process multiple files and store them into DynamoDB and we will cover the cost part for each component

Using AWS Comprehend to analyze customers' feedback

Muhammed Ashraf — Fri, 15 Aug 2025 16:11:17 +0000

Analyzing customer feedback for your product is recommended to enhance and fill the gaps for any business want to enhance their products and customer services

AWS provides several services that can used to achieve this, One of the services that can be used is Amazon Comprehend

Based on AWS documentation Amazon Comprehend "Amazon Comprehend uses natural language processing (NLP) to extract insights about the content of documents. It develops insights by recognizing the entities, key phrases, language, sentiments, and other common elements in a document. Use Amazon Comprehend to create new products based on understanding the structure of documents."

So, in this blog we will discuss how we are going to use Amazon Comprehend to analyze customer reviews

We will start with the functional & non-functional requirements then we will move to the core components and the setup of the solution

We made some assumptions that the application is already deployed on an EC2 instance and users will be able to write their review, the application is Inegrated with S3 and will generate a text file contains review

Functional Requirements

Analyze text documents which will be generated by application
Extract the reviews and categorize them either POSITIVE, NEGATIVE or MIXED based on sentiment score

Non-Functional requirements

Solution should be reliable

HLD

Core Components

EC2: It will host the application accessed by users.
S3: will host the review files generated by application which contains user's reviews
Lambda Function: It will be triggered by S3 event notification and will invoke Amazon Comprehend by using Detect_Sentiment API, it will also save the generated results to DynamoDB
DynamoDB: will store the final results either the review is POSITIVE, NEGATIVE or MIXED

Solution Setup

1- We have created an S3 bucket with a directory uploads which will host the generated review files

The bucket is created with the default configuration and for event notification you can create type of events is (POST, PUT)

2- We have created a table on DynamoDB with the below configuration:
Table Name: CustomerFeedbackAnalysis
Partition Key: FeedbackID

We kept rest of configuration as default but for sure you will configure them based on your requirements

3- We created a Lambda function with defaults configuration except for the Timeout since it's only 3 seconds and IAM Role since it should have permissions for the below:

AWSLambdaBasicExecutionRole (Basic Exection Role for Lambda)
AmazonS3ReadOnlyAccess (to read files from S3)
AmazonDynamoDBFullAccess (Write results to DynamoDB)
ComprehendReadOnly (Invoke Comprehend using Detect_Sentiment API)

Sequence Diagram

The flow should be as below

The results

I generated some random reviews and uploaded them to the S3, the final results stored in DynamoDB table as below:

References

Analyze content with Amazon Comprehend and Amazon SageMaker notebooks
Amazon Comprehend

Content Moderation Using AWS Rekognition

Muhammed Ashraf — Thu, 17 Jul 2025 10:57:02 +0000

Overview

Content moderation is considered very important for organizations, especially for social media, advertising, and education. Any organization requires special analysis for media.

The basic definition of content moderation is reviewing media and content to ensure it complies with the standards and guidelines set by the organization.

Achieving that manually will be very difficult, since you are going to review every content that is uploaded by a user, and it's even impossible if we are talking about a large user base platform.

In this article, we will discuss deploying a content moderation solution on AWS and how to utilize different AWS services to achieve this.

First, we will break down our core components.

Core Components

User
Content

Then we will have our high-level design. Keep in mind that you can have different architectures for this, and you are not only restricted to this.

Architecture

Core Services

S3 Bucket --> It will be our storage services which will be used to store our content.

S3 has a feature called S3 event notification, which allows S3 to trigger events in case of uploading, deleting, or replication events. You can find a list of types of events here and the destinations that can receive this notification.

Lambda Function --> it will be used to put messages contains object details to SQS whenever any object uploaded to S3 and another one will pull SQS messages and triggers Rekognition API to start the content moderation process.
SQS --> It will be utilized to decouple upload and moderation process from each other to avoid bursts and spikes
Rekognition --> The actual moderation process happens here since AWS Rekognition is an AI service that is used for an image/video analysis, the results will be stored in another bucket

Sequence Diagram

The below sequence shows how the flow should be

The user will upload content to an S3 bucket. In the real world, the S3 bucket is the storage for the front-end layer, which can be a mobile or web app. I just used S3 directly for simplicity and explanation.
An S3 event notification will be triggered once an object is uploaded, and the destination will be a Lambda function.
The Lambda function will send a message containing details about the uploaded object to the SQS for decoupling the architecture.
A moderator Lambda function will pull the message from SQS and trigger the Rekognition API (DetectModerationLabels). More info here related to label categories.
The results will be stored into another S3 bucket for better isolation.

Conclusion

For sure, content moderation is used across nearly all social media and advertising organizations, and it became easier with the help of AWS's different AI services. You don't need to have experts in AI or machine learning to deploy your own content moderator, just go through the AWS documentation.

Hope this article helps you, and please let me know if you have any comments.

AWS Migration Services Part1

Muhammed Ashraf — Tue, 17 Jun 2025 12:00:42 +0000

Overview

In this article we are going to discuss the different service related to discovery & migration data either from AWS to AWS or On-premises to AWS and when features, use case for each one of them.

There is a ton of services that can be used to migrate either for application or database such as the below services:

AWS Application Migration Services (AWS MGN)
AWS Database Migration Service (AWS DMS)
AWS Datasync
AWS Migration Hub
AWS Transfer Family
AWS Application Discovery
AWS Modernize Mainframe Application

Let's start with the application & databases then move to the data migration services:

Application & Database Migration Services

AWS MGN: it's a service that used to help companies to lift and shift their physical, virtual or cloud-servers with facing any compatibility or performance issues, it simplifies the moving process for large servers to AWS,

It eliminates compatibility issues by the below features:

Block-level replication: By replicating at the block level, MGN is largely agnostic to the applications, databases, and file systems running on the server. It is copying the underlying disk data, not interpreting the files.
Automated OS conversion: When a test or cutover instance is launched, MGN automatically handles the necessary conversions to make the server boot and run natively on AWS infrastructure.

While performance issues are handled by:

Minimal Source Impact: The replication agent is lightweight and operates in the background with throttled resource consumption.
Choosing the right instance size: In the MGN console, you can define Launch Settings for each source server. This allows you to choose the appropriate EC2 instance type
Optimized data transfer: While the data transfer happens over your network, the continuous replication model avoids the need for a massive, single data transfer that could saturate your connection during business hours.

You can choose AWS MGN when you are looking for rehosting migration and need to quickly move existing application.

you can find more technical details about how it works on the below links https://docs.aws.amazon.com/mgn/latest/ug/General-Questions-FAQ.html

AWS DMS: it's a common service that support a lot of sources DB and many destinations used to migrate databases either an AWS DB or external DB

there are a lot of features for the DMS such as the below features

Support of homogeneous & heterogeneous migrations: DMS supports migrating between the same database engines and between different database engines.
CDC: DMS can capture changes from the source database and apply them to the target in near real-time, minimizing downtime during the migration.
Schema Conversion: DMS works in conjunction with the AWS Schema Conversion Tool to convert the source database schema and code to a format compatible with the target database.
Serverless Option: A serverless feature automatically provisions and scales the migration resources, simplifying the process further.

the major use case for this service is migration of databases either from/to External/AWS

for more technical information about how it works and supported sources, destinations you can visit this link AWS DMS FAQs

AWS DataSync: It's a service that used to transfer & accelerate movements of large data between on-premises storage systems and AWS storage services.

So as defined above the main focus only of this service is moving data between storage systems and it has many features such as:

Accelerated data transfer: by using purpose-built protocol to transfer data up to 10 times faster than open-source tools.
Automated and Managed: It handles many of the tasks involved in data transfer, including scripting copy jobs, scheduling and monitoring transfers, validating data integrity, and optimizing network utilization.
Secure Transfers: Data is encrypted in transit and at rest, and the service integrates with AWS security features like IAM roles and VPC endpoints.
Broad Storage Support: It supports a wide range of storage systems, including Network File System (NFS), Server Message Block (SMB), and Amazon S3, Amazon EFS, and Amazon FSx.

You can use this service for migrating large datasets to AWS or archiving cold data from on-permises to cost effective AWS storage like AWS S3 Glacier, it can be used also in data replication to achieve business continuity.

AWS Migration Hub: it's a like center location that you can manage & track the progress of applications migrations across multiple AWS accounts and partner solutions

It has many features like:

Centralized Tracking: Monitor the status of migrations from various tools like AWS Application Migration Service, AWS Database Migration Service, and partner migration tools.
Application Discovery: Integrates with AWS Application Discovery Service to automatically gather information about your on-premises servers, including specifications, performance data, and network dependencies.
Strategy Recommendations: Provides recommendations on the best migration and modernization strategy for your applications.

So picking the right service based on your needs is mandatory, we can summarize the use case for each service as below:

looking for lift and shift solution then you can go for AWS MGN
Database migration and schema conversion then use AWS DMS
Replicating & moving data from on premises to AWS then the DataSync option is the one
Centrally manage and track the progress of migration you can use migration hub

Based on you migration strategy you can pick the right choice for you.

we will discuss more the remaining services in another article but for now I will drop references below for each service if you want to dig deep in each service

AWS DataSync
AWS MGN
AWS DMS
AWS Migration Hub

[Boost]

Muhammed Ashraf — Sun, 23 Feb 2025 10:27:45 +0000

How I built a simple Twitter-Like System on AWS with the help of Grok AI

Muhammed Ashraf ・ Feb 22

#ai #aws #cloud

How I built a simple Twitter-Like System on AWS with the help of Grok AI

Muhammed Ashraf — Sat, 22 Feb 2025 23:34:49 +0000

As the article title states, Grok AI wrote most of the code, as my expertise lies in solution architecture, So I'm writing this article to share my experience in how I used Grok AI to help me to apply my experience to build this system and enhance my hands-on experience.

I am not an expert coder, but I understand how large systems, such as social media websites, function.

Building an enterprise system requires experience in system integration and service selection within different architectures. AI can assist, but its effective use requires a strong understanding of how these systems work.

But don't worry, this article is written by me, not AI 😎😁

First, you need to list the functional requirements of your system,

Functional requirements are the core things your system should do. If we take a moment to think together about what functions a system like Twitter should have,
the first thing that comes to mind is that the user should be able to sign up for an account and log in using that account.

Also, users should be able to post and delete tweets, upload photo, love tweets, comments and retweets

I tried to cover some core features to just help you understand how we can make this happens and later on we may build on these new features

I will list the function requirements which covered by this system

User can sign up and login
User should be able to post tweets
User Should be able to delete his tweets
User should be able to love and comments on tweets

Non-functional requirments are define how system should behave

They are like

Elasticity
High availability
Scalability

These requirements should enhance the user experience

The second thing you should do is your capacity estimation. This will help you pick the right resources for your system to avoid any spikes or under/high utilization.

We will not cover this here since it's a very simple system. You can search the internet; there are a lot of resources covering this. I will drop some links below 😁 ✌

High level Design & Components

I picked AWS since this is my area i
Of expertise and used common services for building the system

Below is a breakdown of the services I used:

EC2 instance: To host our frontend code and act as a web server.
API Gateway: Built APIs used for signup, login and authorization, posting tweets, deleting tweets, liking tweets, and commenting. Each function has its own URL and Lambda function.
Lambda Functions: Contain the logic for the system functionality mentioned above.
DynamoDB: Contains Users and Tweets tables that store the data.

Sequence Diagrams

Signup Flow

Post Tweet Flow

Love Tweet

Love Tweet, Comment & Delete Tweet they are all the same in terms of getting tweet_id and perform the action

Some screenshots of the UI:

Code

Twitter-Like-App

And now for the interesting part, I uploaded a code on Github
feel free to use it and remember this is a very basic code, Further enhancements are coming 😁

Final Words

I know best practices are not applied here and many features are missing such as decoupling the components, caching service and following/followee system and the system looks dummy that cannot handle heavy workloads 😢🤦‍♀️, but it should give you a vision of how larger systems should work, and you can consider it a start.

And you can make magic happen If you know the way and how you can interact with AI.

I hope this article helped you to understand a little about how you can make use of AI tools and how you can build a system by help of these tools, I will try to work on this base version for further enhancement and features and I may create another article to include these enhancements.

Will be happy to see your comments and suggestions 😃

References

Twitter System design
Hello Interview Youtube Channel

Parsing & Loading Data from S3 to DynamoDB with Lambda Function

Muhammed Ashraf — Sun, 05 Jan 2025 14:40:57 +0000

Many scenarios require you to work with data formatted as JSON, and you want to extract and process the data then save it into table for future use

In this article we are going to discuss loading JSON formatted data from S3 bucket into DynamoDB table using Lambda function

Prerequisites

IAM user with permissions to upload objects to S3
Lambda Execution role with permissions to S3 & DynamoDB

Architecture & Components

The architecture below shows we are using 3 AWS services

S3 bucket
Lambda Function
DynamoDB Table

A brief description of services below as refreshment:

S3 Bucket: Object storage service with scalability, security & high-performance service will be used as our storage service for the data
Lambda Function: Serverless compute service which allows you to run your code without worrying about the infrastructure, easy to setup and support a lot of programming languages, we will utilize it to run our code and deploy our logic.
DynamoDB: Serverless NoSQL database used to store our data in tables, we will use it to store our processed data by the Lambda function

Flow

User will upload JSON file to S3 bucket through console or CLI which behind the scenes PutObject API
Object is Uploaded successfully, S3 Event will be triggered to invoke the lambda function to load & process the file
Lambda will process the data and load it into DynamoDB table

Implementation Steps

We will walk through the steps & configuration for deploying the above diagram

1- Create Lambda Function with below Configuration

Author from Scratch
Function Name: ParserDemo
Runtime: Python 3.1x

Leave the rest as default
After Lambda created, you will need to modify the timeout configuration & Execution role as below:

I wrote this python code to perform the logic

import json
import boto3

s3_client = boto3.client('s3')
dynamodb = boto3.resource('dynamodb')

def lambda_handler(event, context):



    bucket_name = event['Records'][0]['s3']['bucket']['name'] # Getting the bucket name from the event triggered by S3
    object_key = event['Records'][0]['s3']['object']['key'] # Getting the Key of the item when the data is uploaded to S3
    print(f"Bucket: {bucket_name}, Key: {object_key}")


    response = s3_client.get_object(
    Bucket=bucket_name,
    Key=object_key
)


    # We will convert the streamed data into bytes
    json_data = response['Body'].read()
    string_formatted = json_data.decode('UTF-8') #Converting data into string

    dict_format_data = json.loads(string_formatted) #Converting Data into Dictionary 


    # Inserting Data Into DynamoDB

    table = dynamodb.Table('DemoTable')
    if isinstance(dict_format_data, list): #check if the file contains single record
        for record in dict_format_data:
            table.put_item(Item=record)

    elif isinstance(dict_format_data, dict): # check if the file contains multiple records 
        table.put_item(Item=data)

    else:  
        raise ValueError("Not Supported Format") # Raise error if nothing matched

2- Create S3 bucket

BucketName: use a unique name

leave the rest of configuration as default

Add the created S3 bucket as a trigger to lambda function as below:

3- Create a Table in the DynamoDB with the below configuration

Table Name: DemoTable
Partition Key: UserId
Table Settings: Customized
Capacity Mode: Provisioned

To Save costs configure the provisioned capacity units for read/write with low value (1 or 2 units)

Now the setup is ready, you can test it by uploading a file to the S3, then you will find items created on the DynamoDB table with the records you have uploaded into the file.

CloudWatch Logs for Lambda Function

DynamoDB Items

I hope you found this interesting and please let me know if you have any comments.

References

S3 API
DynamoDB API
boto3 practice for AWS services