DEV Community: Roman Tsypuk

From Redis to Valkey: pre-migration Reconnaissance — detect all apps & connections in realtime

Roman Tsypuk — Sun, 05 Apr 2026 17:57:21 +0000

Abstract

Abraham Lincoln: "Give me six hours to chop down a tree and I will spend the first four sharpening the axe"

Redis Valkey is getting more popular due to its performance increase compared to classic Redis version, I'm starting a seris of posts
related to migration from one vendors' Redis implemnetation to AWS Valkey.

To choose proper migration technics, the most important step is a reconnaissance of pre-migration. In this post I'll explain how native Redis features can help to identify all services that have connection to Redis (what is really hard in distributed environment of enterprise level infrastucture, that was created with periodically changed stack, languages, SDKs by multiple engineering generations).

Valkey project bried history

Redis first version release was in 2009, since that time it grown from cache key-value storage to PubSub, Stream, DB and used in a lot of projects.

Thus cloud providers AWS, GCP, Azure, Oracle started providing Redis as a managed service - allowing engineers offload cluster management and all heavy lifting to providers.

But Redis company provided own cloud - RedisCloud with Redis-managed-services. So to have more clients in RedisCloud and bigger monetization, starting from Redis version 7.4 the license was changed oriented to other cloud-providers to pay for Redis if it is offered as a managed service.

At that moment a git fork of Redis was created https://github.com/valkey-io/valkey under name Valkey and it was maintained both by open-source community and Cloud-providers, since this fork had original OS license. Redis is written on C.

Today there are 2 different repositories and projects, each having its own release version and code name:

Redis (https://github.com/redis/redis)
Valkey (https://github.com/valkey-io/valkey)

At this moment the latest release versions are:

Redis v8.6.2
Valkey v9.0.3 (on top of Redis v7.2.4)

So both projects are implementing different features in parallel, like multithreading that was added recently.

Pre-migration Reconnaissance

I'm preparing migration of Redis instances from Redis-cloud to AWS Valkey and before migration need info about data access patterns and all Redis producers/consumers.

Identify clients that are reading/writing to Redis:

There are different technics that can be used to identify all writers/readers of Redis - tracing and monitoring tools
like datadog, x-ray, analysys of ENV variables set for services with aliases to endpoint of Redis.

And things are getting more complex in real life. Do not be surprised to see in your enterprise-leve stack:

distributed environment
AWS multi-account deployments, privatelink, vpc-perrings established, etc.
tons of running services that are written in multiple languages (go, java, ts, ruby)
lack of documentation, stackholders, etc.

Here I will show the technical that I found extra useful, that is natively supported out of the box by Redis, does
not require installation of any third-party agents, monitoring stack, etc.

redis-cli

Redis out of the box has functionality that allows to gather information about clients. You need establish
connection to your Redis server through cli and execute commands.

LIST CLIENT

The CLIENT LIST command returns information and statistics about the client connections server in a mostly human
readable format.

You can use one of the optional subcommands to filter the list. The TYPE type subcommand filters the list by clients'
type, where type is one of normal, master, replica, and pubsub. Note that clients blocked by the MONITOR command
belong
to the normal class.

The ID filter only returns entries for clients with IDs matching the client-id arguments.

redis-cxxxxx.us-east-1.ec2.cloud.xxxx.com:6379> client list
id=3066004040000 addr=xx.xx.xx.135:30746 laddr=xx.20.4.124:18585 fd=4789 name= age=18701 idle=827 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 obl=0 events=r cmd=get user=xxx resp=2 lib-name=go-redis(,go1.24.13) lib-ver=9.17.2
id=3081428040000 addr=xx.xx.xx.145:31848 laddr=xx.20.4.124:18585 fd=4959 name= age=538 idle=221 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 obl=0 events=r cmd=get user=xxx resp=2 lib-name= lib-ver=
id=2956190040001 addr=xx.xx.xx.118:1604 laddr=xx.20.4.124:18585 fd=5117 name= age=148140 idle=5529 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 obl=0 events=r cmd=set user=xxx resp=2 lib-name= lib-ver=
id=3072343040001 addr=xx.xx.xx.126:46494 laddr=xx.20.4.124:18585 fd=4048 name= age=11493 idle=428 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 obl=0 events=r cmd=get user=xxx resp=2 lib-name=go-redis(,go1.24.13) lib-ver=9.17.2
id=3078769040001 addr=xx.xx.xx.173:9717 laddr=xx.20.4.124:18585 fd=4184 name= age=3714 idle=1 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 obl=0 events=r cmd=get user=xxx resp=2 lib-name=go-redis(,go1.24.13) lib-ver=9.17.2
id=3080824040001 addr=xx.xx.xx.158:35937 laddr=xx.20.4.124:18585 fd=5035 name= age=1267 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 obl=0 events=r cmd=get user=xxx resp=2 lib-name=go-redis(,go1.24.13) lib-ver=9.17.2
id=3081894040001 addr=xx.xx.xx.245:36487 laddr=xx.20.4.124:18585 fd=3686 name= age=26 idle=5 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 obl=0 events=r cmd=get user=xxx resp=2 lib-name= lib-ver=
...

The most interesting data:

id: a unique 64-bit client ID.
addr: address/port of the client.
laddr: address/port of local address client connected to (bind address).
name: the name set by the client with CLIENT SETNAME.
age: total duration of the connection in seconds.
idle: idle time of the connection in seconds.
db: current database ID.
cmd: last command played.
user: the authenticated username of the client.
lib-name - the name of the client library that is being used.
lib-ver - the version of the client library.

Having this information from active Redis will allow detecting all client, distinguish them by sesssionID and track
additional information about
library versio, library name, name of the client. Also additional details about the session.

MONITOR

MONITOR is a debugging command that streams back every command processed by the Redis server. It can help in
understanding what is happening to the database. This command can both be used via redis-cli and via telnet.

Because MONITOR streams back all commands, its use comes at a cost. The following (totally unscientific) benchmark
numbers illustrate what the cost of running MONITOR can be.

redis-cxxxxx.us-east-1.ec2.cloud.xxxx.com:6379> monitor
OK
1774787120.638084 [0 xx.xx.xx.222:40277] "get" "prefixkey:namespacea:default_data"
1774787120.649084 [0 xx.xx.xx.243:58512] "get" "prefixkey:namespace2:default_data"
1774787120.652084 [0 xx.xx.xx.222:40277] "get" "prefixkey:namespace3:default_data"
1774787120.687083 [0 xx.xx.xx.50:64040] "zrange" "prefixkey:namespace4:allowed_data" "0" "-1"

CLIENTNAME feature

Output from these commands will be enough in 99% of cases. If you are hosting your apps on EC2, EKS,
ECS, lambda functions - you can easily map IP addresses to dedicated pods/container IPs.

But I got a corner-case - the source Redis is running in RedisLabs - it is a cloud environment that provides
redis-as-a-service (under the hood deploying it to cloud infrasstructure, in case AWS EC2 instances).

But Consumer apps are running in kubernetes cluster and since RedisLabs is hosted externally out of AWS account the network
traffic flows through k8s cluster node.

Since on every kubernetes node is running a lot of payloads in each pod, but all of them when connecting to Redis will have
the same IP address — the IP address of the cluster node on which they are hosted.

So this info can limit the blast radius of services but since node afinity is not in use and due to big landscape of
services, it is hard to identify Redis writers/readers - all pods will have same address set of Cluster nodes.

And here comes CLIENT_NAME - this feature is available since Redis Open Source 2.6.9 - it assigns a name to the
current connection.

The assigned name is displayed in the output of CLIENT LIST so that it is possible to identify the client that
performed a given connection.

However, it is not possible to use spaces in the connection name as this would violate the format of the CLIENT LIST
reply.

Every new connection starts without an assigned name.

setting names to connections is a good way to debug connection leaks due to bugs in the application using Redis.

Once clients are instrumented and deployed, using redis-cli we can check the data.

Writing tool to track clients and operations on Redis:

To make this information in a table format I have created python-based tool that interacts with Redis using RESP
protocol and renders information:

┌────────────────┬─────────────────┬───────────────────────┬─────────┬─────────┬──────────────────────────────────┬─────┐
│ Client IP      │ Name            │ Lib                   │ Lib Ver │ User    │ Full Key                         │ GET │
├────────────────┼─────────────────┼───────────────────────┼─────────┼─────────┼──────────────────────────────────┼─────┤
│ 10.xx.xx.53    │                 │                       │         │ xxxx    │                                  │   0 │
│ 10.xx.xx.173   │                 │ go-redis(,go1.21.1)   │ 8.0.2   │ xxxx    │ prefixkey:namespace1:default_data│   1 │
│                │                 │                       │         │         │ prefixkey:namespace2:default_data│   1 │
│ 10.100.238.99  │                 │                       │         │ xxxx    │                                  │   0 │
│ 10.100.244.76  │                 │                       │         │ xxxx    │                                  │   0 │
│ 10.100.75.149  │                 │ go-redis(,go1.24.13)  │ 9.17.2  │ default │ prefixkey:namespace1:default_data│   1 │
│                │                 │                       │         │         │ prefixkey:namespace1:default_data│   1 │
│ 10.104.127.44  │                 │                       │         │ default │                                  │   0 │
│ 10.104.136.207 │                 │ go-redis(,go1.24.13)  │ 9.17.2  │ default │                                  │   0 │
│ 10.104.94.31   │                 │                       │         │ default │                                  │   0 │
│ 10.170.3.253   │ monitor-v2-prod │ python-socket-monitor │ 1.0     │ default │                                  │   0 │
└────────────────┴─────────────────┴───────────────────────┴─────────┴─────────┴──────────────────────────────────┴─────┘

For some period in realtime we can track what operations are performed on Redis server and who are the client.
Once all of them are identified and there no unknown areas it's time to analyze Redis instance and plan migraion.

Instrument client with clientName

Redis allows instrument redis SDK with name of the client to be identified on every connection. Here is go-lang code
snippet:

rdb := redis.NewClient(&redis.Options{
    Addr:       "REDIS_HOST:REDIS_PORT",
    Password:   "REDIS_PASSS",
    DB:         0,
    ClientName: "service1-writer",
})

Same can be done on every programming languages libraries, or even at the low-level of socket with Redis RESP-protocol
level:

def send_command(sock, *args):
  """Send a Redis RESP protocol command."""
  command = f"*{len(args)}\r\n"
  for arg in args:
    arg_str = str(arg)
    command += f"${len(arg_str)}\r\n{arg_str}\r\n"
  sock.sendall(command.encode("utf-8"))


def set_client_name(sock, client_name):
  """Assign a client name to the current Redis connection."""
  if not client_name:
    return
  send_command(sock, "CLIENT", "SETNAME", client_name)
  response = read_line(sock)
  if not response.startswith("+OK"):
    raise ConnectionError(f"CLIENT SETNAME failed: {response}")


def set_client_info(sock, lib_name=None, lib_version=None):
  """Assign client library metadata to the current Redis connection."""
  for attribute, value in (("LIB-NAME", lib_name), ("LIB-VER", lib_version)):
    if not value:
      continue
    send_command(sock, "CLIENT", "SETINFO", attribute, value)
    response = read_line(sock)
    # Redis < 7.2 may not support CLIENT SETINFO, so ignore failures.
    if response.startswith("-"):
      continue

Challenge 1: that there are too many connections to this RedisDB

With our custom monitor tool, now we are tracking all clients with details and suddenly see that there are too many
clients and keys that are being accessed.
Since Redis instance is used by multiple service and we are interested in extraction from it only specific prefix
pattern key, we can modify the monitor tool to track only those prefixes access.

add predicate for prefix of keyset—after that, you will see only client names, IPs and operations to a limited subset
of keystore are that is a subject of extraction.
{: .prompt-danger }

Challenge 2: client versions are too old and will not support Valkey

It's important to check SDK client versions—do not be surprised that you are using outdated Redis and outdated
libraries.
In this case, if you provision the latest AWS Valkey and will try to just change the endpoint URL in apps config that
will not work - you apps will get error connections in RESP procol and commands.

upgrade and align client versions to the latest supported redis SDK
{: .prompt-danger }

Completed redis client versions and clientnames alignment

After upgrading all clients code to latest golang redis library version (that is for today is 9.18.0) and adding client
name to redis sessions, now we can review active connections and see what keys are being accessed and ready for the
planning the actual migration:

┌────────────────┬─────────────────┬───────────────────────┬─────────┬─────────┬──────────────────────────────────┬─────┐
│ Client IP      │ Name            │ Lib                   │ Lib Ver │ User    │ Full Key                         │ GET │
├────────────────┼─────────────────┼───────────────────────┼─────────┼─────────┼──────────────────────────────────┼─────┤
│ 10.xx.xx.53    │                 │                       │         │ xxxx    │                                  │   0 │
│ 10.xx.xx.173   │ service1-writer │ go-redis-(.go1.24.13) │ 9.18.0  │ xxxx    │ prefixkey:namespace1:default_data│   1 │
│                │                 │                       │         │         │ prefixkey:namespace2:default_data│   1 │
│ 10.xx.xx.149   │ service2-reader │ go-redis-(.go1.24.13) │ 9.18.0  │ xxxx    │ prefixkey:namespace1:default_data│   1 │
│ 10.xx.xx.142   │ service3-reader │ go-redis-(.go1.24.13) │ 9.18.0  │ xxxx    │ prefixkey:namespace1:default_data│   1 │
│ 10.xx.xx.141   │ service4-reader │ go-redis-(.go1.24.13) │ 9.18.0  │ xxxx    │ prefixkey:namespace1:default_data│   1 │
│ 10.xx.xx.129   │ service5-writer │ go-redis-(.go1.24.13) │ 9.18.0  │ xxxx    │ prefixkey:namespace1:default_data│   1 │
│ 10.xx.xx.159   │ service3-reader │ go-redis-(.go1.24.13) │ 9.18.0  │  xxxx   │ prefixkey:namespace1:default_data│   1 │
│ 10.xx.xx.44    │                 │                       │         │ xxxx    │                                  │   0 │
│ 10.xx.xx.207   │ service2-reader │ go-redis-(.go1.24.13) │ 9.18.0  │ xxxx    │                                  │   0 │
│ 10.xx.xx.31    │                 │                       │         │ xxxx    │                                  │   0 │
│ 10.xx.xx.253   │ monitor-v2-prod │ python-socket-monitor │ 1.0     │ xxxx    │                                  │   0 │
└────────────────┴─────────────────┴───────────────────────┴─────────┴─────────┴──────────────────────────────────┴─────┘

On top of this tool can be added read/write ios per client measurement, key access detection or any analytic
operations for troubleshooting & refactoring. Its intention is not limited to pre-migration analysis.
{: .prompt-danger }

Reconnaissance step output summary

updated and aligned library versions of all Redis SDKs
properly trackable client names and all connections
fully identified all writers/readers that are planned to be migrated to AWS Valkey
clear understanding of data access patterns (read/write to keystore)

In the next post, I’ll dive into Redis topologies and the critical components to consider when preparing and running a migration, as they depend on the chosen topology.

How switching to SQS Batch operations improves Performance an Billing

Roman Tsypuk — Tue, 23 Dec 2025 19:36:59 +0000

Abstract

In this post, we explore how refactoring SQS message processing from individual SendMessage calls to Batch SendMessage operations can
significantly improve application performance and reduce SQS billing costs by lowering IOPS usage.

The idea

When monitoring a Golang application with DataDog, we can measure SQS message sending in detail. By comparing a traditional loop-based send approach versus batch sending, we can see clear differences in timing, network calls, and resource usage.

Full Datadog tracing of SQS is not supported for all languages:

Set DD_TRACE_CLOUD_REQUEST_PAYLOAD_TAGGING=all and DD_TRACE_CLOUD_RESPONSE_PAYLOAD_TAGGING=all on this service to enable complete payload tagging.

https://docs.datadoghq.com/tracing/guide/aws_payload_tagging/?tab=nodejs

DD_TRACE_CLOUD_REQUEST_PAYLOAD_TAGGING=all
DD_TRACE_CLOUD_RESPONSE_PAYLOAD_TAGGING=all

For Golang, you can leverage Datadog attribute tags to inspect payload metadata.

Regular SQS message send operations

Sending messages one by one involves multiple network calls and extra overhead.

Following tracing diagram displays how timing looks like when using loop operation.

For example, sending 7 messages individually took 175ms, with 7 separate HTTP requests. The first call typically dominates the timing due to DNS lookup and connection setup.

But since the service in running in same K8S cluster, we can assume that experiment is clean and no additional overhead is present.

Sending messages in a Batch

AWS SQS allows sending up to 10 messages per batch. Sending 20 messages in 2 batches demonstrates significant efficiency gains:

Sent 3x more messages.
Made 10x fewer HTTP requests.
Total processing time reduced by ~3x.

Response examples:

When batch send is performed in a batch response, for each given message there is a status, including the error.
So the batch can be completed, but some messages in a batch can fail, parsing this response status will allow to efficiently
replay or handle with a fallback business logic of such cases.



{
  "Successful": [
    {
      "ID": "0",
      "MessageID": "655f3404-fbe4-4c51-8868-b5c604bd5f6d",
      "Error": null
    },
    {
      "ID": "1",
      "MessageID": "daf36653-9abb-490b-b620-608efa24a219",
      "Error": null
    },
    {
      "ID": "2",
      "MessageID": "93f4dcfd-0500-4076-90f2-3b880b32c943",
      "Error": null
    },
    {
      "ID": "3",
      "MessageID": "f6c7b079-98f5-4290-b293-2ac6e43ed6f2",
      "Error": null
    },
    {
      "ID": "4",
      "MessageID": "2b4a96bc-b4ec-4711-9473-d887dd3213f7",
      "Error": null
    },
    {
      "ID": "5",
      "MessageID": "1bd30cd9-f9c1-4b47-8d6d-2e23ce771841",
      "Error": null
    },
    {
      "ID": "6",
      "MessageID": "8eed75ef-2563-442e-a191-6b3dff29d635",
      "Error": null
    },
    {
      "ID": "7",
      "MessageID": "c65a36ce-7ce0-444c-9974-96648dcae0ea",
      "Error": null
    },
    {
      "ID": "8",
      "MessageID": "75379265-52f9-4a60-8c3a-0537cffdaa80",
      "Error": null
    },
    {
      "ID": "9",
      "MessageID": "59239903-d4d9-498f-9a08-6d7d7ae8beba",
      "Error": null
    },
    {
      "ID": "10",
      "MessageID": "9a614c58-113b-487d-a8f1-7509f93b42f9",
      "Error": null
    },
    {
      "ID": "11",
      "MessageID": "1077de5c-8f0f-4d5b-a0fe-dca45712bfdf",
      "Error": null
    },
    {
      "ID": "12",
      "MessageID": "8b0f5836-0e01-4a88-9793-4bac2a6d879a",
      "Error": null
    }
  ],
  "Failed": []
}

AWS Console Behavior

Batch sending does not change how messages appear in SQS. Each message is stored individually, so consumers don’t need any changes to handle batches.

Same messages, same structures are posted and present in SQS.

However, there are other optimization technics that allow optimise consumer batch size when polling messages from SQS.

Golang Implementation Example

entry := &sqs.SendMessageBatchRequestEntry{
  Id:          aws.String(fmt.Sprintf("%d", i+idx)), // Unique ID within batch
  MessageBody: aws.String(string(b)),
}

if taskConfig.MessageGroupId != "" {
  entry.SetMessageGroupId(taskConfig.MessageGroupId)
}
if taskConfig.MessageDeduplicationId != "" {
  entry.SetMessageDeduplicationId(taskConfig.MessageDeduplicationId)
}
if taskConfig.DelaySeconds > 0 {
  entry.SetDelaySeconds(taskConfig.DelaySeconds)
}
entries = append(entries, entry)
}

type BatchResult struct {
    Successful []BatchResultEntry
    Failed     []BatchResultEntry
}

// BatchResultEntry represents a single entry in a batch result
type BatchResultEntry struct {
    ID        string
    MessageID string
    Error     error
}

// Send batch
input := &sqs.SendMessageBatchInput{
  QueueUrl: stp.url,
  Entries:  entries,
}

output, err := stp.c.SendMessageBatchWithContext(ctx, input)
if err != nil {
  err = handleSqsErrors(err)
  // Mark all entries in this batch as failed
  for idx := range batch {
    result.Failed = append(result.Failed, BatchResultEntry{
      ID:    fmt.Sprintf("%d", i+idx),
      Error: err,
    })
  }
  continue
}

Dedicated message details

Exactly this messageID was returned in a batch is success response section.

Additional things to check and optimize

Deduplication technic

Before sending the messages, perform the deduplication—this will reduce the SQS iops usage, decrease the latency of processing and reduce the load on the consumer side, avoid unneeded storage read queries, rewrites, etc.

Distributed Tracing frameworks can consume SQS batch slot for metaiformation

Some frameworks for distributed tracing are propagating metainformation through async transports like SQS.
If you are using them, check integrations, it can affect the max batch size. I.e. Datadog uses 1 batch element to propagate metainformation with tracing,
that will be consumed and applied with span to a same trace.

x-ray since it is a proprietary AWS technology does not utilize any slots in a SQS batch and uses UDP server to submit span/trace info.

Limitations:

message size payload (1Mb)
batch size (10 messages)
payload serialization (JSON)

Conclusions:

Switching implementation from loop Send to Batch Send, allowed significantly decrease the overall timing, decrease network round trips and as a bonus decrease the SQS billing (due to decrease API calls in x10 times).

Orchestrating AI multi-agent infrastructure with AWS Bedrock, OpenAI and n8n

Roman Tsypuk — Fri, 26 Sep 2025 17:39:06 +0000

Abstract

This article explores how to build a multi-agent AI ecosystem using n8n, AWS Bedrock, OpenAI, and MCP servers—all with a no-code approach.

The idea

Each AI agent is designed with its own dedicated model (optimized for its role) and separate memory storage (ensuring context persistence and isolation). By connecting agents to AWS documentation via MCP, custom AWS news feeds via JSON, and enabling agent-to-agent communication, we demonstrate how to create a flexible system that interacts directly based on user prompts.

What is AI-agent and its parts

An AI agent is not just a single model—it’s a structured system made up of several interconnected components. Think of
it as a worker in a digital team, equipped with a brain, memory, and tools.

Core Components of an AI Agent

LLM Model: The reasoning engine. Can be OpenAI GPT, Anthropic Claude (via AWS Bedrock), Amazon Titan, or any other
large language model.
Responsible for interpreting prompts, generating responses, and orchestrating tool usage.
Memory: A storage layer where past interactions and context are recorded.
Often implemented as a database table or key/value storage. Ensures continuity—so the agent doesn’t “forget” what was
asked earlier.
Tools: Interfaces that extend the agent’s capabilities. Examples: HTTP endpoints, databases, MCP services, or custom
APIs.
Tools allow the agent to access real-time or domain-specific data beyond the model’s training cutoff.

Connecting to AWS documentation server as MCP

AWS offers a free streaming service for official documentation, and we can integrate it into our agent through the Model
Context Protocol (MCP).

By registering MCP client tools, our AI agent gains the ability to search, read, and recommend content directly from AWS
docs.

AWS MCP configuration settings:

Parameter	Value
Endpoint	https://knowledge-mcp.global.api.aws
Server Transport	HTTP streamable
Authentication	none
Tools	read, recommend, search

With this setup, whenever a user asks something like “How do I configure DynamoDB streams?”, the agent can fetch the
latest instructions directly from the AWS documentation server.

Adding AI agent tools for AWS news

Besides documentation, agents can also consume custom news feeds. I maintain a curated set of AWS news in JSON format,
hosted on GitHub Pages. These feeds cover categories such as architecture, big data, and machine learning.

By connecting the agent’s HTTP tool to these feeds, we can provide real-time AWS announcements and updates inside the
agent’s workflow.

REST http tools setup:

Tool	details
Tool info	Makes an HTTP request and returns the latest aws news Training Certification
Tool endpoint	https://tsypuk.github.io/aws-news/news/training_certification.json

Tool	details
Tool info	Makes an HTTP request and returns the latest aws news Architecture
Tool endpoint	https://tsypuk.github.io/aws-news/news/architecture.json

The full list of feeds is available in my repo: tsypuk/aws-news

With this integration, a user can ask: “What are the latest updates in AWS architecture?”, and the agent will pull fresh
content directly from the JSON feed.

You can get and use the full list of news at my aws news repo

Full Agent2Agent connection and communication

Once we have multiple agents ecosystem (Documentation Agent + News Agent), we can introduce a third agent as
orchestrator.

The Orchestrator Agent decides which agent to query.
The Documentation Agent connects via MCP to AWS docs.
The News Agent pulls JSON feeds with announcements.

Example:

Prompt: “Explain the latest DynamoDB updates and show me how to configure them.”
Orchestrator → News Agent: get latest DynamoDB announcements.
Orchestrator → Documentation Agent: fetch relevant setup docs.
Orchestrator synthesizes the results into a single, user-friendly answer.

The beauty of such configuration that each individual agent can be tuned and configured separately, i.e. we can set
different model for each agent - for some agent will be enough small model, so there is no need
to use more expensive model; we can use a specific model that provides better performance for a particular domain or
trained on a dedicated dataset.

This separation lowers costs while improving domain accuracy.

Checking the Memory storage

AI agents persist interaction history in Memory, which is critical for:

Maintaining context across multi-turn conversations.
Remembering what data was fetched previously.
Enabling personalization (e.g., user preferences).

Memory storage is organized as a simple table:

create table n8n_chat_reseach_histories
(
    id         serial
        primary key,
    session_id varchar(255) not null,
    message    jsonb        not null
);

Let's check how it is populated with data based on different agent flows communication.

Request of flow through aws news agent:

what are the latest news in s3 storage?
{: .prompt-tip }

Here are the latest AWS blog items related to Amazon S3 storage, with source links:
Amazon SageMaker introduces Amazon S3-based shared storage for enhanced project collaboration. Link: https://aws.amazon.com/blogs/big-data/amazon-sagemaker-introduces-amazon-s3-based-shared-storage-for-enhanced-project-collaboration/
Optimize Amazon EMR runtime for Apache Spark with EMR S3A. Link: https://aws.amazon.com/blogs/big-data/optimize-amazon-emr-runtime-for-apache-spark-with-emr-s3a/
A scalable, elastic database and search solution for 1B+ vectors built on LanceDB and Amazon S3. Link: https://aws.amazon.com/blogs/architecture/a-scalable-elastic-database-and-search-solution-for-1b-vectors-built-on-lancedb-and-amazon-s3/

Sources:

https://aws.amazon.com/blogs/big-data/amazon-sagemaker-introduces-amazon-s3-based-shared-storage-for-enhanced-project-collaboration/
https://aws.amazon.com/blogs/big-data/optimize-amazon-emr-runtime-for-apache-spark-with-emr-s3a/
https://aws.amazon.com/blogs/architecture/a-scalable-elastic-database-and-search-solution-for-1b-vectors-built-on-lancedb-and-amazon-s3/

Param	Value
sessionid	15925d170d1e4ce3956667dbb9b172ba
action	sendMessage
chatInput	what are the latest news in s3 storage?

Next, we see that agent called Postgres to load Memory, but since it is a first interaction, there is empty chat history.
Also, we can trace agent to agent communications.

Agent calls tools for bigdata and for architecture to get the latest info about S3.
Just before the response, results are persisted in the memory storage:

Data from main AI-agent memory

id	session_id	message
3	15925d170d1e4ce3956667dbb9b172ba	{"type": "human", "content": "what are the latest news in s3 storage?. Include links to all used sources.", "additional_kwargs": {}, "response_metadata": {}}
4	15925d170d1e4ce3956667dbb9b172ba	{"type": "ai", "content": "Here are the latest AWS blog items related to Amazon S3 storage, with source links:\n\n- Amazon SageMaker introduces Amazon S3-based shared storage for enhanced project collaboration. Link: https://aws.amazon.com/blogs/big-data/amazon-sagemaker-introduces-amazon-s3-based-shared-storage-for-enhanced-project-collaboration/\\n\\n- Optimize Amazon EMR runtime for Apache Spark with EMR S3A. Link: https://aws.amazon.com/blogs/big-data/optimize-amazon-emr-runtime-for-apache-spark-with-emr-s3a/\\n\\n- A scalable, elastic database and search solution for 1B+ vectors built on LanceDB and Amazon S3. Link: https://aws.amazon.com/blogs/architecture/a-scalable-elastic-database-and-search-solution-for-1b-vectors-built-on-lancedb-and-amazon-s3/\\n\\nSources:\\n- https://aws.amazon.com/blogs/big-data/amazon-sagemaker-introduces-amazon-s3-based-shared-storage-for-enhanced-project-collaboration/\\n- https://aws.amazon.com/blogs/big-data/optimize-amazon-emr-runtime-for-apache-spark-with-emr-s3a/\\n- https://aws.amazon.com/blogs/architecture/a-scalable-elastic-database-and-search-solution-for-1b-vectors-built-on-lancedb-and-amazon-s3/", "tool_calls": [], "additional_kwargs": {}, "response_metadata": {}, "invalid_tool_calls": []}

Data from latest-news AI-agent memory

id	session_id	message
73	15925d170d1e4ce3956667dbb9b172ba	{"type": "human", "content": "You are news facts agent. Should provide accurate response based on latest news, in the response add section with links to all sources that you have used.\nPlease provide the latest news related to Amazon S3 storage, with links to all sources.", "additional_kwargs": {}, "response_metadata": {}}
74	15925d170d1e4ce3956667dbb9b172ba	{"type": "ai", "content": "Here are the latest AWS blog items related to Amazon S3 storage:\n\n- Amazon SageMaker introduces Amazon S3 based shared storage for enhanced project collaboration. Link: https://aws.amazon.com/blogs/big-data/amazon-sagemaker-introduces-amazon-s3-based-shared-storage-for-enhanced-project-collaboration/\\n\\n- Optimize Amazon EMR runtime for Apache Spark with EMR S3A. Link: https://aws.amazon.com/blogs/big-data/optimize-amazon-emr-runtime-for-apache-spark-with-emr-s3a/\\n\\n- A scalable, elastic database and search solution for 1B+ vectors built on LanceDB and Amazon S3. Link: https://aws.amazon.com/blogs/architecture/a-scalable-elastic-database-and-search-solution-for-1b-vectors-built-on-lancedb-and-amazon-s3/\\n\\nSources\\n- https://aws.amazon.com/blogs/big-data/amazon-sagemaker-introduces-amazon-s3-based-shared-storage-for-enhanced-project-collaboration/\\n- https://aws.amazon.com/blogs/big-data/optimize-amazon-emr-runtime-for-apache-spark-with-emr-s3a/\\n- https://aws.amazon.com/blogs/architecture/a-scalable-elastic-database-and-search-solution-for-1b-vectors-built-on-lancedb-and-amazon-s3/", "tool_calls": [], "additional_kwargs": {}, "response_metadata": {}, "invalid_tool_calls": []}

Amazing that n8n stores history of executions and allows visually represent flows execution - what path the ai-agent has chosen, what tools were invoked, etc.:

Request of flow through documentation agent

what is the supported knowledge base storages for aws bedrock?
{: .prompt-tip }

Here are the storage options AWS Bedrock documents list as supported for Bedrock Knowledge Bases (for Retrieval Augmented Generation), with the official source links. Where possible, I note the exact integration concept and point you to the corresponding AWS docs for details on configuration and fields.

Amazon OpenSearch Serverless (vector store)
What it is: A vector search collection in OpenSearch Serverless used as Bedrock’s vector store for embeddings.
How it integrates with Bedrock Knowledge Bases: Bedrock connects to an OpenSearch Serverless vector collection and stores embeddings plus text chunks and Bedrock metadata; mappings/fields are defined in the knowledge-base setup flow.
Key docs/fields: Bedrock knowledge-base-setup guidance covers how to configure the OpenSearch Serverless integration.
Source: https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base-setup.html
Amazon OpenSearch Service (Managed Clusters) as vector store
...

Data from main AI-agent memory

id	session_id	message
7	15925d170d1e4ce3956667dbb9b172ba	{"type": "human", "content": "what is the supported knowledge base storages for aws bedrock. Include links to all used sources.", "additional_kwargs": {}, "response_metadata": {}}
8	15925d170d1e4ce3956667dbb9b172ba	{"type": "ai", "content": "Here are the storage options AWS Bedrock documents list as supported for Bedrock Knowledge Bases (for Retrieval Augmented Generation), with the official source links. Where possible, I note the exact integration concept and point you to the corresponding AWS docs for details on configuration and fields.\n\n1) Amazon OpenSearch Serverless (vector store)\n- What it is: A vector search collection in OpenSearch Serverless used as Bedrock’s vector store for embeddings.\n- How it integrates with Bedrock Knowledge Bases: Bedrock connects to an OpenSearch Serverless vector collection and stores embeddings plus text chunks and Bedrock metadata; mappings/fields are defined in the knowledge-base setup flow.\n- Key docs/fields: Bedrock knowledge-base-setup guidance covers how to configure the OpenSearch Serverless integration.\n- Source: https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base-setup.html\\n\\n2\) Amazon OpenSearch Service (Managed Clusters) as vector store\n- What it is: A managed OpenSearch Service domain used as a vector store for Bedrock Knowledge Bases.\n- How it integrates with Bedrock Knowledge Bases: Create a vector index on the OpenSearch domain and connect it to the Bedrock KB; mappings include vector index name, vector field, and metadata field, plus Bedrock’s text chunk fields.\n- Key docs/fields: Bedrock knowledge-base-setup guidance (OpenSearch Clusters section).\n- Source: https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base-setup.html\\n\\n3\) Amazon S3 Vectors (vector store in S3)\n- What it is: Bedrock-supported vector storage that stores embeddings in S3-based vector structures.\n- How it integrates with Bedrock Knowledge Bases: Configure an S3 vector bucket/index; Bedrock stores embeddings and metadata and uses defined mappings for fields.\n- Key docs/fields: Bedrock knowledge-base-setup guidance (S3 Vectors section).\n- Source: https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base-setup.html\\n\\n4\) Amazon Aurora (RDS) as vector store\n- What it is: Aurora database (PostgreSQL-compatible) used as the relational vector store for Bedrock Knowledge Bases.\n- How it integrates with Bedrock Knowledge Bases: Bedrock expects a relational table with embedding/vector, text chunks, and metadata columns; Bedrock maps UI/API fields to the table schema.\n- Key docs/fields: Bedrock knowledge-base-setup guidance (Aurora section).\n- Source: https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base-setup.html\\n\\n5\) Neptune Analytics graphs (GraphRAG)\n- What it is: Neptune Analytics graphs used with GraphRAG as a Bedrock knowledge base storage option.\n- How it integrates with Bedrock Knowledge Bases: Create a Neptune Analytics graph and connect it to Bedrock KB; configure a vector index on the graph and map Bedrock embeddings/text/metadata to graph fields.\n- Key docs/fields: Bedrock knowledge-base-setup guidance (Neptune Analytics/GraphRAG section).\n- Source: https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base-setup.html\\n\\n6\) Pinecone (vector store)\n- What it is: Pinecone vector index service used as a Bedrock Knowledge Bases vector store.\n- How it integrates with Bedrock Knowledge Bases: Provide Pinecone index name/endpoint and credentials (via Secrets Manager); Bedrock stores embeddings and metadata and maps fields accordingly.\n- Key docs/fields: Bedrock knowledge-base-setup guidance (Pinecone section).\n- Source: https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base-setup.html\\n\\n7\) Redis Enterprise Cloud (vector store)\n- What it is: Redis Enterprise Cloud used as a Bedrock Knowledge Bases vector store.\n- How it integrates with Bedrock Knowledge Bases: Configure endpoint, vector index, vector field, metadata field; credentials via Secrets Manager and TLS settings as needed.\n- Key docs/fields: Bedrock knowledge-base-setup guidance (Redis Enterprise Cloud section).\n- Source: https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base-setup.html\\n\\n8\) MongoDB Atlas (vector store)\n- What it is: MongoDB Atlas used as a Bedrock Knowledge Bases vector store.\n- How it integrates with Bedrock Knowledge Bases: Connect to Atlas, specify database/collection and vector index; Bedrock stores embeddings, text chunks, and metadata in the Atlas collection.\n- Key docs/fields: Bedrock knowledge-base-setup guidance (MongoDB Atlas section).\n- Source: https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base-setup.html\\n\\n9\) Kendra GenAI index (special integration path)\n- What it is: Build a Bedrock knowledge base from an Amazon Kendra GenAI index to combine Kendra’s retrieval with Bedrock’s generation.\n- How it integrates with Bedrock Knowledge Bases: Create a KB from a Kendra GenAI index to reuse indexed content with Bedrock GenAI.\n- Key docs/fields: Bedrock knowledge-base-build-kendra-genai-index guidance.\n- Source: https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base-build-kendra-genai-index.html\\n\\nAdditional context and related docs\n- Overview and retrieval concepts for Bedrock Knowledge Bases: https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base.html and https://docs.aws.amazon.com/bedrock/latest/userguide/kb-how-retrieval.html\\n- Prerequisites and setup for vector-store usage: https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base-prereq.html\\n- Build or connect a knowledge base (general flow): https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base-build.html\\n- Kendra GenAI integration path (detailed steps): https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base-build-kendra-genai-index.html\\n\\nWhat’s New / related announcements (contextual confirmations)\n- OpenSearch-based vector storage support for Bedrock Knowledge Bases (GA/availability note): \n https://aws.amazon.com/about-aws/whats-new/2025/03/amazon-bedrock-knowledge-bases-opensearch-cluster-vector-storage/\\n- MongoDB Atlas vector storage support for Bedrock Knowledge Bases:\n https://aws.amazon.com/about-aws/whats-new/2024/05/knowledge-bases-amazon-bedrock-mongodb-atlas-vector-storage/\\n\\nIf you’d like, I can distill this into a compact quick-reference table (one line per option with integration point and official doc URL), or provide a per-option summary with exact field names as shown in the Bedrock setup docs. All information above is drawn from official AWS Bedrock documentation and related AWS What’s New posts linked here.", "tool_calls": [], "additional_kwargs": {}, "response_metadata": {}, "invalid_tool_calls": []}

Data from aws documentation AI-agent memory

id	session_id	message
1	15925d170d1e4ce3956667dbb9b172ba	{"type": "human", "content": "You are research agent. Should provide accurate response based on documentation and statements, in the response add section with links to all sources that you have used.\nPlease provide the official AWS Bedrock documentation on supported knowledge base storages. Specifically, list the storage options (e.g., vector stores or document stores) that Bedrock can use for retrieval-augmented generation, or knowledge bases, including any required formats, integrations (such as S3, Kendra, etc.), and limitations. Include links to all used sources with exact URLs. If multiple AWS services are involved (Kendra, S3, etc.), include their relevant Bedrock integration docs. Provide only verifiable information from AWS docs.", "additional_kwargs": {}, "response_metadata": {}}
2	15925d170d1e4ce3956667dbb9b172ba	{"type": "ai", "content": "Below is a consolidated, source-backed view of the official AWS Bedrock documentation on supported knowledge base storages (the vector stores and document stores Bedrock Knowledge Bases can use for Retrieval Augmented Generation, or knowledge bases). For each option, I note what it is, how it integrates with Bedrock Knowledge Bases, any required formats or fields, and any limitations or special notes found in AWS docs. Links to the exact AWS sources are provided at the end.\n\nWhat Bedrock Knowledge Bases can use for storage (summary)\n- Bedrock Knowledge Bases support storing and indexing data in multiple vector stores (and related data stores) to power RAG. Data from your sources is converted into vector embeddings and stored in a chosen vector store; Bedrock then retrieves relevant vectors and uses them to augment generation.\n- You can either connect to a data source directly (unstructured or structured) or use Bedrock’s built-in/managed vector stores; you can also create a knowledge base by connecting to a data source, or build a knowledge base with a Kendra GenAI index.\n- Important notes:\n - Multimodal data (text plus images, charts, etc.) is supported only with Amazon S3 and custom data sources.\n - Some vector stores support binary embeddings, others only floating-point embeddings; the available options and capabilities depend on the store.\n - Vector stores require index/collection setup, including mapping fields for embeddings, text chunks, and metadata, as described in each store’s setup.\n - There are integration previews/GA status and regional availability notes in various sources (see specific entries below).\n- Relevant Bedrock doc sections and integration pages include guides on building and using knowledge bases, setting up vector stores, and special integration options with Kendra GenAI.\n\nDetailed storage options (Bedrock Knowledge Bases) with integration details\n\n1) Amazon OpenSearch Serverless (vector storage)\n- What it is: A vector store option via Amazon OpenSearch Serverless that Bedrock Knowledge Bases can use for vector search indexing of embeddings.\n- How it integrates with Bedrock:\n - You configure a vector search collection in OpenSearch Serverless and connect it to the Bedrock knowledge base as the vector store.\n - You must align the vector embedding dimensions with the embeddings model you use.\n - In Bedrock’s knowledge base setup, you map Bedrock to OpenSearch Serverless, including fields for embeddings, text chunks, and Bedrock-managed metadata.\n- Required formats and fields:\n - Vector index configuration in OpenSearch Serverless with a vector field (embeddings) and metadata/text fields (e.g., AMAZON_BEDROCK_TEXT_CHUNK, AMAZON_BEDROCK_METADATA) as part of the index mapping.\n - The embedding space is configured to use the selected embedding model; the vector type is typically a knn_vector with engine faiss and a suitable distance metric (euclidean commonly recommended for floating-point embeddings).\n- Limitations / notes:\n - Guidance includes specific dimension recommendations and how to map Bedrock’s data into the vector index.\n - Documentation notes the OpenSearch Serverless integration as a supported vector store; see the knowledge base setup materials for exact mapping details.\n- Source:\n - Build a knowledge base by connecting to a data source (OpenSearch Serverless section) and general knowledge base setup (knowledge-base-setup.html)\n - Knowledge Base setup text references OpenSearch Serverless as a vector store option and provides detailed steps and field mappings\n - Knowledge-base-setup URL: https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base-setup.html\\n\\n2\) Amazon OpenSearch Service Managed Clusters (vector storage)\n- What it is: A vector store option via Amazon OpenSearch Service domains (managed OpenSearch clusters) used as a vector store for Bedrock Knowledge Bases.\n- How it integrates with Bedrock:\n - You create a vector index in the OpenSearch Service domain and connect it to the Bedrock knowledge base.\n - Bedrock provides mappings for domain ARN, domain endpoint, vector index name, vector field name, and metadata field name.\n- Required formats and fields:\n - Mappings include vectorIndexName, vectorField (embedding field), and metadataField, plus Bedrock-specific text chunk fields.\n- Limitations / notes:\n - Prerequisites and permissions for using OpenSearch Managed Clusters are documented; there are guidance notes on permissions, encryption, and indexing requirements.\n - Similar to OpenSearch Serverless, there are dimensionality requirements and embedding-field configurations you must align with your embeddings model.\n- Source:\n - Knowledge-base-setup.html (OpenSearch Clusters section)\n - knowledge-base-setup page content includes both OpenSearch Serverless and OpenSearch Managed Clusters guidance\n - Knowledge-base-setup URL: https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base-setup.html\\n\\n3\) Amazon S3 Vectors (vector storage in S3)\n- What it is: A vector store option that stores vector embeddings in S3 Vectors (S3-based vector storage). This is a Bedrock-supported way to keep embeddings in S3 with vector indexing.\n- How it integrates with Bedrock:\n - You set up an S3 vector bucket and a corresponding vector index; Bedrock stores embeddings and metadata in S3-based vector structures.\n- Required formats and fields:\n - Vector bucket ARN, vector index ARN, vector index name, and vector field name in Bedrock’s knowledge base setup (as fields to fill when creating the knowledge base).\n - Metadata fields (e.g., AMZON_BEDROCK_TEXT, AMAZON_BEDROCK_METADATA) and the ability to attach non-filterable metadata.\n - Dimensions: embedding dimension must be between 1 and 4096; S3 Vectors only supports floating-point embeddings.\n - The knowledge base builder provides a sample of how to configure the vector index with fields for embeddings and metadata in a JSON-like mapping.\n- Limitations / notes:\n - S3 Vectors integration is noted as a non-trivial, scalable vector storage solution; it has limits on the vector dimension, supports only floating-point embeddings, and provides metadata handling and filtering.\n - Metadata limits and filtering behavior are described (e.g., 40 KB per vector metadata with 2 KB filterable portion, etc., in the general guidance). See the detailed guidance in the S3 vectors section.\n - The integration can be in preview or general depending on the time/region; the Bedrock doc references the S3 Vectors integration as a supported store.\n- Source:\n - Knowledge-base-setup.html (S3 Vectors section)\n - Knowledge-base-setup HTML content explicitly describes S3 Vectors, vector bucket/ARN, vector index, dimension limits, and metadata handling\n - Knowledge-base-setup URL: https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base-setup.html\\n\\n4\) Amazon Aurora (RDS) vector store\n- What it is: A Bedrock-supported vector store using Amazon Aurora (PostgreSQL-compatible) as the data store for vector embeddings.\n- How it integrates with Bedrock:\n - Bedrock expects a relational table to store embeddings, chunks (text), and metadata; you create a table with specific column names for embedding vectors and text chunks, plus a metadata column.\n - You need to map Bedrock’s UI/API fields to the table’s schema when creating the knowledge base.\n- Required formats and fields:\n - Columns including embedding (vector), chunks (text), and metadata (Bedrock-managed, plus optional custom metadata as needed).\n - You must create a DB index on the vector column and text column; optional GIN index on metadata if using custom metadata.\n- Limitations / notes:\n - The Aurora cluster must reside in the same AWS account as the Bedrock knowledge base.\n - The table schema is fixed in Bedrock’s guidance; you must provide those fields when creating the knowledge base, and they cannot be updated after creation.\n- Source:\n - Knowledge-base-setup.html (Aurora section)\n\n5) Neptune Analytics graphs (GraphRAG)\n- What it is: Neptune Analytics graphs used with GraphRAG (a Neptune-based vector-augmented approach) as a Bedrock knowledge base storage option.\n- How it integrates with Bedrock:\n - You create a Neptune Analytics graph and connect it to Bedrock Knowledge Bases; you configure a vector search index on the graph and map Bedrock’s embeddings/text/metadata to the graph’s fields.\n- Required formats and fields:\n - Graph ARN, vector index dimensions, and Bedrock text/metadata field mappings.\n- Limitations / notes:\n - The guidance describes how to set up the graph and the vector index, including dimensions matching the embeddings model.\n- Source:\n - Knowledge-base-setup.html (Neptune Analytics/GraphRAG section)\n\n6) Pinecone\n- What it is: Pinecone as a vector store option for Bedrock Knowledge Bases.\n- How it integrates with Bedrock:\n - You set up a Pinecone index, provide endpoint URL, and provide credentials (credentials secret ARN) to Bedrock via AWS Secrets Manager.\n- Required formats and fields:\n - Vector index name, endpoint URL, credentials secret ARN, optional KMS key for decrypting credentials.\n - Metadata handling: text field for raw chunk text, metadata field for source attribution metadata, optional text search index name, etc.\n - You must supply a secret in Secrets Manager with the API key for the Pinecone index (and secret ARN for Bedrock to use).\n- Limitations / notes:\n - Pinecone integration requires providing access credentials securely via Secrets Manager.\n - You’ll supply metadata/text field mappings for Bedrock to store and retrieve vectors and associated data.\n- Source:\n - Knowledge-base-setup.html (Pinecone section)\n\n7) Redis Enterprise Cloud\n- What it is: Redis Enterprise Cloud as a vector store option for Bedrock Knowledge Bases.\n- How it integrates with Bedrock:\n - You configure Redis connection settings via Bedrock (endpoint URL, vector index name, vector field, and metadata field). You must provide credentials via Secrets Manager and TLS settings as part of the integration.\n- Required formats and fields:\n - Endpoint URL, vector index name, vector field, metadata field, and Bedrock-managed metadata naming.\n - Secrets Manager secret with credentials (username, password, and TLS details, if applicable).\n- Limitations / notes:\n - TLS and secret configuration requirements are described; you must provide secret values in Secrets Manager for Bedrock to use.\n- Source:\n - Knowledge-base-setup.html (Redis Enterprise Cloud section)\n\n8) MongoDB Atlas\n- What it is: MongoDB Atlas as a vector store option for Bedrock Knowledge Bases.\n- How it integrates with Bedrock:\n - You connect to a MongoDB Atlas cluster, configure the database, collection, and vector index; Bedrock will store embeddings, text chunks, and metadata in the Atlas collection.\n- Required formats and fields:\n - Endpoint URL, database name, collection name, credentials secret ARN for Atlas user, vector index name, vector field name, text field name, metadata field name, optional text search/index fields.\n - Optional PrivateLink for AWS PrivateLink connectivity.\n- Limitations / notes:\n - Metadata filtering may require manual configuration in MongoDB Atlas vector index settings; some features require explicit configuration beyond the Bedrock setup.\n- Source:\n - Knowledge-base-setup.html (MongoDB Atlas section)\n\n9) Other notes on storage options (summary from knowledge base setup)\n- The Bedrock Knowledge Bases setup emphasizes you can connect to various data sources (unstructured or structured). It also notes:\n - You can choose to “set up your own supported vector store” or let Bedrock automatically create a vector store (e.g., via the Console for OpenSearch Serverless).\n - If you plan to use structured data stores, you can transform queries into structured data language queries (like SQL).\n- OpenAI-style “Kendra GenAI” integration option\n - Build a Bedrock knowledge base with an Amazon Kendra GenAI index to reuse indexed content and combine Bedrock GenAI with Kendra retrieval capabilities.\n - This integration is documented as a specific knowledge base-building path.\n- Source:\n - knowledge-base.html (Overview of knowledge bases)\n - knowledge-base-build-kendra-genai-index.html (Kendra GenAI integration)\n\nKendra GenAI integration (special case)\n- What it is: A Bedrock knowledge base built from an Amazon Kendra GenAI index to combine Bedrock’s GenAI capabilities with Kendra’s retrieval strengths.\n- How it integrates with Bedrock:\n - You create a Bedrock knowledge base from a Kendra GenAI index to reuse content across Bedrock applications.\n - It leverages Kendra’s indexing + Bedrock’s generation capabilities for improved accuracy and retrieval.\n- Source:\n - knowledge-base-build-kendra-genai-index.html\n\nRelated AWS documentation notes (context and confirmations)\n- OpenSearch-based vector storage integration status and availability:\n - AWS What’s New: Amazon Bedrock Knowledge Bases now supports Amazon OpenSearch Managed Cluster for vector storage (general availability note).\n - URL: https://aws.amazon.com/about-aws/whats-new/2025/03/amazon-bedrock-knowledge-bases-opensearch-cluster-vector-storage/\\n- MongoDB Atlas integration for Bedrock Knowledge Bases (vector storage):\n - AWS What’s New: Knowledge Bases for Amazon Bedrock now supports MongoDB Atlas for vector storage (vector storage availability).\n - URL: https://aws.amazon.com/about-aws/whats-new/2024/05/knowledge-bases-amazon-bedrock-mongodb-atlas-vector-storage/\\n- Retrieval Augmented Generation (RAG) guidance and integration options (AWS Prescriptive Guidance):\n - URL: https://docs.aws.amazon.com/prescriptive-guidance/latest/retrieval-augmented-generation-options/introduction.html\\n- General Bedrock knowledge base docs (for knowledge bases, setup, build, retrieval):\n - Knowledge Bases main overview: https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base.html\\n - Build a knowledge base by connecting to a data source: https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base-build.html\\n - Prerequisites for creating a Bedrock knowledge base with a unstructured data source: https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base-prereq.html\\n - Retrieving information from data sources using Bedrock Knowledge Bases: https://docs.aws.amazon.com/bedrock/latest/userguide/kb-how-retrieval.html\\n - Installing and using Kendra GenAI index with Bedrock Knowledge Bases: https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base-build-kendra-genai-index.html\\n - Prerequisites and setup for using a vector store (OpenSearch, S3 Vectors, etc.): https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base-setup.html\\n\\nSources (exact URLs used)\n- https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base.html\\n- https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base-build.html\\n- https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base-prereq.html\\n- https://docs.aws.amazon.com/bedrock/latest/userguide/kb-how-retrieval.html\\n- https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base-setup.html\\n- https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base-build-kendra-genai-index.html\\n- https://aws.amazon.com/about-aws/whats-new/2025/03/amazon-bedrock-knowledge-bases-opensearch-cluster-vector-storage/\\n- https://aws.amazon.com/about-aws/whats-new/2024/05/knowledge-bases-amazon-bedrock-mongodb-atlas-vector-storage/\\n- https://docs.aws.amazon.com/prescriptive-guidance/latest/retrieval-augmented-generation-options/introduction.html\\n\\nIf you’d like, I can format this into a compact quick-reference with a side-by-side comparison table of each storage option (store name, Bedrock integration points, required Bedrock fields, and notable limitations) and include only the most relevant bullets for quick decision-making.", "tool_calls": [], "additional_kwargs": {}, "response_metadata": {}, "invalid_tool_calls": []}
3	15925d170d1e4ce3956667dbb9b172ba	{"type": "human", "content": "You are research agent. Should provide accurate response based on documentation and statements, in the response add section with links to all sources that you have used.\nProvide official AWS Bedrock documentation on supported knowledge base storages. List all storage options Bedrock Knowledge Bases can use for Retrieval Augmented Generation (RAG), including vector stores or document stores, integrations (S3, Kendra GenAI, OpenSearch, etc.), required formats/fields, limitations, and exact URLs to AWS docs. Include separate sources for each option if possible.", "additional_kwargs": {}, "response_metadata": {}}
4	15925d170d1e4ce3956667dbb9b172ba	{"type": "ai", "content": "Official AWS Bedrock documentation: Supported knowledge base storages for Retrieval Augmented Generation (RAG)\n\nThe Bedrock Knowledge Bases support a variety of storage options (vector stores and document stores) that Bedrock can use for RAG. Below is a structured list of each option, what it is, how it integrates with Bedrock Knowledge Bases, the required formats/fields, notable limitations, and the exact AWS docs that describe it. When possible, each item links to separate Bedrock-specific documentation pages; in some cases the same Bedrock page covers multiple options.\n\n1) Amazon OpenSearch Serverless (vector store)\n- What it is\n - A vector search collection in Amazon OpenSearch Serverless used as the vector store for embedding vectors generated from your data.\n- How it integrates with Bedrock Knowledge Bases\n - In Bedrock Knowledge Bases, you map to an OpenSearch Serverless vector collection. Bedrock stores embeddings in a vector field and maps accompanying text chunks and Bedrock-managed metadata.\n- Required formats/fields (Bedrock setup guidance)\n - Create a vector index with:\n - A vector field (e.g., embeddings) using the faiss engine and an appropriate dimension (dimensions depend on your embedding model; Euclidean distance is recommended for floating-point embeddings).\n - Metadata fields to pair with vectors (e.g., text chunks and Bedrock metadata).\n - Mapping examples discuss:\n - Field for the vector embeddings\n - Field for the text chunks\n - Bedrock-managed metadata field\n- Limitations / notes (Bedrock doc context)\n - OpenSearch Serverless is one of the supported options for vector storage with explicit guidance on how to map Bedrock data into the index.\n - Requires configuring permissions and collection details in OpenSearch Serverless; Bedrock provides the mapping fields in the knowledge-base setup flow.\n- Bedrock doc source\n - Knowledge Base setup (OpenSearch Serverless section) \n - URL: https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base-setup.html\\n\\n2\) Amazon OpenSearch Service (Managed Clusters) as vector store\n- What it is\n - A managed OpenSearch cluster (OpenSearch Service domain) used as a vector store for Bedrock Knowledge Bases.\n- How it integrates with Bedrock Knowledge Bases\n - You create a vector index on the OpenSearch domain and connect it to the Bedrock knowledge base. Bedrock requires mappings for:\n - Domain ARN, domain endpoint\n - Vector index name, vector field, and metadata field\n- Required formats/fields (Bedrock setup guidance)\n - Mappings include:\n - vectorIndexName\n - vectorField (embedding field)\n - metadataField\n - Bedrock text chunk and Bedrock metadata fields\n- Limitations / notes\n - Prerequisites include required IAM permissions and domain configuration. Guidance covers encryption, indexing requirements, and domain capacity considerations.\n - Dimensionality and embedding-field configurations must align with your embedding model (including K-NN index considerations when supported).\n- Bedrock doc source\n - Knowledge Base setup (OpenSearch Clusters section)\n - URL: https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base-setup.html\\n\\n3\) Amazon S3 Vectors (vector store in S3)\n- What it is\n - Vector storage in Amazon S3 using S3 Vectors (Bedrock-supported) to hold embeddings and related metadata.\n- How it integrates with Bedrock Knowledge Bases\n - You configure an S3 vector bucket and a corresponding vector index. Bedrock stores embeddings and metadata in S3-based vector structures and uses a defined mapping for fields.\n- Required formats/fields\n - Vector bucket ARN, vector index ARN, vector index name, and vector field name\n - Metadata fields (Bedrock-managed) and text chunk fields\n - Embedding dimension constraints (1 to 4096); only floating-point embeddings are supported\n - Sample mappings show fields for:\n - embeddings field\n - text chunk field\n - metadata field\n- Limitations / notes\n - S3 Vectors integration is noted as a supported (and scalable) vector store, but described with several constraints:\n - Preview status (as of documentation) and ongoing availability notes\n - Dimension limits and floating-point embeddings only\n - Metadata handling and filtering limitations (e.g., 40 KB per vector metadata with 2 KB filterable portion)\n- Bedrock doc source\n - Knowledge Base setup (S3 Vectors section)\n - URL: https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base-setup.html\\n\\n4\) Amazon Aurora (RDS) vector store\n- What it is\n - A Bedrock-supported vector store using Amazon Aurora (PostgreSQL-compatible) as the data store for embeddings.\n- How it integrates with Bedrock Knowledge Bases\n - Bedrock expects a relational table with:\n - An embedding/vector column\n - A text chunks column\n - A metadata column (Bedrock-managed, plus optional custom metadata)\n - Bedrock maps its UI/API fields to the table schema during knowledge base creation.\n- Required formats/fields\n - Relational table with columns for:\n - embedding vector\n - text chunks\n - metadata\n - Optional metadata filtering/indexing (e.g., GIN index)\n- Limitations / notes\n - Aurora cluster must be in the same AWS account as the Bedrock knowledge base.\n - The table schema is fixed per Bedrock guidance and cannot be updated after creation.\n- Bedrock doc source\n - Knowledge Base setup (Aurora section)\n - URL: https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base-setup.html\\n\\n5\) Neptune Analytics graphs (GraphRAG)\n- What it is\n - Neptune Analytics graphs used with GraphRAG as a Bedrock knowledge base storage option.\n- How it integrates with Bedrock Knowledge Bases\n - Create a Neptune Analytics graph and connect it to Bedrock Knowledge Bases; configure a vector index on the graph and map Bedrock embeddings/text/metadata to the graph’s fields.\n- Required formats/fields\n - Graph ARN, vector index dimensions, and Bedrock text/metadata field mappings\n- Limitations / notes\n - Guidance covers graph/vector index setup and dimension matching to embedding models.\n- Bedrock doc source\n - Knowledge Base setup (Neptune Analytics/GraphRAG section)\n - URL: https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base-setup.html\\n\\n6\) Pinecone (vector store)\n- What it is\n - Pinecone as a dedicated vector index service to store and query embeddings for Bedrock Knowledge Bases.\n- How it integrates with Bedrock Knowledge Bases\n - Bedrock references a Pinecone index (name), endpoint URL, and credentials stored in AWS Secrets Manager (secret ARN; optional KMS key for decryption).\n- Required formats/fields\n - Vector index name\n - Endpoint URL\n - Secrets Manager credentials secret ARN (and optional KMS decryption key)\n - Metadata/text fields to store the raw chunk text and source metadata\n - Optional text-search index name\n- Limitations / notes\n - Credentials must be provided securely via Secrets Manager\n - Metadata/text field mappings must be supplied to Bedrock for storage and retrieval\n- Bedrock doc source\n - Knowledge Base setup (Pinecone section)\n - URL: https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base-setup.html\\n\\n7\) Redis Enterprise Cloud (vector store)\n- What it is\n - Redis Enterprise Cloud as a vector store option for Bedrock Knowledge Bases.\n- How it integrates with Bedrock Knowledge Bases\n - Bedrock connects to Redis, providing endpoint URL, vector index name, vector field, and metadata field; credentials via Secrets Manager and TLS settings are configured as part of the integration.\n- Required formats/fields\n - Endpoint URL\n - Vector index name\n - Vector field\n - Metadata field\n - Bedrock-managed metadata naming\n - Secrets Manager secret with credentials (and TLS details if applicable)\n- Limitations / notes\n - TLS and secret configuration requirements; secrets must be provided to Bedrock\n- Bedrock doc source\n - Knowledge Base setup (Redis Enterprise Cloud section)\n - URL: https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base-setup.html\\n\\n8\) MongoDB Atlas (vector store)\n- What it is\n - MongoDB Atlas as a vector store option for Bedrock Knowledge Bases.\n- How it integrates with Bedrock Knowledge Bases\n - Connect to a MongoDB Atlas cluster, configure database/collection and vector index; Bedrock stores embeddings, text chunks, and metadata in the Atlas collection.\n- Required formats/fields\n - Endpoint URL\n - Database name\n - Collection name\n - Credentials secret ARN for Atlas user\n - Vector index name\n - Vector field name\n - Text field name\n - Metadata field name\n - Optional PrivateLink connectivity\n- Limitations / notes\n - Metadata filtering may require additional Atlas vector index configuration; some features may require explicit setup beyond Bedrock’s default guidance\n- Bedrock doc source\n - Knowledge Base setup (MongoDB Atlas section)\n - URL: https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base-setup.html\\n\\n9\) Kendra GenAI index (special integration path for Bedrock)\n- What it is\n - Build a Bedrock knowledge base from an Amazon Kendra GenAI index to combine Bedrock GenAI with Kendra’s retrieval capabilities.\n- How it integrates with Bedrock Knowledge Bases\n - The Bedrock knowledge base reuses content indexed in a Kendra GenAI index; benefits include reusing indexed content and leveraging Bedrock’s generation features with Kendra’s retrieval strength.\n- Required formats/fields\n - You build a Bedrock KB from a Kendra GenAI index; the process is documented in Bedrock’s KB build flow for Kendra GenAI\n- Limitations / notes\n - This is a specialized integration path that leverages Kendra GenAI indexing for Bedrock KB creation\n- Bedrock doc source\n - Knowledge Base build path for Kendra GenAI index\n - URL: https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base-build-kendra-genai-index.html\\n\\nNotes on broader context (RAG and Bedrock knowledge bases)\n- RAG in Bedrock relies on combining retrieved source content with generation. Bedrock provides operations such as Retrieve, RetrieveAndGenerate, and GenerateQuery to support the RAG workflow; see the Bedrock knowledge base retrieval overview for context.\n - Bedrock retrieval and RAG overview: https://docs.aws.amazon.com/bedrock/latest/userguide/kb-how-retrieval.html\\n - General knowledge base concepts (overview, building, prerequisites, etc.): https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base.html\\n - Prerequisites for unstructured data sources and vector stores: https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base-prereq.html\\n - Build or connect a knowledge base (data source connections): https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base-build.html\\n - Build a knowledge base with Kendra GenAI index: https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base-build-kendra-genai-index.html\\n\\nSources (exact URLs used for the above options)\n- Knowledge Base setup (general, includes multiple storage options)\n - https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base-setup.html\\n- Knowledge Base build: Kendra GenAI index path\n - https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base-build-kendra-genai-index.html\\n- Knowledge Bases overview and retrieval capabilities\n - https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base.html\\n - https://docs.aws.amazon.com/bedrock/latest/userguide/kb-how-retrieval.html\\n- Prerequisites for a Bedrock knowledge base with unstructured data (vector store setup)\n - https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base-prereq.html\\n- Build a knowledge base with a Kendra GenAI index (detailed steps)\n - https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base-build-kendra-genai-index.html\\n\\nIf you’d like a quick-reference comparison, I can format this into a compact side-by-side table listing: storage option, Bedrock integration point, required Bedrock fields, known limitations, and the exact source URL for each. All information above is drawn directly from the official Bedrock documentation pages cited.", "tool_calls": [], "additional_kwargs": {}, "response_metadata": {}, "invalid_tool_calls": []}

Checking consumption of tokens and switching between different models

Another practical concern is token usage.

Using a large model for every task is wasteful.
Instead, agents can switch between models dynamically:
Use Nano GPT-5.1 (cheap, fast) for small lookups.
Use AWS Bedrock Claude 3.5 Sonnet (accurate, large context) for reasoning-heavy prompts.

This hybrid approach optimizes both cost and performance.

The flow context data is fully customizable

The extracted data and accumulated response with LLM can be routed to any other flow, custom node chat output, etc.
Here, for any prompt besides providing the response to chat we are creating rtf document and persist it on a file system
for further review and usage.

Conclusions

What we’ve built is a full end-to-end multi-agent ecosystem—all without writing a single line of code. By leveraging n8n’s no-code orchestration, the system allows:

Natural interaction: Agents collaborate dynamically based on user prompts, without predefined rigid flows.
Dedicated integrations: Each agent can connect to specialized MCP servers, custom tools, or even other agents, extending its knowledge far beyond the base LLM.
Persistent memory: All interactions and context are stored in memory, so agents can build on previous sessions instead of starting from scratch.
Transparency and control: With UI-based execution dumps, we can inspect how decisions were made, track history, and debug workflows visually.

This setup proves that multi-agent systems don’t have to be locked away in research papers—they can be practical, maintainable, and production-ready, combining Bedrock models, lightweight GPTs, and n8n’s no-code tools into a flexible AI ecosystem that feels less like a chatbot and more like a team of digital experts.

AWS OpenSearch Full Documents Reindexing: When? Why? How?

Roman Tsypuk — Mon, 22 Sep 2025 06:02:00 +0000

We will explore AWS OpenSearch Service, how it was introduced and deep dive into challenge of documents reindexing.

OpenSearch Brief History: The Forking of Elasticsearch

Elasticsearch is an open-source search engine developed by Elastic NV. It became incredibly popular due to its scalability, distributed nature, and powerful search capabilities. However, in 2021, Elastic NV changed the licensing model of Elasticsearch from Apache 2.0 to a Server Side Public License (SSPL).
This move was made to prevent cloud providers from offering Elasticsearch as a managed service without contributing back to the open-source community.

In response, AWS decided to fork the last Apache 2.0-licensed version of Elasticsearch and create a new service called Amazon OpenSearch Service. This fork not only preserved the open-source nature of the software but also allowed AWS to continue offering a managed search service with full control over its development.

Licensing

After the license change, Elasticsearch is now under the SSPL, which is not recognized as an open-source license by the Open Source Initiative (OSI). The SSPL imposes restrictions on how the software can be used, particularly for cloud services.

ES provides Elastic Cloud installation that can be installed into any public cloud provider resources or on-prem.

AWS OpenSearch: OpenSearch remains under the Apache 2.0 license, which is fully open-source. This means anyone can use, modify, and distribute the software without restrictions, making it more attractive for users who prefer open-source solutions.

Understanding AWS Elasticsearch: Internal Implementation of Indices and Reindexing

Amazon Elasticsearch Service (Amazon ES), now known as Amazon OpenSearch Service, is a managed service that makes it easy to deploy, operate, and scale Elasticsearch clusters in the AWS Cloud. It’s
widely used for real-time application monitoring, log analytics, full-text search, and more. One of the core components of Elasticsearch is its indices, which serve as the primary structure for
storing and querying data. This article delves into the internal implementation of indices in AWS Elasticsearch and provides a guide on how to reindex a database within this environment.

What is an Index in Elasticsearch?

An index in Elasticsearch is akin to a database in traditional relational databases. It contains a collection of documents that are stored and managed together. Each document is a JSON object, and
each field within the document is a data point that can be searched and analyzed.

Key Components of an Elasticsearch Index:

Shards: An index is divided into smaller pieces called shards. Each shard is a self-contained, fully functional instance of Lucene, the underlying search engine for Elasticsearch.
Replicas: For high availability and fault tolerance, Elasticsearch allows you to create replica shards. These are copies of the primary shards that can serve search requests in case the primary shard fails.
Mappings: Mappings define the structure of the documents within an index, including the data types of fields and how they should be indexed and stored.
Internal Implementation of Indices in AWS Elasticsearch

AWS Elasticsearch handles indices similarly to a standard Elasticsearch deployment but with additional layers of management, security, and scaling capabilities provided by AWS.

Key Aspects of AWS Elasticsearch Indices:

Managed Clusters: AWS Elasticsearch manages the underlying infrastructure, including node provisioning, shard allocation, and index replication.
Scaling: Indices can be scaled horizontally by adjusting the number of shards and vertically by increasing the instance size or adding more nodes.
Security: AWS integrates Elasticsearch with other AWS services like AWS IAM, AWS KMS for encryption, and VPC for network isolation, ensuring that your indices are secure and accessible only to authorized users.
Snapshots: AWS Elasticsearch provides automated snapshots for indices, which are stored in Amazon S3 and can be used for backup and recovery purposes.

Why Reindexing is Important

Reindexing in Elasticsearch is the process of copying the data from one index to another. This is often necessary when you need to:

Change the structure of the index, such as altering mappings (changing types).
Improve performance by reconfiguring the number of shards or replicas (also merge/split indecies).
Upgrade Elasticsearch versions that might require data format changes.
Introduce new fields in documents and make them available for search
Reindexing can be a resource-intensive operation, and AWS Elasticsearch provides tools and best practices to ensure that it’s done efficiently without disrupting service availability.

Steps to Reindex in AWS Elasticsearch. Here’s a step-by-step guide to reindexing an index in AWS Elasticsearch:

Step 1: Create the Target Index

Before you start reindexing, you need to create the target index with the desired mappings, settings, and shard configuration.

PUT /new-index
{
  "settings": {
    "number_of_shards": 3,
    "number_of_replicas": 1
  },
  "mappings": {
    "properties": {
      "field1": {
        "type": "text"
      },
      "field2": {
        "type": "date"
      }
    }
  }
}

Step 2: Use the Reindex API

Elasticsearch provides a _reindex API that allows you to copy data from the source index to the target index.

POST /_reindex
{
  "source": {
    "index": "old-index"
  },
  "dest": {
    "index": "new-index"
  }
}

Step 3: Monitor the Reindexing Process

Reindexing can take time depending on the size of your data. You can monitor the progress using the Task API.

GET /_tasks?detailed=true&actions=*reindex

Step 4: Update Aliases (Optional)

Once reindexing is complete, you may want to switch an alias to point to the new index.

POST /_aliases
{
  "actions": [
    {
      "remove": {
        "index": "old-index",
        "alias": "my-alias"
      }
    },
    {
      "add": {
        "index": "new-index",
        "alias": "my-alias"
      }
    }
  ]
}

Step 5: Delete the Old Index (Optional)

After verifying that the new index is functioning correctly, you can delete the old index to free up resources.

DELETE /old-index

Also, during indexing it is possible to update structure of documents (by adding script section):

POST _reindex
{
   "source":{
      "index":"source"
   },
   "dest":{
      "index":"destination"
   },
   "script":{
      "lang":"painless",
      "source":"ctx._account.number++"
   }
}

No indices' migration technic (acutal for smaller clusters)

If the Data volume is not high, once index template is updated, all new indices will be created from new template (with new fields available for search).
For all existing indicies we need update index mapping (this will allow any update/adding documents to these indices to be in the latest schema).

For older documents we can run scripted update with defined predicate to update needed documents with new fields default values:

Step1: Patch search template

PUT _template/{index}
{
  ...full_body with new mappings
}

Step2: Add to each index new mapping

PUT /{
  index
}/_mappings
{
  "properties": {
    "new_field": {
      "null_value": false,
      "type": "boolean"

  }
}

Step3: Patch all documents with scripted update

POST /{index}/_update_by_query
{
  "query": {
    "bool": {
      "must_not": {
        "exists": {
          "field": "hidden"
        }
      }
    }
  },
  "script": {
    "source": "ctx._source.new_field = false",
    "lang": "painless"
  }
}

Best Practices for Reindexing in AWS Elasticsearch

Snapshot Before Reindexing: Always take a snapshot of your data before starting the reindexing process to safeguard against data loss.
Monitor Cluster Health: Keep an eye on the cluster’s health during reindexing to avoid overwhelming the system.
Use Aliases: Aliases can help minimize downtime by allowing you to switch indices without changing your application code.
Test in Staging: Before reindexing in production, test the process in a staging environment to catch any potential issues.

Conclusion

Reindexing in AWS Elasticsearch is a crucial operation for maintaining and optimizing your search infrastructure. By understanding the internal implementation of indices and following best practices
for reindexing, you can ensure that your Elasticsearch environment remains robust, scalable, and ready to meet the demands of your applications. AWS Elasticsearch, with its managed capabilities and
tight integration with other AWS services, provides a powerful platform for deploying and managing your search workloads.

References (Links)

Amazon OpenSearch Indices migration

AWS-nuke controlled resources cleanup in multiple aws accounts

Roman Tsypuk — Mon, 22 Sep 2025 06:01:29 +0000

AWS-Nuke is an open-source tool, designed to automate the removal of AWS resources from an account. It can be used to quickly and safely delete all resources in a given AWS account, or it can be customized to delete specific types of resources based on user-defined rules.

What is AWS-Nuke?

Whether you’re spinning up test environments or decommissioning an old project, AWS-Nuke is particularly useful for keeping AWS accounts clean, ensuring that old, unused resources do not pile up, leading to cost overruns, security risks, and unnecessary complexity.

AWS-Nuke is a command-line tool that supports multiple AWS services and can be run from any machine that has access to AWS credentials.

Why Use AWS-Nuke?

Before diving into the technical details of AWS-Nuke, it’s important to understand why this tool is useful.

Cost Management: One of the biggest challenges in cloud environments is managing costs. AWS charges based on the resources you use, and sometimes resources that are no longer in use are forgotten and left running. By using AWS-Nuke, you can regularly clean up old resources, ensuring that you only pay for what you actively use.
Environment Cleanup: In development and test environments, resources are often created, modified, and left without proper deletion. AWS-Nuke can help by regularly purging these environments of unnecessary resources, making it easier to manage infrastructure.
Security: Resources left running for extended periods can pose security risks. For instance, unused EC2 instances, security groups, or S3 buckets might expose your environment to attacks. AWS-Nuke helps minimize such risks by cleaning up unused resources.
Automation: AWS-Nuke offers automation capabilities, making it easier to integrate into continuous integration (CI) pipelines or scheduled jobs for periodic account cleanups.
Simplification: With hundreds of AWS services and resources to manage, AWS-Nuke simplifies the task of cleaning up resources by providing a centralized and automated way to perform deletions.

How AWS-Nuke Works

AWS-Nuke works by scanning all the resources in your AWS account and then deleting them based on a configuration file. You can configure AWS-Nuke to delete specific types of resources or to leave certain critical resources untouched.

AWS-Nuke uses the AWS SDK to list and delete resources. It can handle a wide range of AWS services, including but not limited to:

EC2 Instances
S3 Buckets
RDS Instances
IAM Users and Roles
VPCs
Elastic Load Balancers
CloudWatch Alarms
CloudFormation Stacks
ECR Repositories

Configuration

AWS-Nuke uses a YAML configuration file to specify what resources should or should not be deleted. This configuration file is highly customizable, allowing users to target specific AWS resources while leaving others intact.

Here’s a simple example of an AWS-Nuke configuration:

yaml
Copy code
regions:

"us-east-1"
"us-west-2"

resource-types:
excludes:

EC2::KeyPair
IAM::Role
IAM::User This configuration specifies that AWS-Nuke should only run in the us-east-1 and us-west-2 regions, and it will exclude deleting EC2 Key Pairs, IAM Roles, and IAM Users.

Dry-Run Mode
One of AWS-Nuke's most useful features is the dry-run mode, which allows users to see what resources will be deleted without actually deleting them. This is useful for reviewing the results of a cleanup before making any irreversible changes.

To run AWS-Nuke in dry-run mode, simply use the --dry-run flag:

aws-nuke --config config.yml --dry-run

This command will output a list of resources that AWS-Nuke will delete based on the configuration file, without actually performing the deletions.

Running AWS-Nuke
Once you’re confident in your configuration, running AWS-Nuke is simple. First, ensure you have the correct AWS credentials for the account you want to clean up. Then, use the following command:

aws-nuke --config config.yml

This command will delete all resources in the account, except for the ones excluded in the configuration.

Resource Dependencies
AWS-Nuke also handles resource dependencies. For example, it will not delete an S3 bucket until all objects within the bucket have been deleted. This ensures that AWS resources are deleted in the correct order, preventing errors and failed deletions.

brew install aws-nuke

After checking version installed from brew repo is outdated:

 aws-nuke -c config/nuke-config.yml --profile technology
Incorrect Usage: flag provided but not defined: -c

NAME:
   aws-nuke - remove everything from an aws account

USAGE:
   aws-nuke [global options] command [command options]

VERSION:
   3.22.0

AUTHOR:
   Erik Kristensen <erik@erikkristensen.com>

FATA[0000] flag provided but not defined: -c

So I will use the latest version installed from sources:

Contributing to aws-nuke

When running locally aws-nuke with aws profile I got error.

aws_nuke_v2 -c config/nuke-config.yaml --profile personal_profile
Error: failed get caller identity: NoCredentialProviders: no valid providers in chain. Deprecated.
For verbose messaging see aws.Config.CredentialsChainVerboseErrors

Process finished with the exit code 255

And after checking the sources, I find out that profile is present in parameters for aws-nuke, but not implemented.
So I have added this feature to aws-nuke:

if params.Profile != "" {
            cfg, err := awsSDKConfig.LoadDefaultConfig(context.TODO(), awsSDKConfig.WithSharedConfigProfile(params.Profile))
            if err != nil {
                log.Fatalf("unable to load SDK config, %v", err)
            }
            log.Infof("using profile %v", params.Profile)
            /// Extract credentials from the configuration
            cred, err := cfg.Credentials.Retrieve(context.TODO())
            if err != nil {
                log.Fatalf("unable to retrieve credentials, %v", err)
            }
            creds.AccessKeyID = cred.AccessKeyID
            creds.SecretAccessKey = cred.SecretAccessKey
            defaultRegion = cfg.Region
        }

        if !creds.HasKeys() && !creds.HasProfile() {
            creds.AccessKeyID = os.Getenv("AWS_ACCESS_KEY_ID")
            creds.SecretAccessKey = os.Getenv("AWS_SECRET_ACCESS_KEY")
        }

Also added Contribution PR to aws-nuke: https://github.com/rebuy-de/aws-nuke/pull/1263/files

Now aws-nuke correctly process --profile cli argument and is ready to run

Working with implemented profile feature:

go run main.go -c config/nuke-config.yaml --profile personal_account

INFO[0000] using profile personal_account
aws-nuke version unknown - unknown - unknown

Do you really want to nuke the account with the ID 123456789012 and the alias 'personal_account'?
Do you want to continue? Enter account alias to continue.
> personal_account

Tool will query existing resources, match them with filters that we have configured in exclusion and will do dry run.

Resources that we want to leave in account

Let's say we do not want to affect IAM users/permissions/policies, also MFA devices and some route53 records.
These all resources should be declared in aws-nuke config, so when running it will not touch them.

regions:
  - eu-east-1
  - global

accounts:
123456789012:
filters:
PolicyName:
  - property: "UserName"
  - value: "aws-cli"
IAMRole:
  - "OrganizationAccountAccessRole"
IAMRolePolicyAttachment:
  - "OrganizationAccountAccessRole"
  - property: arn
  - value: "arn:aws:iam::aws:policy/AdministratorAccess"
IAMUser:
  - "admin"
  - "aws-cli"
IAMUserAccessKey:
  - property: UserName
value: "aws-cli"
IAMUserPolicyAttachment:
  - property: "arn"
  - value: "arn:aws:iam::aws:policy/AdministratorAccess"
IAMVirtualMFADevice:
  - property: arn
  - value: "arn:aws:iam::123456789012:mfa/to.root.iphone"
Route53HostedZone:
  - property: "Name"
  - value: "xxxxx.xx."

account-blocklist:
  - 1234567890

resource-types:
excludes:
  - Route53ResourceRecordSet
  - Route53HostedZone

Review resources in output

...
global - IAMPolicy - arn:aws:iam::123456789012:policy/terraform-20230725102234859300000005 - [ARN: "arn:aws:iam::123456789012:policy/terraform-20230725102234859300000005", Name: "terraform-20230725102234859300000005", Path: "/", PolicyID: "ANPAVFCI6B4SXG62NJLCB"] - would remove

global - IAMRole - AmazonGrafanaServiceRole-Skz3Lbeo2 - [CreateDate: "2023-03-21T16:02:03Z", LastUsedDate: "2023-03-21T16:02:03Z", Name: "AmazonGrafanaServiceRole-Skz3Lbeo2", Path: "/service-role/"] - would remove

123456789012", PolicyName: "terraform-20230725102234859300000005", RoleCreateDate: "2023-07-25T10:22:27Z", RoleLastUsed: "2023-07-25T10:22:27Z", RoleName: "terraform-20230725102226616700000002", RolePath: "/"] - filtered by config
global - IAMRolePolicy - terraform-20230725102226616700000003 -> terraform-20230725102227744300000004 - [PolicyName: "terraform-20230725102227744300000004", role:CreateDate: "2023-07-25T10:22:27Z", role:LastUsed: "2023-07-25T10:22:56Z", role:Path: "/", role:RoleID: "AROAVFCI6B4SWQNDRGVXF", role:RoleName: "terraform-20230725102226616700000003"] - would remove
Scan complete: 44 total, 13 nukeable, 31 filtered.

The above resources would be deleted with the supplied configuration. Provide --no-dry-run to actually destroy resources.

Perform cleanup

go run main.go -c config/nuke-config.yaml --profile personal_account --no-dry-run

INFO[0000] using profile personal_account
aws-nuke version unknown - unknown - unknown

Do you really want to nuke the account with the ID 123456789012 and the alias 'personal_account'?
Do you want to continue? Enter account alias to continue.
> personal_account

First aws-nuke will scan as in dry-run, and also propose to confirm deletion.

Tool has issues deleting IAMs - it has logic to retry but does not understand resources dependencies. For instance, it can not delete role with attached policies that it also should delete:

ERRO[0373] DeleteConflict: Cannot delete entity, must detach all policies first.
status code: 409, request id: 9506e3fd-5945-48b3-9031-826d28d2abec

so you have manually delete from CLI or aws console

Integration with CI/CD

One of the best use cases for AWS-Nuke is integrating it into a CI/CD pipeline. You can use AWS-Nuke as part of your continuous integration (CI) process to clean up test environments after running tests. This ensures that no unnecessary resources remain after testing is complete, keeping your environment clean and your costs under control.

Here’s an example of how you might integrate AWS-Nuke into a CI/CD pipeline:

Run AWS-Nuke in dry-run mode before deploying a new version of your application to ensure there are no unnecessary resources left over from previous tests.
After the deployment and testing are complete, run AWS-Nuke in full mode to clean up all resources.

Conclusion

AWS-Nuke is a powerful and versatile tool for cleaning up AWS environments. It helps prevent cost overruns, reduces security risks, and simplifies the management of AWS resources. Whether you’re managing a development environment, performing testing, or decommissioning an old project, AWS-Nuke provides an efficient way to automate the cleanup process.

However, AWS-Nuke also requires careful configuration to avoid deleting critical resources. Always use the dry-run mode to review deletions before making any changes, and ensure that your configuration file accurately reflects your environment.

In an age where cloud resources can grow exponentially, tools like AWS-Nuke are indispensable for keeping your infrastructure lean and efficient.

AWS CSM Mode: Advanced monitoring of AWS client

Roman Tsypuk — Mon, 22 Sep 2025 06:01:02 +0000

After exploring aws boto3 core sources on github, I found this interesting commit, that enables monitor mode called CSM.
Once exploring how it works I decided write a post, cause this feature is extremely useful but also start writing aws-client-monitor toolbox on top of it.

Introduction to AWS CSM (Client-Side Monitoring) Mode

AWS Client-Side Monitoring (CSM) is a powerful feature designed to track and analyze the performance of your AWS SDK calls. When enabled, it provides detailed metrics on API requests, response times,
and error rates, helping developers gain a deeper understanding of their application's behavior when interacting with AWS services. This is crucial for debugging, optimizing performance, and ensuring
that applications are running efficiently in production environments.

CSM mode works by capturing information about SDK API calls and sending that data to a local monitoring agent. It helps you:

Track API request latencies.
Identify high failure rates in SDK requests.
Gain visibility into the most frequently called AWS services.

In this blog post, we'll explore AWS CSM mode in more detail, look at common use cases, and provide Golang code snippets to demonstrate how to implement it.

Why Use AWS CSM Mode?

As applications become increasingly reliant on cloud services, monitoring and optimizing the performance of these interactions becomes critical. AWS SDKs are widely used to interface with AWS services
such as S3, DynamoDB, Lambda, and many others. However, managing and tracking these interactions can be challenging, especially when it comes to identifying latency issues or bottlenecks in the
communication between your application and AWS.

CSM provides a granular view of how SDK requests are performing. It allows you to gather metrics like:

Latency: How long each request takes.
Errors: Which AWS services are returning errors and why.
Request Frequency: Which services are being called the most.

This data is invaluable for performance tuning, debugging, and capacity planning.

Common Use Cases for AWS CSM Mode

Performance Optimization:

By tracking the latency of AWS service calls, you can identify the API requests that are taking the longest to execute. This can help you optimize the application's performance, whether through
caching, retries, or parallelizing requests.

Error Tracking:

If your application experiences frequent errors while interacting with AWS services, CSM can help identify the root cause. For example, if a specific AWS service is returning a large number of 5xx
errors, CSM will capture this information, enabling developers to troubleshoot quickly.

Capacity Planning:

Monitoring the number of requests made to AWS services can help forecast capacity needs and adjust resources accordingly. For example, if you're making a large number of requests to DynamoDB, it may
be time to scale your read/write capacity.

Debugging Production Issues:

When something goes wrong in production, CSM can provide critical insights into which AWS services or API calls are causing issues, allowing for quick resolution.

Setting Up AWS CSM in Golang

The AWS SDK for Go provides native support for client-side monitoring. To enable CSM, you need to configure the SDK to send data to the local CSM agent, which processes and forwards it to monitoring
tools like Amazon CloudWatch.

Step 1: Install the AWS SDK for Go

First, you need to install the AWS SDK for Go, if you haven't already:

go get -u github.com/aws/aws-sdk-go

Step 2: Enable CSM in the AWS SDK

To enable CSM in the AWS SDK for Go, you need to configure the environment variables that control CSM behavior, or you can do this programmatically within your application.

Here’s an example of how you can enable CSM using environment variables:

export AWS_CSM_ENABLED=true
export AWS_CSM_HOST=127.0.0.1
export AWS_CSM_PORT=31000
export AWS_CSM_CLIENT_ID=my-client-id

AWS_CSM_ENABLED: Enables or disables CSM.
AWS_CSM_HOST: The hostname where the CSM agent is running (usually localhost).
AWS_CSM_PORT: The port where the CSM agent is listening.
AWS_CSM_CLIENT_ID: A client identifier used to differentiate between clients.

Step 3: Configure AWS SDK to Use CSM

In Golang, you configure the SDK to use CSM when making AWS API calls. Here’s a simple example where we create an S3 client and make a request with CSM enabled.

package main

import (
  "fmt"
  "net"
  "time"
)

func listenUDP(port int, ch chan<- []byte) {
  addr := net.UDPAddr{
    Port: port,
    IP:   net.ParseIP("0.0.0.0"),
  }

  conn, err := net.ListenUDP("udp", &addr)
  if err != nil {
    fmt.Println("Error listening on UDP:", err)
    return
  }
  defer func(conn *net.UDPConn) {
    err := conn.Close()
    if err != nil {
      print("Error closing UDP connection:", err)
    }
  }(conn)

  buffer := make([]byte, 2048)
  for {
    n, _, err := conn.ReadFromUDP(buffer)
    if err != nil {
      fmt.Println("Error reading from UDP:", err)
      continue
    }
    // Send received data to channel
    ch <- buffer[:n]
  }
}

func writeToConsole(ch <-chan []byte) {
  for msg := range ch {
    fmt.Println("Received from channel:", string(msg))
  }
}

func main() {
  byteChannel := make(chan []byte)

  // Goroutine to listen on UDP and write to the channel
  go listenUDP(31000, byteChannel)

  // Goroutines to read from the channel
  go writeToConsole(byteChannel)

  // Prevent the main function from exiting
  for {
    time.Sleep(1 * time.Second)
  }
}

Step 4: Invoke aws cli API

aws s3 ls

Step5: aws-client-monitor will display calls

aws-cli will as usual list the buckets but at the same time it will make 2 calls via UDP to our server.

For each AWS API call there 2 entities ApiCallAttempt and ApiCall, they have the following structure:

{
  "Version": 1,
  "ClientId": "my-client-id",
  "Type": "ApiCallAttempt",
  "Service": "S3",
  "Api": "ListBuckets",
  "Timestamp": 1728194484982,
  "AttemptLatency": 266,
  "Fqdn": "s3.eu-west-1.amazonaws.com",
  "UserAgent": "aws-cli/1.27.92 md/Botocore#1.31.2 ua/2.0 os/macos#21.6.0 md/arch#x86_64 lang/python#3.10.14 md/pyimpl#CPython cfg/retry-mode#legacy botocore/1.31.2",
  "AccessKey": "ASIAWFOD4FPxxx",
  "Region": "eu-west-1",
  "SessionToken": "IQoJb3JpZxxx=",
  "HttpStatusCode": 200,
  "XAmzRequestId": "8K3P9AWACxxx",
  "XAmzId2": "vZDGgBpIwz6Jfxxx="
}

{
  "Version": 1,
  "ClientId": "my-client-id",
  "Type": "ApiCall",
  "Service": "S3",
  "Api": "ListBuckets",
  "Timestamp": 1728194484981,
  "AttemptCount": 1,
  "Region": "eu-west-1",
  "UserAgent": "aws-cli/1.27.92 md/Botocore#1.31.2 ua/2.0 os/macos#21.6.0 md/arch#x86_64 lang/python#3.10.14 md/pyimpl#CPython cfg/retry-mode#legacy botocore/1.31.2",
  "FinalHttpStatusCode": 200,
  "Latency": 267,
  "MaxRetriesExceeded": 0
}

Tracking errors of aws API

We can also track error of AWS API, let's try to create bucket that exists:

aws s3api create-bucket --bucket existing-bucket

An error occurred (IllegalLocationConstraintException) when calling the CreateBucket operation: The unspecified location constraint is incompatible for the region specific endpoint this request was sent to.

Same error message also present in aws-client-monitor:

{
  "Version": 1,
  "ClientId": "my-client-id",
  "Type": "ApiCallAttempt",
  "Service": "S3",
  "Api": "CreateBucket",
  "Timestamp": 1728195105316,
  "AttemptLatency": 215,
  "Fqdn": "existing-bucket.s3.eu-west-1.amazonaws.com",
  "UserAgent": "aws-cli/1.27.92 md/Botocore#1.31.2 ua/2.0 os/macos#21.6.0 md/arch#x86_64 lang/python#3.10.14 md/pyimpl#CPython cfg/retry-mode#legacy botocore/1.31.2",
  "AccessKey": "ASIAWFODxxx",
  "Region": "eu-west-1",
  "SessionToken": "IQoJxxx=",
  "HttpStatusCode": 400,
  "XAmzRequestId": "FPWTJWZC7114XQJE",
  "XAmzId2": "+G9yOxxx",
  "AwsException": "IllegalLocationConstraintException",
  "AwsExceptionMessage": "The unspecified location constraint is incompatible for the region specific endpoint this request was sent to."
}

{
  "Version": 1,
  "ClientId": "my-client-id",
  "Type": "ApiCall",
  "Service": "S3",
  "Api": "CreateBucket",
  "Timestamp": 1728195105312,
  "AttemptCount": 1,
  "Region": "eu-west-1",
  "UserAgent": "aws-cli/1.27.92 md/Botocore#1.31.2 ua/2.0 os/macos#21.6.0 md/arch#x86_64 lang/python#3.10.14 md/pyimpl#CPython cfg/retry-mode#legacy botocore/1.31.2",
  "FinalHttpStatusCode": 400,
  "FinalAwsException": "IllegalLocationConstraintException",
  "FinalAwsExceptionMessage": "The unspecified location constraint is incompatible for the region specific endpoint this request was sent to.",
  "Latency": 219,
  "MaxRetriesExceeded": 0
}

Metrics Extraction

Following metrics can be extracted on UDP server side:

API call duration (latency)
History of app operations with AWS API (service name : operation)
Success and failure rates
Error messages
Region
User-agent
IP
clientName
UserAgent
AccessKey
Request payload size.

We can use these insights to fine-tune our application's performance and catch potential bottlenecks or errors.

Conclusion

AWS Client-Side Monitoring (CSM) is a valuable tool for gaining deep insights into how your application interacts with AWS services. It helps you monitor performance, identify errors, and optimize API
usage. By integrating CSM with the AWS SDK for Go, developers can track API calls and gather real-time metrics, ultimately improving application performance and reliability.

If you're working in a production environment where optimizing performance and troubleshooting issues is crucial, CSM is a feature worth enabling. With just a few configuration steps, you can gain
access to a wealth of data that will help you fine-tune your AWS-based applications.

CDC in AWS: Content Data Capture from AWS RDS MySQL into AWS MSK Kafka topic using Debezium

Roman Tsypuk — Mon, 22 Sep 2025 06:00:29 +0000

Kafka connect is a powerful open-source platform for Change Data Capture (CDC), enabling real-time event streaming from databases like MySQL.
In this post, we'll explore how to set up one of the most popular Open Source Debezium to capture changes from an AWS MySQL RDS instance and publish them to a Kafka topic.
We'll also dive into the Debezium message format and explain the difference between standalone and cluster modes.

What problem does it solve

Debezium allows you to:

Stream real-time changes from a MySQL RDS instance.
Track inserts, updates, and deletes as they occur.
Publish these changes to Kafka topics for downstream processing.

This is particularly useful for building event-driven architectures, data pipelines, and synchronizing databases with other systems.

Besides Debezium there are multiple opensource connectors available on confluent platform to provide intergration point with different sink and source systems like AWS S3, ElasticSearch
**, **MongoDB, etc

Setting Up Debezium for MySQL RDS

1. Enable Binary Logging on MySQL RDS:

Log in to your RDS instance and ensure binary logging is enabled in your parameter group.
Configure the following parameters:

binlog_format = RAW
binlog_row_image = FULL
Ensure binlog_retention_period is set to a sufficient duration for your use case.

SHOW
VARIABLES LIKE 'binlog_format';

Variable_name	Value
binlog_format	RAW

SHOW VARIABLES LIKE 'log_bin%';
SHOW VARIABLES LIKE 'binlog_format';
SHOW VARIABLES LIKE 'binlog_row_image';

SET SESSION binlog_format = 'ROW';
SET
GLOBAL binlog_format = 'ROW';

3. Run Debezium with Kafka Connect:

Use Docker Compose to start Debezium and Kafka Connect:

services:
  debezium:
    image: debezium/connect:2.7.3.Final
    ports:
      - "8083:8083"
    healthcheck:
      test: [ "CMD-SHELL", "curl -f http://localhost:8083/ || exit 1" ]
    networks:
      - kafka-cluster
    environment:
      - BOOTSTRAP_SERVERS=b-4.kafka-dev-0.xxxx.c6.kafka.us-east-1.amazonaws.com:9092,b-3.kafka-dev-0.xxxx.c6.kafka.us-east-1.amazonaws.com:9092,b-1.kafka-dev-0.xxxx.c6.kafka.us-east-1.amazonaws.com:9092
      - GROUP_ID=MYSQL_1
      - CONFIG_STORAGE_TOPIC=debezium_connect_configs
      - OFFSET_STORAGE_TOPIC=debezium_connect_offsets
      - STATUS_STORAGE_TOPIC=debezium_source_connect_statuses
      - CONFIG_STORAGE_REPLICATION_FACTOR=1
      - OFFSET_STORAGE_REPLICATION_FACTOR=1
      - STATUS_STORAGE_REPLICATION_FACTOR=1
networks:
  kafka-cluster:
    driver: bridge

CONFIG_STORAGE_TOPIC, OFFSET_STORAGE_TOPIC, STATUS_STORAGE_TOPIC are topics in Kafka where connector will store its own configuration and perform synchronization.
CONFIG_STORAGE_REPLICATION_FACTOR, OFFSET_STORAGE_REPLICATION_FACTOR, STATUS_STORAGE_REPLICATION_FACTOR should always be 1 in replication factor.

4. Detect server_id that will be used in connector configuration

SHOW
VARIABLES LIKE 'server_id';

5. Configure the MySQL Connector

Kafka connect exposes HTTP port to establish and monitor connectors.

Create a MySQL source connector by sending a POST request to the Kafka Connect REST API:

curl -X POST http://localhost:8083/connectors -H "Content-Type: application/json" -d @../payload.json

{
  "name": "mysql-moderation-comments-connector",
  "config": {
    "connector.class": "io.debezium.connector.mysql.MySqlConnector",
    "tasks.max": "1",
    "database.hostname": "xxxx.us-east-1.rds.amazonaws.com",
    "database.port": "3306",
    "database.user": "user",
    "database.password": "password",
    "database.server.id": "this_is_mysql_server_id",
    "topic.prefix": "mysql-cdc",
    "database.include.list": "public",
    "table.include.list": "public.users",
    "schema.history.internal.kafka.bootstrap.servers": "b-4.kafka-dev-0.xxxx.c6.kafka.us-east-1.amazonaws.com:9092,b-3.kafka-dev-0.xxxx.c6.kafka.us-east-1.amazonaws.com:9092,b-1.kafka-dev-0.xxxx.c6.kafka.us-east-1.amazonaws.com:9092",
    "schema.history.internal.kafka.topic": "schema-changes.moderation",
    "include.schema.changes": true,
    "key.converter.schemas.enable": false,
    "key.converter": "org.apache.kafka.connect.json.JsonConverter",
    "value.converter.schemas.enable": false,
    "value.converter": "org.apache.kafka.connect.json.JsonConverter",
    "database.history.skip.unparseable.ddl": true
  }
}

Parameter	Details
connector.class	Class name of connector from JAVA_PATH in running container. There are pre-built available connectors - "plugins", also we can add any exising plugin to container or write our own.
tasks.max	only 1 task should be operatable at time - proper order and handling of bin log Kafka connect service uses connectors to start 1 or more task that do the actual work and distributes running tasks across the cluster of Kafka connect services. If any of services stopped or crashed those tasks will be redistributed to other running services
database.hostname	RDS endpoint IP or DNS
database.port
database.user
database.password
database.server.id	unique identifier of MySQL server - this is a master server ID
database.server.name	logical ID of the server or cluster of services, used as prefix for all kafka topics
topic.prefix	"mysql-server" - prefix that can be added to Kafka topic to distinguish it from other existing topics
database.include.list	coma-separated list of DBs whose CDC should be captured
schema.history.internal.kafka.topic	connector puts all DDL statements in this topic while reading the binlog. On restart the connector will recover the schema of the DB that existed in point in
key.converter	className of converter or transformer for Event
value.converter	value converter
table.include.list	name_of_the_table

6. Export from MySQL table

Once connector is setup it will create a snapshot of data and will ingest into Kafka. After that connector will monitor for new records and updates in CDC.

debezium-1 | INFO   MySQL|mysql-cdc|snapshot    Exported 496538 of 548957 records for table 'public.users' after 00:41:53.279   [io.debezium.relational.RelationalSnapshotChangeEventSource]
debezium-1 | INFO   MySQL|mysql-cdc|snapshot    Exported 605082 of 548957 records for table 'public.users' after 00:53:07.497   [io.debezium.relational.RelationalSnapshotChangeEventSource]
debezium-1 | INFO   MySQL|mysql-cdc|snapshot    Finished exporting 605274 records for table 'public.users' (1 of 1 tables); total duration '00:53:07.522'   [io.debezium.relational.RelationalSnapshotChangeEventSource]
debezium-1 | INFO   MySQL|mysql-cdc|snapshot  Releasing table read lock to enable MySQL writes   [io.debezium.connector.binlog.BinlogSnapshotChangeEventSource]
debezium-1 | INFO   MySQL|mysql-cdc|snapshot  Writes to MySQL tables prevented for a total of 00:53:13.755   [io.debezium.connector.binlog.BinlogSnapshotChangeEventSource]
debezium-1 | INFO   ||  WorkerSourceTask{id=mysql-moderation-comments-connector-0} Committing offsets for 10242 acknowledged messages   [org.apache.kafka.connect.runtime.WorkerSourceTask]

7. Verify Kafka Topic

Use kafka-console-consumer to check the topic for messages:

kafka-console-consumer --bootstrap-server localhost:9092 --topic mysql-cdc.your_database.your_table --from-beginning

Debezium Data Format

Debezium emits messages to Kafka in a JSON format with three main parts:

key (Identifies the specific database row.)
value (Contains the actual change event, with fields like before, after, op, ts_ms, etc.)

Fields:

before: State of the row before the change.
after: State of the row after the change.
op: Type of operation (c for create, u for update, d for delete).
source: Metadata about the event source.
ts_ms: Timestamp of the event.

{
  "op": "u",
  "source": {
    ...
  },
  "ts_ms": "...",
  "ts_us": "...",
  "ts_ns": "...",
  "before": {
    "userid": "1",
    "name": "bob"
  },
  "after": {
    "userid": "1",
    "field2": "alice"
  }
}

Hide schema from payload

There are multiple configurations and data formats that allow to transform event, hide not needed fields, also register custom SingleMessageTransformers etc.

Here we are instructing Debezium to exclude schema from the payload:

debezium.source.value.converter=org.apache.kafka.connect.json.JsonConverter
debezium.source.value.converter.schemas.enable=false

Conclusion

Debezium is an excellent choice for capturing data changes in real time from MySQL RDS and streaming them into Kafka.
Its support for schema change tracking, rich message format, and scalability make it ideal for modern data pipelines.

Whether you choose standalone mode for simplicity or cluster mode for fault tolerance, Debezium provides the flexibility to meet your needs.

Remote Debugging/Profiling NodeJS app running in private ECS cluster using AWS SSM and inspect protocol

Roman Tsypuk — Mon, 22 Sep 2025 05:59:41 +0000

Debugging a Node.js application running inside a container within an ECS cluster can be daunting. The challenge becomes especially significant in restricted environments like AWS ECS, where you lack direct control over the running workloads. Traditional debugging methods, such as SSH or exposing ports, are often not viable in these setups.

AWS provides a secure and efficient solution through AWS Systems Manager (SSM). By leveraging SSM documents and port forwarding, you can securely establish connections to your Node.js containers and expose the debugger port without compromising the security of your infrastructure.

In this post, we will walk through the steps to set up and perform remote debugging of a Node.js container using an SSM.

Debugging

The Node.js debugging process using the --inspect flag allows developers to attach a debugging client (such as Chrome DevTools) to the application and analyze its behavior in real-time.

Node.js uses the Chrome DevTools Protocol (CDP) for debugging.

Capabilities of Node.js Inspector:

Breakpoints: Pause execution at specific lines to inspect the state of variables and objects.
Heap Analysis: Capture and analyze snapshots of memory usage to identify leaks.
Call Stack: View the sequence of function calls leading to a particular point in execution.
Performance Profiling: Measure the performance of code execution and detect bottlenecks.
Step Execution: Step through code one line at a time to observe how the application progresses.

If successful, you’ll see output indicating that the port has been forwarded and the session is waiting for connections.

AWS ECS specifics and limitations for debugger setup

The main challenge in remote debugging is enabling port-forwarding between the application runtime and the machine where the debugger (e.g., Chrome DevTools) is running. This process is straightforward in local environments but complex in cloud-managed services like ECS.

Key challenges include:

Lack of direct access to the underlying infrastructure: ECS manages the container instances, making traditional SSH-based debugging methods impossible.
Maintaining security: Exposing debugger ports publicly can introduce vulnerabilities.
Securely forwarding ports: Ensuring the debugger port is accessible only to authorized clients while retaining private IP configurations.

AWS SSM addresses these issues by allowing secure execution of commands and port forwarding on remote workloads, providing a robust solution for debugging applications running in ECS.

Step 1: Prepare the Node.js Application

To debug the Node.js application, it must be started with the --inspect flag enabled. This exposes the debugging interface on a specified port.

Modify the ECS task definition to include the following command:

"command": ["node", "--inspect=0.0.0.0:9229", "app.js"]

/usr/src/node-red # node --inspect index.js
Debugger listening on ws://127.0.0.1:9229/9d82bc40-6fe9-4148-b1e8-8089b3351b9f
For help, see: https://nodejs.org/en/docs/inspector
Data
Starting memory leak simulation...
Memory usage: 3 MB
Memory usage: 3 MB

Here is a test sample application that emulates memory leak for analysis:

const memoryLeakArray = [];

function createLeak() {
  const leakObject = {
    timestamp: new Date(),
    data: Buffer.alloc(1024 * 1024) // Allocate 1MB of memory
  };
  memoryLeakArray.push(leakObject);
}

function printMemoryUsage() {
  const used = process.memoryUsage();
  console.log(`Memory usage: ${Math.round(used.heapUsed / 1024 / 1024)} MB`);
}

async function runLoop() {
  while (true) {
    createLeak();
    printMemoryUsage();
    await new Promise(resolve => setTimeout(resolve, 1000)); // 1 second delay
  }
}

console.log("Starting memory leak simulation...");
runLoop();

After making this change, redeploy the ECS task.

Step 2: Use SSM to Connect to the Container

AWS provides several predefined Systems Manager documents (SSM documents) for various tasks, and AWS-StartPortForwardingSession is among them. This document allows secure port forwarding from an AWS resource (such as an ECS container) to your local machine.

To forward the Node.js debugging port (9229) to your local machine, run:

aws ssm start-session \
--target ecs:<cluster-name>_<task-id>_<container-runtime-id> \
--document-name AWS-StartPortForwardingSession \
--parameters '{"portNumber":["9229"], "localPortNumber":["9229"]}'

Starting session with SessionId: xxxx-xxxx
Port 9229 opened for sessionId xxxx-xxxx.
Waiting for connections...

Connection accepted for session [xxxx-xxxx]

Step 3: Debug the Application

Open Chrome and navigate to chrome://inspect.

Under “Remote Target,” locate the forwarded debugging session and click “Inspect.”

By default, Chrome is configured with 9229 port, but you can use any custom port that was decided to be used for debugging.

Use Chrome DevTools to analyze and debug the Node.js process:

At the same time in console output message of remote debugger attached will be listed.

Memory usage: 3 MB
Debugger attached.
Memory usage: 3 MB
Memory usage: 3 MB
Memory usage: 3 MB

Monitor the event loop:

Analyze heap snapshots to identify memory leaks:

Step through code execution to pinpoint issues:

Conclusion

The beauty of this solution is a high security - the debugging port does not need to be exposed publicly, tunel is initiated dynamically between machine and container.
Even having no inbound/outbound, security groups for cluster (fully private isolated one) we can access to workloads.
And access to such Private Networks without additional network configuration decreases operational overhead.

Amazon ElastiCache Redis as a Vector Embeddings Storage for Semantic Search in AWS Community Blog posts

Roman Tsypuk — Thu, 18 Sep 2025 09:08:37 +0000

Abstract

The AWS Community Builders program has produced an enormous trove of insightful blog content over the years. These
posts, authored by community members across the globe, capture deep technical expertise, innovative use cases, and
countless hours of experience.
But as this body of content grows, finding related information becomes increasingly difficult using traditional keyword
search.

Challenge

To tackle this challenge, I experimented with using Amazon ElastiCache for Redis as a vector store to power semantic
search across all blog posts starting 2021.

This blog post outlines how I vectorized blog content, stored embeddings in Redis, and leveraged K-Nearest Neighbors (
KNN) to discover content semantically related to phrases or topic of interest, discover posts, cluster posts, identify
the trends etc.

Why Use Redis for Vector Search?

Traditionally, Redis is used for caching and fast key-value operations. However, with the introduction of the Redis
Vector Similarity Search (VSS) module, Redis can now be used as a high-performance vector database. This makes it a
powerful choice when you need millisecond-latency search over large datasets of embeddings, such as blog post vectors.

Benefits of Redis VSS:

Speed: In-memory performance with extremely low latency.
Simplicity: Store and retrieve vectors using Redis CLI or SDKs.
Scalability: Redis clusters on ElastiCache scale with demand.
Integration: Easy to integrate with Python NLP libraries and AWS services.

What is an Embedding and How Does it Work?

An embedding is a numerical representation of text that captures its meaning in a multi-dimensional space. Words,
sentences, or even entire documents are transformed into fixed-size vectors of floating-point numbers using models like
BERT or Sentence Transformers.

These vectors are constructed such that semantically similar content lies close together in vector space. For example,
posts discussing "AWS re:Invent keynote" and "major AWS announcements" would have similar embeddings even if they use
different words.

Target Architecture:

Each blog post description and content was passed through a transformer-based model.
The output vector (e.g., 384 dimensions) was stored in Redis.
When a query is made (e.g., "golden jacket"), it is also converted into an embedding.
Redis performs a KNN search, returning the closest blog vectors by cosine similarity or Euclidean distance.

Prepare schema in Redis

Once I have collected the raw data, I created index in Redis, all documents will be injected as json document
also having postDescriptionEmbeddings defined as SchemaFieldTypes.VECTOR

This vector will be calculated using same algorithm that will be used for the search.

when creating index, the dimension should exactly match to the dimension of model that is used for embeddings

Since Xenova/all-MiniLM-L6-v2 was used dimension 384. You can set any other model to check if accuracy is increased,

but vector size can not be changed so new index creation and full data re-ingestion will be required.

/*    (RAW COMMAND)
          FT.DROPINDEX idx:posts

          FT.CREATE idx:posts
          ON JSON
              PREFIX 1 "posts:"
          SCHEMA
          '$.postDescriptionEmbeddings': {
            type: SchemaFieldTypes.VECTOR,
            TYPE: 'FLOAT32',
            ALGORITHM: VectorAlgorithms.FLAT,
            DIM: 384,
            DISTANCE_METRIC: 'L2',
            INITIAL_CAP: 111,
            BLOCK_SIZE: 111,
            AS: 'postDescriptionEmbeddings',
          },
          '$.postDescription': {
              type: SchemaFieldTypes.TEXT,
              NOSTEM: true,
              SORTABLE: true,
              AS: 'postDescription',
          },
          '$.link': {
            type: SchemaFieldTypes.TEXT,
            NOSTEM: true,
            SORTABLE: true,
            AS: 'link',
          },

Ingesting Data into Vector-based storage

I choose Xenova/all-MiniLM-L6-v2 for Embedding and vectorization, so once model is initialized, all data in a
processing loop
will be passed to it to augment blog post with on more field postDescriptionEmbeddings

let modelName = "Xenova/all-MiniLM-L6-v2";
let pipe = await transformers.pipeline("feature-extraction", modelName);

const strippedText = stripHTML(post.description);
post["_id"] = post.id
post["postDescription"] = strippedText;

const sentenceEmbedding = await generateSentenceEmbeddings(strippedText, pipe);
post["postDescriptionEmbeddings"] = sentenceEmbedding;


await nodeRedisClient.json.set(`${POSTS_KEY_PREFIX}:${post.id}`, "$", {
  ...post
});

async function generateSentenceEmbeddings(_sentence, pipe): Promise<number[]> {
  let vectorOutput = await pipe(_sentence, {
    pooling: "mean",
    normalize: true,
  });

  const embeddings: number[] = Object.values(vectorOutput?.data);
  return embeddings;
}

Ingested data in Redis:

using redis-cli we can JSON:GET any random document and explore its fields:

JSON.GET posts:0a1a9e88-32e3-43af-86ce-c70b27ef9482

Besides standard data now it contains also postDescriptionEmbeddings vector (here I'm cutting digits not to represent the full 384 size of it):

{
  "link": "https://dev.classmethod.jp/articles/awsbasics-s3-tables-athena-apache-iceberg/",
  "createdAt": "2025-07-20T02:02:36.821Z",
  "postDescription": "This article introduces AWS S3 Tables, a new service simplifying Apache Iceberg table management on S3. It shows creating tables via console/CLI, CRUD operations with Athena, and highlights automatic optimization, ACID transactions, and seamless integration.",
  "postDescriptionEmbeddings": [
    -0.05445345863699913,
    -0.03639936074614525,
    -0.09256941080093384,
    -0.02023949660360813,
    0.016897333785891533,
    -0.05080459639430046,
    -0.047063130885362625,
    ...

    0.05633614584803581,
    0.030490988865494728,
    -0.035038333386182785,
    -0.014955290593206882,
    0.03450820595026016
  ]
}

Congratulations!!! Now we have a digital footprint of all CloudBuiders blog posts.

Algorithms to search vectors

No using same model Xenova/all-MiniLM-L6-v2 we can convert any search phrase into vector:

const nodeRedisClient = getNodeRedisClient();
let modelName = "Xenova/all-MiniLM-L6-v2";
let pipe = await transformers.pipeline("feature-extraction", modelName);

const searchTxtVectorArr = await generateSentenceEmbeddings(searchTxt, pipe);

Here is example of search phrase Golden Jacket that is transformed using Embedding model Xenova/all-MiniLM-L6-v2 into a Vector, this is what vector looks like (with dimension 384):

[
      -0.0519644059240818,    0.08905413746833801,    0.01799578033387661,
     0.049779586493968964,   0.035871658474206924,   0.020033041015267372,
      0.18083901703357697,   -0.05256553739309311,  0.0003853921080008149,
     -0.04665731266140938,    0.01234174519777298,   -0.02134784311056137,
      0.03652122616767883,  0.0066316151060163975,   0.001835306640714407,
      -0.0348852276802063,   0.026533348485827446,  -0.008266004733741283,
     0.023030489683151245,  -0.025720935314893723,   -0.04542113095521927,
   -0.0034457785077393055,   -0.00999429076910019,    0.05410130321979523,
     -0.07294181734323502,    0.04549195244908333,   -0.03297987952828407,
     0.039497267454862595,   -0.05792579427361488,   -0.08336731791496277,
     0.028136223554611206, -0.0004739225551020354,   -0.03645763173699379,
      0.02329356223344803,    -0.1444464474916458,   -0.02707810141146183,
     0.033310066908597946,  0.0071496316231787205,   -0.11640160530805588,
      0.10475143790245056,    -0.0539717860519886,  -0.039327893406152725,
     -0.08680122345685959,   -0.03737214207649231,   -0.02752811089158058,
     0.005382977891713381,  -0.037658315151929855,  -0.009982343763113022,
    -0.014092876575887203,     0.0367547869682312,   -0.07399916648864746,
     0.008599113672971725,   -0.07765820622444153,   0.013821756467223167,
      0.06757687777280807,    0.09657421708106995,  -0.008364601992070675,
     -0.05157032981514931,    -0.0255973469465971,   0.030314497649669647,
    -0.036268994212150574,   0.004683461040258408,   -0.05857793241739273,
     0.009458839893341064,     0.0592721663415432,     0.0403432659804821,
     -0.06441762298345566,    0.05017191544175148,   -0.05505743995308876,
     -0.04220070689916611,    0.05020679160952568,    0.02721044048666954,
  0.000019773140593315475,  0.0006230986909940839, -0.0004238316905684769,
       0.0330372080206871,  -0.040411293506622314,    0.02342337928712368,
      0.09403083473443985,  -0.030885351821780205,    0.03300361707806587,
     -0.06219857931137085,   -0.06710682064294815,    0.06534728407859802,
      0.02729918621480465,  -0.021849708631634712,    0.04849373549222946,
     -0.04060141369700432,   -0.01779020205140114,   0.015026750974357128,
      0.03931884467601776,  -0.016998404636979103,   0.020048541948199272,
      0.07344327867031097,  -0.027000991627573967,  -0.013987813144922256,
      0.07202176004648209,   -0.10249660909175873,   -0.07182241231203079,
        0.177199587225914,
  ... 284 more items
]

Having search vector and all posts descriptions vectorized and stored in Redis, using KNN we will search the nearest neighbor similarities:

const searchQuery =
  `*=>[KNN ${_resultCount} @postDescriptionEmbeddings $searchBlob AS score]`;
results = await nodeRedisClient.ft.search(POSTS_INDEX_KEY, searchQuery, {
  PARAMS: {
    "searchBlob": float32Buffer(searchTxtVectorArr)
  },
  RETURN: ["score", "description", "createdAt", "region", "link"],
  SORTBY: {
    BY: "score",
  },
  DIALECT: 2,
});

Let's find a CloudBuilders based on a particular topic

Searching Golden Jacket relevant topics. Got search output:

total, number of found documents found with limit
score, vectors compatibility for each document returned

{
  "total": 5,
  "documents": [
    {
      "id": "posts:79256de8-00aa-4127-b025-df410fb63414",
      "value": {
        "score": "1.01139831543",
        "description": "Although I cannot have a AWS 100% fully certified Golden Jacket now, I still buy myself a 25% fully certified Golden Hat 👒 for a little FUN 🤩 Never too late to learn Amazon Web Services (AWS) ! Start your cloud journeys TODAY ! You will become Fully Certified in the future.",
        "link": "https://www.linkedin.com/posts/walterwlee_aws-cloud-future-activity-7020547672751169536-T3r8?utm_source=share&utm_medium=member_desktop"
      }
    },
    {
      "id": "posts:e05e61f3-1cd9-4483-98c7-ad3681a1ea00",
      "value": {
        "score": "1.03479576111",
        "description": "Thanks a lot to the Amazon Web Services (AWS) community builder jacket ! 😀 very warm and high quality with YKK zipper 🥰",
        "link": "https://www.linkedin.com/posts/walterwlee_aws-awscommunity-awscommunitybuilders-activity-7010055038479859712-ahY-?utm_source=share&utm_medium=member_desktop"
      }
    },
    {
      "id": "posts:2f63f16d-8080-4e27-baed-f5c582617106",
      "value": {
        "score": "1.07405209541",
        "description": "A blog post sharing my experience doing the AWS certifications for the Golden Jacket, for others who might be interested in going through the same experience.",
        "link": "https://chamila.dev/blog/2024-10-28_completing-11-professional-certifications-in-8-weeks/"
      }
    },
    {
      "id": "posts:554df865-951d-4937-abad-5ef84f6e67fe",
      "value": {
        "score": "1.12301290035",
        "description": "How you can achieve all AWS certifications and get the golden jacket - Part 1 ",
        "link": "https://dev.to/electrokat/how-you-can-achieve-all-aws-certifications-and-get-the-golden-jacket-part-1-16dp"
      }
    },
    {
      "id": "posts:9a772e6c-f3fe-419a-afe2-b5b0038dbdcb",
      "value": {
        "score": "1.2975268364",
        "description": "How you can achieve all AWS certifications and get the golden jacket - Part 2 - How to study ",
        "link": "https://dev.to/electrokat/how-you-can-achieve-all-aws-certifications-and-get-the-golden-jacket-part-2-how-to-study-2b4c"
      }
    }
  ]
}

search results for "Golden Jacket":

Score	Details
1.01139831543	Although I cannot have a AWS 100% fully certified Golden Jacket now, I still buy myself a 25% fully certified Golden Hat 👒 for a little FUN 🤩 Never too late to learn Amazon Web Services (AWS) ! Start your cloud journeys TODAY ! You will become Fully Certified in the future.
1.03479576111	Thanks a lot to the Amazon Web Services (AWS) community builder jacket ! 😀 very warm and high quality with YKK zipper 🥰
1.07405209541	A blog post sharing my experience doing the AWS certifications for the Golden Jacket, for others who might be interested in going through the same experience.
1.12301290035	How you can achieve all AWS certifications and get the golden jacket - Part 1
1.2975268364	How you can achieve all AWS certifications and get the golden jacket - Part 2 - How to study

search results for "play games":

Score	Details
1.49744784832	🚀 Vibe coding a simple STEM Game with Amazon Q CLI —so I created this simple game to help make learning easier with the help of Amazon Q CLI 🤖✨.
1.54997515678	Learn DevOps By Doing Projects
1.56706357002	I wrote a blog post about building a web-based game using only Amazon Q Developer. The post received 199 readers and 26 reactions from the community.
1.57633209229	This article about my personal challenges as a sales and business developer to have interaction with tech. persons.
1.61025881767	Join the AWS GenAI PartyRock hackathon!

What else can we do with vectors?

Discover similar blog posts and engage with authors.
Track evolving themes across community content.
Understand what topics resonate or need more visibility.
Retrieve relevant document chunks or paragraphs to feed into an LLM (e.g., via Retrieval-Augmented Generation or RAG).
Document Classification
Posts with similar embeddings can be linked with a virtual “related-to” edge. You have read a post and enjoy it, vectorDB can recommend other posts that have similar vector

Observations:

The majority of CPU intensive work is on the producer that is running the model and performs the transformation.

The important caviat is that model must be instantiated only once and all data is fed through it. Otherwise, if model is
created for
every data entry, it will take a lot of time to finish the ingession plus unneeded memory alocation will be performed.

Redis instance based on htop info does not have spike during data ingestion, same when search queries are performed.
On data retrieval, the most time-consuming operation (1sec) was request transformation into vector

There are some parameters that allow to adjust output:

return results count — you can return different response batches or implement a cascaded system, that will do next batch call and lower the threshold to establish data
score threshold — the lower the value, the more closer vectors are. However, this is true for a mathematical model that was used but sometimes not to common scence.

Conclusions

Using Elastic Cache Redis as a vector store offers a powerful, low-latency way to bring semantic search to the AWS
Community Builder blog archive. By moving beyond keywords and enabling intelligent discovery, we help amplify the voices
and contributions of community members everywhere.

This architecture is not limited to AWS community content—it can be adapted for any developer-driven knowledge base. The
future of search is vectorized, and Redis makes that future fast, scalable, and accessible.

Running analytics queries from your machine with Zero-BigData stack needed on top of AWS S3

Roman Tsypuk — Thu, 18 Sep 2025 08:59:52 +0000

Abstract

Cloud data lakes often store large volumes of logs, metrics, or event data in AWS S3. The formats vary:
newline-delimited JSON (often gzipped), CSV, or more optimized columnar formats like Parquet. Traditionally, querying
this data meant spinning up heavyweight tools (Athena, Spark, Presto, EMR).

But what if you could query the data instantly, from your laptop, without standing up an
entire cluster? Enter DuckDB.

DuckDB is an embedded analytical database — think “SQLite for analytics” — with first-class support for Parquet, JSON,
CSV, and remote storage like S3. The dreams come true.

In this post, we’ll explore:

Querying gzipped JSON files directly from S3
Comparing JSON vs Parquet vs CSV performance
Joining local and remote datasets
Calling HTTP APIs from DuckDB (yes, you can!)

Getting Started with DuckDB

DuckDB works in Python, R, Go, Nodejs, Rust, Java, CLI, or as a C library. If you a fan of JetBrains DataGrip it also can includes duckdb connector. The DuckDB CLI (Command Line Interface) is a single, dependency-free executable. Installation is simple.

pip install duckdb

brew install duckdb

1. Querying Local FS

After starting cli, we can easily access csv file stored in local FS, querying them as they are regular tables,
using SQL operators and predicates. Very often we need to open some csv file, analyze its structure or extract particular raws based on prodicate.Here I'm querying data from local FS csv file with limit:

SELECT *
FROM read_csv_auto('/tmp/oxford_5000.csv') LIMIT 100;

column0	word	type	cefr	phon_br	phon_n_am	definition	example
0	a	indefinite article	a1	/ə/	/ə/	used before countable or singular nouns referring to people or things that have not already been mentioned	a man/horse/unit
1	abandon	verb	b2	/əˈbændən/	/əˈbændən/	to leave somebody, especially somebody you are responsible for, with no intention of returning	abandon somebody, The baby had been abandoned by its mother.
2	ability	noun	a2	/əˈbɪləti/	/əˈbɪləti/	the fact that somebody/something is able to do something	People with the disease may lose their ability to communicate.

2. Querying Gzipped JSON on AWS S3

Suppose you have logs or events data that is ingested into DataLake by pipeline and they are stored in AWS S3 partitioned by date with a virtual path and also gzipped json. This is very common architecture.

Duckdb works perfectly with partitions, based on query it can access proper paths and also unzip the data. It auto-detects
schema, even across multiple files.

s3://my-bucket/logs/year=2025/month=09/day=16/data.json.gz

Directly querying data with DuckDB:

SELECT *
FROM read_json_auto(
  's3://my-bucket/logs/*.json.gz',
  format = 'json',
  compression = 'gzip'
     );

The beauty is that we do not need to bootstap any engine, create AWS IAM permissions, add service and pay for them - just use cli tool that is local on your machine.

Duckdb extensions and AWS credentials

The httpfs extension supports reading/writing/globbing files on object storage servers using the S3 API. S3 offers a standard API to read and write to remote files (while regular http servers, predating S3, do not offer a common write API). DuckDB conforms to the S3 API, that is now common among industry storage providers.

To query S3 firstly we need to add and enable extensions:

-- set s3 compatibility
INSTALL
httpfs;
LOAD
httpfs;

The httpfs filesystem is tested with AWS S3, Minio, Google Cloud, and lakeFS. Other services that implement the S3 API (such as Cloudflare R2)

After setup add S3 credentials via environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY) or IAM role of the profile (more prefered).

Adding secrets

-- set aws creds
INSTALL
aws;
LOAD
aws;

CREATE
SECRET env_dev (
    TYPE s3,
    PROVIDER credential_chain,
    REFRESH auto
);

SET
s3_region='us-east-1';
SET
s3_access_key_id='ASIAXXXXXX';
SET
s3_secret_access_key='';
SET
s3_session_token='';

Another option is to use exisitng AWS profiles and switch between them

CALL load_aws_credentials('dev');
CALL load_aws_credentials('prod');

Check credentials:

SELECT *
FROM FROM duckdb_secrets();

3. Parquet and CSV on S3

DuckDB shines with columnar formats like Parquet:

SELECT user_id, COUNT(*) AS events
FROM read_parquet('s3://my-bucket/logs/*.parquet')
GROUP BY user_id
ORDER BY events DESC;

For CSV:

SELECT *
FROM read_csv_auto('s3://my-bucket/data/*.csv.gz');

Parquet is much faster and smaller than JSON/CSV, but the magic is that DuckDB handles them all.

4. Loading Data from S3 storing locally in Parquet or other format

-- dev
CREATE
OR REPLACE VIEW events_2025_block AS
SELECT *
FROM read_json_auto(
  's3://events/block-events/block/year=2025/month=*/day=*/hour=*/*.json.gz',
  filename = true, -- adds a _filename column
  hive_partitioning = 1, -- pulls year/month/day/hour from the path into columns
  format = 'newline_delimited' -- omit if they’re arrays/objects; keep if NDJSON
     );

SELECT *
FROM events_2025_block
WHERE
  action == 'data-verified'
  LIMIT 10;

Writing extracted data from S3 to local FS and automatically converting into Parquet :

COPY events_2025_block TO '/tmp/dev.s3.events.parquet' (FORMAT parquet);

SELECT *
FROM '/tmp/dev.s3.events.parquet'
WHERE action == 'data-verified'
ORDER BY created_at DESC
  LIMIT 100;

5. Local + Remote Joins

You can join S3 data with local datasets:

WITH s3_data AS (
SELECT user_id, timestamp
FROM read_parquet('s3://my-bucket/events/*.parquet')
),
local_users AS (
SELECT * FROM read_csv_auto('users.csv')
)
SELECT u.name, COUNT(*) AS actions
FROM s3_data s
JOIN local_users u USING(user_id)
GROUP BY u.name;

This means you don’t need to ETL everything into a central DB just to query.

6. Calling HTTP APIs from DuckDB

DuckDB supports reading remote HTTP URLs directly. I have JSON-based aws news for different topics hosted at my github pages, let's call it to get latest aws architecture news:

SELECT *
FROM read_json_auto('https://tsypuk.github.io/aws-news/news/architecture.json')
WHERE title LIKE '%Lambda%' LIMIT 50;

title	link
How Zapier runs isolated tasks on AWS Lambda and upgrades functions at scale	https://aws.amazon.com/blogs/architecture/how-zapier-runs-isolated-tasks-on-aws-lambda-and-upgrades-functions-at-scale/
How Launchpad from Pega enables secure SaaS extensibility with AWS Lambda	https://aws.amazon.com/blogs/architecture/how-launchpad-from-pega-enables-secure-saas-extensibility-with-aws-lambda/

Same queries work with Parquet/CSV hosted over HTTPS. This is handy for blending APIs + S3 data.

7. Using standard DBs like MySQL Postgres

INSTALL postgres;
LOAD postgres;
ATTACH 'dbname=postgres user=postgres host=127.0.0.1' AS db (TYPE postgres, READ_ONLY);

Why DuckDB?

Lightweight: No cluster, runs anywhere (laptop, container, Lambda).
Format-flexible: JSON, CSV, Parquet, ORC.
Fast: Vectorized execution engine, optimized for analytics.
Great for ad-hoc exploration of data lakes.

When to Use Athena vs DuckDB

Use Athena when: queries must run at scale, multiple teams need SQL access, complex analytics require a lot of cpu/memory resources that are not available of local machine
Use DuckDB when: you need quick exploration, prototyping, data science, or embedding analytics in an app, access to QA/STG env where not so big volume size

Conclusions

DuckDB is like a Swiss Army knife for data lakes. With just a few lines of SQL, you can query gzipped JSON from S3, join
it with local CSVs, compare with Parquet, and even call APIs. It’s lightweight, fast, and great for both prototyping and
production embedding.

Try it next time you need to peek inside your S3 bucket without firing up a full big-data stack.

Turning Text into Art: Using AWS Nova to augment Blog post with illustrations

Roman Tsypuk — Wed, 14 May 2025 20:11:00 +0000

Abstract

When hosting a post content majority of hosting platforms provide the top header image section that acts as a hook to associate blog's content with some visualization and this is actually the first big thing that audience will see.

For content creators there are many libraries with free images like pexel, also there non-free subscription based photos/images catalogues. Often you spend some time to select an image that is more accurate to blog post content.

But, such platforms perform images search only based on single tag input - so you can not fully search/describe the image that you are interested in.

The idea

Having access to AWS Nova and looking at its latest features I got the idea that such images search (actually generation) based on post content can be automated using AWS Nova Canvas.

The implementation is a cli tool, that:

perform as a regular cli script (can be integrated into makefile, pipelines, flows and triggered manually)
should have blog-post file in MD-format as input
should be wizard-based (have step-by-step guided for user)
should automatically summarize blog post text into few sentences (using some of Hugging Face summary models), but also have possibility to manually enter the summary
read specific settings of fine tuning from yaml configuration file
call AWS Nova Canvas to generate images according to summary and tuning settings

Configuration is extracted into YAML file

AWS Nova Canvas has specific features that are a good candidate for extraction into commonly used configuration format yaml. Having them isolated allows to tune parameters without cli tool modification.

By using them we can control the behaviour of image generator:

colors:
  - "#FF9900"
  - "#3366FF"
numberOfImages: 5
width: 1280
height: 720
seed: 12
cfgScale: 6.5
summaryModel: "facebook/bart-large-cnn"

colors: set the color gamma that will be used by AWS Nova when generating images
numberOfImages: choose 3-5 so you can peel up from them
width x height in pixels
seed, cfgScale are Nova model controls:
summaryModel: by default facebook large bart is used to prepare summary based on blog post content, can be switched to small or other model provider

Text Summarization

This can be achieved using online cloud services, like AWS Comprehend, Titan Models AWS Bedrock, Sagemaker, etc.

But I decided to apply offline local summarization using one of Hugging Face model facebook/bart-large-cnn" - it is Meta pretrained transformer model some variant of the BART (Bidirectional and Auto-Regressive Transformers) architecture, that fine-tuned for text summarization (particularly on the CNN/DailyMail dataset).

The goal is that later I can easily switch model to any other from Hugging Face, that is more specific to knowledge domain, latest or more accurate.

CLI tool execution is wizard-based

Trigger flow

I have a new blog post, (actually this one) that is under 2025-05-12-amazon-nova-canvas.md file MD-format and need to generate image for header.

To do this I'm invoking with python in cli:

python3 ./scripts/nova_augment.py 2025-05-13-amazon-nova-canvas.md

Or through Makefile:

make gen_image FILE=2025-05-13-amazon-nova-canvas.md

Description is automatically extracted from post (if present)

If description is accurate and enough for images generation we can continue.
Otherwise, we can select - generate summary from content or input summary manually:

--- Description extracted from front matter ---
Amazon Nova Canvas: creating images for blog posts

 What to use as a summary?
1. Direct input
2. Generate based on content
3. Use current Description

Your choice (1/2/3):

Next we can directly input summary or ask our model to generate summary based on post content.

Summary is generated from post content

--- Generated Summary by facebook/bart-large-cnn model---
I have a new blog post, actually this one that is under ``2025-05-12-canvas.md`` file and need to write. It's about creating images for blog posts with Amazon Nova Canvas.
First we check if post contains title or description, if so it outputs it. Next we can directly input summary or ask our model to generate summary based on post content. generate the header image.

Specify color schema

Color schema defined in yaml file can instrument AWS Nova to generate images using main defined colors. At this step we are selecting to apply such config:

Color Guided Generation:
1. Apply color guided generation ['#FF9900', '#3366FF']
2. Skip color guided generation

Do you want Color Guided Generation (1/2): 1

Generating 5 images (1280x720) ...
Image saved to output_image_0.png
Image saved to output_image_1.png
Image saved to output_image_2.png
Image saved to output_image_3.png
Image saved to output_image_4.png

Once processed - 5 images are generated, now I can review them and select the most appropriate for the post or perform re-prompt.

Few examples of prompts VS images generation by AWS Nova

I was impressed by images quality and want to add here some examples that have initial prompt and generated images:

Prompt 1:

"Ship arrives to Robinson Crusoe island"

Prompt 2:

"In the shadow of the House of Dragons, even kings kneel and the brave hold their breath."

Nova is great for RPG games images generation - I'm adding in prompt a phrase from "Dungeons & Dragons" table game, we played in the childhood
a clone copy of this amazing game that was available in our country under the name "Enchanted Land".
So long ago our mind and generation were drawing all those characters and now it is so easy to build using AI.

Prompt 3:

Prompt I used to generate header image to this post:

"Car of the future for entire family. It can drive fly and swim and has maximum comfort and safety."

Lifehack: images resize in a second

At my blog I maintain same images in different copies with custom resolutions - for PC and mobile (to save users data traffic).

To resize image I'm using mogrify utility - just hit enter and image is done, no need to open any application, navigate menus, enter px - extremely fast & useful:

mogrify -resize 300 -quality 100 generated/nova/output_image_4.png

Code snippets of core blocks:

Summarization with Meta

def summarize_text(text: str, model) -> str:
  from transformers import pipeline
  summarizer = pipeline("summarization", model=model)
  # Handle long content by summarizing in chunks (Bart's max is ~1024 tokens)
  max_chunk = 1000
  chunks = [text[i:i + max_chunk] for i in range(0, len(text), max_chunk)]
  summary = ''
  for chunk in chunks:
    out = summarizer(chunk, max_length=150, min_length=40, do_sample=False)
    summary += out[0]['summary_text'] + '\n'
  return summary

AWS Nova Canvas image generation

def generate_image_with_nova(prompt: str, model_name: str, config, color: bool = None) -> None:
  bedrock = boto3.client('bedrock-runtime', region_name='us-east-1')  # adjust region

  nova_request = json.dumps(
    {
      "taskType": "TEXT_IMAGE",
      "textToImageParams": {
        "text": prompt
      },
      "imageGenerationConfig": {
        "cfgScale": config["cfgScale"],
        "seed": config["seed"],
        "width": config["width"],
        "height": config["height"],
        "numberOfImages": config["numberOfImages"],
        "quality": "standard"
      }
    }
  )

  if color:
    nova_request = json.dumps(
      {
        "colorGuidedGenerationParams": {
          "text": prompt,
          "colors":
            config["colors"]
        },
        "taskType": "COLOR_GUIDED_GENERATION",
        "imageGenerationConfig": {
          "cfgScale": config["cfgScale"],
          "seed": config["seed"],
          "width": config["width"],
          "height": config["height"],
          "numberOfImages": config["numberOfImages"],
        }
      }
    )

  response = bedrock.invoke_model(
    modelId=model_name,
    body=nova_request,
    contentType='application/json',
    accept='application/json'
  )

Images extraction from payload

result = json.loads(response['body'].read())
  from base64 import b64decode

  # Check if images are available in the response
  # make directory 'generated/current_date'
  folder_name = datetime.now().strftime("%Y-%m-%d")
  os.makedirs(f"generated/{folder_name}", exist_ok=True)

  if 'images' in result and result['images']:
    for idx, image_data in enumerate(result['images']):
      if image_data:
        with open(f"generated/{folder_name}/output_image_{idx}.png", "wb") as f:
          f.write(b64decode(image_data))
        print(f"Image saved to output_image_{idx}.png")
      else:
        print(f"No image data returned for image {idx}.")
  else:
    print("No images found in the response.")

Conclusions

AWS Nova Canvas is a great tool for generation of images that has prebuilt configurations and also allows to customize the uniqueness of data.

It is solid and easy with integration through SDK into any flow and provides excellent visualizations.

Interesting observation - weeks later same prompts are generating the same images (maybe it is a cache/map per aws account or global model state, that after retrained and redeployment will show other results).

Developing chrome extension for converting web-pages into obsidian-like Second Brain with Amazon Q cli

Roman Tsypuk — Sat, 03 May 2025 14:23:14 +0000

This is a submission for the Amazon Q Developer "Quack The Code" Challenge: Crushing the Command Line

What I Built

There’s an overwhelming amount of great content on the web, and alongside that, local tools like Notion, Obsidian, and Quartz have become increasingly popular. They’re excellent for indexing information, performing advanced searches, and staying accessible — even offline.

The challenge is, while browsing online, we often come across content we’d love to revisit later. But after jumping through a few more links, we forget where we started. And if we try to save a page to something like Quartz in Markdown format, it turns into a tedious manual process: cleaning up the content, converting it to Markdown, copying over all the images, fixing links, creating sections, adding formatting and styles, etc.

This kind of repetitive work is a perfect candidate for automation.

That’s why we’re building a Chrome browser extension that handles all of it for you. It will parse the page you’re interested in, convert it into clean Markdown, download and link images properly, and store everything in your second brain.

We’ll build it from scratch using Amazon Q — automating the boring stuff so you can focus on what actually matters: reading and thinking.

Extension installation

If your are new to development the use option1, otherwise feel free to checkout the code and build it

option1: download extension from https://github.com/tsypuk/mdextension use pre-build extension in dist folder
option2: checkout & build extension

git clone https://github.com/tsypuk/mdextension.git
npm install
npm run build

Once extension is built (or you have downloaded a pre-built), open browser -> Manage Extensions -> Load unpacked -> open path to *dist folder and add.

Demo

While browsing the web, we came across an intriguing article on data restoration from SSDs — a hot topic right now. Since we maintain a structured local knowledge base (our “second brain”) with tagged, categorised, and interlinked content for easy future reference, this article definitely deserves a place there.

It's valuable information that we’ll want to revisit later, so capturing and storing it properly is a must.

second brain is the concept when there is hosted content storage that allows to save text, media content, also build relationship for this content and search. Most popular second brains are: quartz/obsidia/notion, etc.

Click on extension icon in browser extensions bar:

Add tags if needed or leave a default tags, specify the name of folder where to download (it will be folder in downloads) and click Save page as Markdown

extension will parse page, convert it to markdown, download images, integrate images into markdown and save to you local download folder

Now you can continue browsing, the content will not be forgotten - it is in your second brain library.

For each saved markdown file, extension also adds yaml-based obsidian description - it extracts page title, description adds tags that you have entered on save.

---
title: Recover data from an SSD drive | Information Recovery Center
description: Data Recovery from SSD - Kingston, OCZ, Transcend, Intel, Corsair, Silicon Power, Patriot, A-Data, Crucial, Western Digital, Samsung, Apacer
tags:
  - ssd
  - recovery
published: true
date: 2025-05-03
---

All the page text is read from html and is reformatted into markdown format including references, links, images, headers, etc. Here is example of few first sentenced.

# Recover data from an SSD drive

We recover information from **SSD (Solid State Drive) **drives with the help of specialized software and hardware complexes. SSD drives are quite reliable media. However, they are not protected from damage and at one moment can fail, which entails loss of valuable information. We are always ready to help you restore lost photos, videos, documents, music, and more.

**Types of media **

We work with all types of **SATA, PATA, mSATA, M.2 (NGFF), M.2 PCIe NVMe/АHCI, LIF, ZIF, NVMe, USB, Apple Macbook, PCI-Express, PCIe x16 и SATA Express **interfaces: **from 64 Gb to 2 TB. **.

Now these downloaded markdowns can be easily integrated into you second brain. Here is how they are rendered in my quartz brain:

So you will not lost any interesting ideas, content, details - they will always be stored into your second brain.

Code Repository

https://github.com/tsypuk/mdextension

How I Used Amazon Q Developer

start a q cli and use prompting

q
To learn more about MCP safety, see https://docs.aws.amazon.com/amazonq/latest/qdeveloper-ug/command-line-mcp-security.html
✓ awslabsaws_documentation_mcp_server loaded in 3.93 s
✓ 1 of 1 mcp servers initialized

Welcome to

 █████╗ ███╗   ███╗ █████╗ ███████╗ ██████╗ ███╗   ██╗     ██████╗
██╔══██╗████╗ ████║██╔══██╗╚══███╔╝██╔═══██╗████╗  ██║    ██╔═══██╗
███████║██╔████╔██║███████║  ███╔╝ ██║   ██║██╔██╗ ██║    ██║   ██║
██╔══██║██║╚██╔╝██║██╔══██║ ███╔╝  ██║   ██║██║╚██╗██║    ██║▄▄ ██║
██║  ██║██║ ╚═╝ ██║██║  ██║███████╗╚██████╔╝██║ ╚████║    ╚██████╔╝
╚═╝  ╚═╝╚═╝     ╚═╝╚═╝  ╚═╝╚══════╝ ╚═════╝ ╚═╝  ╚═══╝     ╚══▀▀═╝



╭───────────────────────── Did you know? ─────────────────────────╮
│                                                                 │
│   Q can use tools without asking for confirmation every time.   │
│                     Give /tools trust a try                     │
│                                                                 │
╰─────────────────────────────────────────────────────────────────╯


/help all commands  •  ctrl + j new lines  •  ctrl + k fuzzy search
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

let's create chrome extension written on ts in this folder - init the project

I'll help you create a Chrome extension using TypeScript in your current directory. Let's start by initializing the project.

First, let's check what's currently in the directory:


🛠️  Using tool: fs_read (trusted)
 ⋮
 ● Reading directory: ~/projects/markdown-extension with maximum depth of 0
 ⋮
 ● Completed in 0.1s

Let's initialize a new npm project and set up the TypeScript configuration:
🛠️  Using tool: execute_bash
 ⋮
 ● I will run the following shell command:
cd ~/projects/markdown-extension && npm init -y

At this moment Q has generated a scratch template for chrome extension, connected dependencies, folders structure and also defined README.MD

let's add .gitignore file with node_modules to exclude, also exclude webstorm idea files and builds

Gitignore is perfectly structures contains all requested exclussions.

Do not forget to use git and init it in your project, otherwise it will be a ball of mud.

git init
git commit -m "init"

the extension should be active on any site, when clicking on the extention icon in the bar, in will get current page content, transform it to markdown, we should save this markdown file and also all images that are contained on the page, images have references in markdown according to its syntax, download as a folder

Very glad to see that Q responses with a summarised plan before the implementation.

I'll help you modify the extension to meet these requirements. The extension should:
1. Be active on any site
2. When clicking the extension icon, get the current page content
3. Transform the page content to markdown
4. Save the markdown file
5. Download all images from the page
6. Include proper image references in the markdown
7. Save everything as a folder

Let's update the extension code:

Q performs updates of files, let's check what is done. The only missing thing is to perform npm install cause packages are missing and compilation does not pass. Q missed that but has reported that it was done.

Let's do it manually:

npm run build

markdown-extension@1.0.0 build
webpack --config webpack.config.js

assets by path *.js 109 KiB
  asset popup.js 104 KiB [emitted] (name: popup) 1 related asset
  asset content.js 4.34 KiB [emitted] (name: content) 1 related asset
  asset background.js 804 bytes [emitted] (name: background) 1 related asset
asset popup.html 1.07 KiB [emitted] [from: src/popup.html] [copied]
asset manifest.json 639 bytes [emitted] [from: src/manifest.json] [copied]
runtime modules 221 bytes 1 module
modules by path ./src/*.ts 10.9 KiB
  ./src/background.ts 602 bytes [built] [code generated]
  ./src/content.ts 4.15 KiB [built] [code generated]
  ./src/popup.ts 6.15 KiB [built] [code generated]
./node_modules/jszip/dist/jszip.min.js 95.3 KiB [built] [code generated]
webpack 5.99.7 compiled successfully in 2116 ms

No issue with build.

First failure

Now when installing extension to browser, we got the error:

This is due to icons are defined with different resolutions in manifest file, but there are no icon files at all.

Let's see if Q will understand this error and could fix it - we are pasting error message as is without any additional prompt at all:

🛠️  Using tool: fs_write
 ⋮
 ● Path: ~/projects/markdown-extension/src/icons/create_icons.sh

+     1: #!/bin/bash
+     2:
+     3: # This script creates simple placeholder icons for the Chrome extension
+     4: # It requires ImageMagick to be installed
+     5:
+     6: # Create a simple colored square for each icon size
+     7: convert -size 16x16 xc:#4285f4 icon16.png
+     8: convert -size 48x48 xc:#4285f4 icon48.png
+     9: convert -size 128x128 xc:#4285f4 icon128.png
+    10:
+    11: echo "Icons created successfully!"

Creating: ~/projects/markdown-extension/src/icons/create_icons.sh
 ⋮
 ● Completed in 0.0s

🛠️  Using tool: execute_bash
 ⋮
 ● I will run the following shell command:
cd ~/projects/markdown-extension/src/icons && chmod +x create_icons.sh && ./create_icons.sh

Amazing Q understood the error, and also generated images with updating and gives the instructions:

I've fixed the issue with the missing icons and made several improvements to the extension. Here's what I did:

1. Created the missing icon files:
   • Generated placeholder icons in the required sizes (16x16, 48x48, 128x128)
   • Added them to the src/icons directory

2. Updated the webpack configuration:
   • Made sure the icons directory is properly copied to the dist folder
   • Added fallbacks for Node.js modules that might be referenced

3. Improved the content script:
   • Enhanced the HTML to Markdown conversion logic
   • Added better image collection with proper error handling
   • Improved filename sanitization

4. Updated the popup script:
   • Simplified the download process using Chrome's downloads API
   • Added better error handling
   • Improved the user feedback in the status area

5. Fixed the manifest.json:
   • Added the required "downloads" permission
   • Made sure all icon references are correct

Now the extension should load properly and function as expected. When you click on the extension icon and then click "Save Page as Markdown", it will:

1. Convert the current page to markdown
2. Download the markdown file
3. Download all images from the page into a folder

To build and load the extension:
1. Run npm install to install dependencies
2. Run npm run build to build the extension
3. Load the extension from the dist folder in Chrome's extension page

The extension now has a simpler approach that uses Chrome's native downloads API instead of trying to create a ZIP file, which makes it more reliable and
easier to maintain.

Let's rebuild and check extension upload again:

Now it is fully installed and available at chrome:

After that steps, doing some minor adjustments, format transformations and headline tuning extension is ready.

What I learned from Q

Prompt - the clear your message is - the better are results, define edge cases and clear requirements for the task. Try to be iterative, not follow big bang implementation. Add feature by feature, step by step.
You should use git and perform commits between prompts and output results. Otherwise it will be hard to troubleshoot and do modifications.
Q holds the context and tracks right direction with more input prompts you give.
On the step were built extension was not able to install into the browser, I passed error message to Q without any details, it understood the root cause and performed missing images generation.
Interesting idea is for web developer will be great to have feedback loop - this can be a q developer extension that is installed into web browser, so developer can build new version of ui application and q extension can verify how it is rendered in browser or what are errors in browser console - with such improvement Q will be even more robust cause it will immediately have a feedback loop without even human interaction only under human control.
Q pretty good handles linux cli, scripting prompts, but also can write a code. With MCP integration to Q (that was released recently) it will be even more interesting to see its evolution.
Very good to see summaries from Q, execution plan, and clarification question - this sound more natural and you got feeling of more control.
If you are doing changes from IDEA - Q does not track them. These new features are out of Q context. And if after manually adding features/code you will return to Q and ask to implement something new - it will drop all you code. Be careful!!!
Styling problem - Q uses checkstyle different from default IDEA (spacing, tabs, etc.). When developing in parallel make sure you have aligned styles, otherwise all files will be periodically change even with formatting, making fully imposible to track cahnges.

Links:

@tsypuk