DEV Community: Kentaro Matsumoto

Comparing Bedrock Knowledge Bases: S3 Vectors vs. Neptune Analytics

Kentaro Matsumoto — Wed, 17 Jun 2026 06:41:09 +0000

1. Introduction

My initial understanding was that OpenSearch or S3 Vectors would be the go-to databases when building a RAG system with Bedrock Knowledge Base. However, after hearing that GraphRAG might be a better fit for certain use cases, I decided to dive into GraphRAG using Neptune Analytics to explore the differences.

2. What I Did

Prepared a list of documents where using GraphRAG is likely to improve search and retrieval accuracy.
Created two Bedrock Knowledge Bases (an S3 Vectors version and a Neptune Analytics version) using the exact same source documents.
Submitted queries to both and compared the accuracy and quality of the responses between the S3 Vectors and Neptune Analytics versions.

3. Architecture Diagram

4. Procedure

4.1 Preparing the Document List

I had Gemini generate a list of documents that share relationships with one another, and saved them to S3. The following five documents were used:

doc_01_billing_service.txt

Internal Service Architecture: Billing Application
The company's core Customer Billing Service (Service-ID: BILL-PROD) is designed as a microservice to handle all subscription renewals. To process payments and retrieve ledger data in real-time, this Billing Application establishes a persistent, high-throughput connection to the primary transaction database engine, known in our inventory as "DB-Alpha-9".

doc_02_database_hosting.txt

Database Infrastructure Inventory
The transaction database "DB-Alpha-9" is configured with a high-availability active-passive clustering setup. The primary active node of DB-Alpha-9 is hosted on and powered by the physical hardware server labeled "Rack-Host-Mercury" located in our Oregon Data Center (Zone-A).

doc_03_hardware_rack.txt

Data Center Hardware Layout
The physical server "Rack-Host-Mercury" is mounted in Rack 12 of Zone-A. Power delivery to Rack-Host-Mercury is managed by the Intelligent Power Distribution Unit (PDU) identified as "PDU-West-03". Additionally, all network traffic for this server passes through the primary top-of-rack network switch "Switch-Nexus-X".

doc_04_maintenance_schedule.txt

Urgent Infrastructure Maintenance Bulletin
This weekend, the network operations team will perform an urgent hardware replacement. Due to recurring port failures, the top-of-rack network switch "Switch-Nexus-X" will be shut down and replaced with a newer model. This operation is scheduled for Sunday, June 14, at 02:00 AM UTC and will result in a temporary network blackout for all connected hardware under Switch-Nexus-X.

doc_05_unrelated_pdu_info.txt

Facility Power Grid Upgrades
The backup power distribution unit "PDU-West-04" is undergoing routine load testing. Please note that "PDU-West-03" (which powers adjacent server racks) is running at normal capacity and is not scheduled for any maintenance this month.

The key relationships within the document list are structured as follows:
- The service BILL-PROD depends on the database DB-Alpha-9.
- The database DB-Alpha-9 is hosted on Rack-Host-Mercury.
- Rack-Host-Mercury is located in the Oregon Data Center Zone-A.
- Rack-Host-Mercury relies on PDU-West-03.
- Rack-Host-Mercury relies on Switch-Nexus-X.
- Switch-Nexus-X is scheduled for maintenance on June 14.
- PDU-West-03 is operating normally.
- PDU-West-04 is undergoing load testing.
The table below shows which files contain these keywords.

For a question like "What services are affected by the maintenance on June 14?", the system needs to understand the relationships between the documents. The goal is for it to trace the connections—from Maintenance to Switch-Nexus-X, then to Rack-Host-Mercury, DB-Alpha-9, and finally BILL-PROD—and correctly conclude with the answer: "BILL-PROD."

4.2 Creating the Bedrock Knowledge Bases

I created two types of Bedrock Knowledge Bases. (Detailed step-by-step procedures are omitted here; only the key parameters are listed).
S3 Vectors Version
- Data source:
- S3 (The bucket containing the five document files)
- Parsing strategy: Amazon Bedrock default parser
- Chunking strategy: Default chunking
- Data storage and processing
- Embedding model: Titan Text Embedding V2
- Vector store: Amazon S3 Vectors
Neptune Analytics Version
- Data source:
- S3 (The bucket containing the five document files)
- Parsing strategy: Amazon Bedrock default parser
- Chunking strategy: Default chunking
- Data storage and processing
- Embedding model: Titan Text Embedding V2
- Vector store: Amazon Neptune Analytics (GraphRAG)
- Model for graph construction: Claude 4.5 Haiku

4.3 Validation

4.3.1 Initial Validation

I started by asking a few standard questions to observe the differences in behavior.
While I ran these tests directly in the AWS Management Console, they are equivalent to executing the following AWS CLI command. The LLM used is Claude 4.6 Sonnet, and the maximum number of retrieved chunks (numberOfResults) is set to 10.

aws bedrock-agent-runtime retrieve-and-generate
  --input "{\"text\": \"What user-facing services or applications will be affected by the maintenance scheduled on Sunday, June 14?\"}" 
  --retrieve-and-generate-configuration "{\"type\": \"KNOWLEDGE_BASE\", \"knowledgeBaseConfiguration\": {\"knowledgeBaseId\": \"XXXXXXXXXX\", \"modelArn\": \"arn:aws:bedrock:ap-northeast-1:XXXXXXXXXXXXX:inference-profile/global.anthropic.claude-sonnet-4-6\", \"retrievalConfiguration\": {\"vectorSearchConfiguration\": {\"numberOfResults\": 10}}}}"  
  --region ap-northeast-1

Questions	S3 Vectors	Neptune Analytics
What user-facing services or applications will be affected by the maintenance scheduled on Sunday, June 14?	Result: OK The maintenance on Sunday, June 14 involves shutting down the network switch "Switch-Nexus-X," which will cause a temporary network blackout for all hardware connected to it. One of those connected servers is "Rack-Host-Mercury."[1] [2] Rack-Host-Mercury is the physical server hosting the primary active node of the transaction database "DB-Alpha-9."[3]Since DB-Alpha-9 will lose network connectivity during the maintenance window, the Customer Billing Service (BILL-PROD) — which relies on a persistent, real-time connection to DB-Alpha-9 to process subscription renewals and retrieve ledger data — will be directly impacted and likely unable to function during that period.[4]	Result: OK The maintenance scheduled for Sunday, June 14 involves replacing the top-of-rack network switch "Switch-Nexus-X," which will cause a temporary network blackout for all hardware connected to it.[1]One of the servers connected to Switch-Nexus-X is "Rack-Host-Mercury," which handles all its network traffic through that switch.[2]Rack-Host-Mercury hosts the primary active node of the transaction database "DB-Alpha-9."[3] DB-Alpha-9 is the database that the core Customer Billing Service (Service-ID: BILL-PROD) relies on for processing payments and retrieving ledger data in real-time. Therefore, the Customer Billing Service — responsible for all subscription renewals — will be directly impacted by this maintenance window.[4]
Is there any maintenance scheduled for the power unit PDU-West-03?	Result: OK PDU-West-03 is not scheduled for any maintenance this month. It is currently running at normal capacity.[1]	Result: OK There is no maintenance scheduled for PDU-West-03 this month. It is currently running at normal capacity.[1]

There is no difference in the answers provided by S3 Vectors and Neptune Analytics; both answered correctly. In both Knowledge Bases, one file maps to one chunk, meaning there are 5 source chunks total. Because the maximum chunk retrieval limit was set to 10, both systems were able to fetch all 5 chunks to construct their answers, resulting in identical outcomes.

4.3.2 Restricting the Maximum Chunk Retrieval Limit

To accurately answer the question "What user-facing services or applications will be affected by the maintenance scheduled on Sunday, June 14?", documents #01 through #04 are required (document #05 is irrelevant and not needed).
I changed the maximum chunk retrieval limit to 4 to verify whether each Knowledge Base could select the appropriate chunks, and to see how the final answers would look based on that selection. (Executed the command from section 4.3.1 with numberOfResults changed from 10 to 4).

Questions	S3 Vectors	Neptune Analytics
What user-facing services or applications will be affected by the maintenance scheduled on Sunday, June 14?	[Result: OK] [Retrieved Docs: #01, #03, #04, #05] On Sunday, June 14 at 02:00 AM UTC, the network switch "Switch-Nexus-X" will be shut down and replaced due to recurring port failures. This will cause a temporary network blackout for all hardware connected to it.The physical server "Rack-Host-Mercury" routes all its network traffic through Switch-Nexus-X, meaning it will lose network connectivity during the maintenance window. The primary transaction database "DB-Alpha-9" is hosted on Rack-Host-Mercury (based on the infrastructure layout). The Customer Billing Service (Service-ID: BILL-PROD) maintains a persistent, high-throughput connection to DB-Alpha-9 to process payments and retrieve ledger data in real-time. Therefore, the Billing Application — which handles all subscription renewals — is the key user-facing service expected to be disrupted during this maintenance window.	[Result: NG] [Retrieved Docs: #03, #04, #05] The physical server "Rack-Host-Mercury" will be affected by the Sunday June 14 maintenance. Its network traffic runs through "Switch-Nexus-X", which is being shut down and replaced that night at 02:00 AM UTC, causing a temporary network blackout for all hardware connected to it. The power supply to Rack-Host-Mercury (via PDU-West-03) is not affected, as PDU-West-03 is running at normal capacity and has no maintenance scheduled this month. The outage will be network-only.

S3 Vectors selected documents #01, #03, #04, and #05. Although it selected an unnecessary chunk (#05) and missed document #02, it managed to infer the relationship from the other documents and (somehow) arrived at the correct answer.
Neptune Analytics selected documents #03, #04, and #05. Because it selected an unnecessary chunk (#05) and dropped a chunk absolutely vital for the answer (#01), the final response became inaccurate. Furthermore, despite the maximum retrieval count being set to 4, it only fetched 3 chunks. (The exact reason for only retrieving 3 chunks is unclear, but I speculate that chunks falling below a certain score threshold are excluded from retrieval).
Note that you can verify which chunks were selected—and their priority order—for each Knowledge Base using the section below.

aws bedrock-agent-runtime retrieve 
  --knowledge-base-id "XXXXXXXX" 
  --retrieval-query "{\"text\": \"What user-facing services or applications will be affected by the maintenance scheduled on Sunday, June 14?\"}" 
  --retrieval-configuration "{\"vectorSearchConfiguration\": {\"numberOfResults\": 4}}" 
  --region ap-northeast-1

Priority	S3 Vectors	Neptune Analytics
1	#04 (0.485)	#03 (1.898)
2	#05 (0.385)	#05 (1.612)
3	#01 (0.356)	#04 (1.050)
4	#03 (0.354)	None

Direct comparison is not possible because the score ranges for S3 Vectors and Neptune Analytics differ (the exact calculation methods are unknown).

4.3.3 Verifying Node and Edge Registration in Neptune Analytics

My current understanding is that with Neptune Analytics, the node and edge traversal results within the GraphDB might be influencing how chunks are selected.
Therefore, I will verify exactly how nodes and edges have been registered within the Neptune Analytics instance created for the Bedrock Knowledge Base.

aws neptune-graph execute-query 
  --graph-identifier g-XXXXXXXXX 
  --query-string "MATCH (n)-[r]->(m) RETURN n, type(r), m LIMIT 50" \
  --language open_cypher 
  --region ap-northeast-1 
  outfile.json

Here are the results from a visualization script (built with kiro) using the JSON file (list of nodes and edges) retrieved via the command above.

When you select Neptune Analytics as the vector DB in Bedrock Knowledge Base, the following nodes and edges are automatically created:
- Nodes
- Chunk: Chunks created for each source document (5 total). Example: doc_01.
  - Entity: Proper nouns, etc. (13 total). Example: DB-Alpha-9, Rack-Host-Mercury.
  - DocumentID: The source document (5 total), represented as a roughly 64-byte hash string.
- Edges
  - CONTAINS: Maps which Entity is contained in which Chunk (18 total). Example: doc_01 (Chunk) ---> DB-Alpha-9 (Entity).
  - FROM: Maps which Chunk was created from which source document (5 total). Example: doc_01 (Chunk) ---> xxxxxxxxxxxxxxxx (DocumentID).
Essentially, the system only registers which keywords appear in which documents. It did not automatically register semantic relationships between Entities (such as "DB-Alpha-9 is hosted on Rack-Host-Mercury"), which is what I had optimistically expected. Because of this, I speculate that chunk selection is not based on semantic relationships, but rather on traversing the connections between chunks that share the same Entities.

5. Takeaways / Conclusion

Through this evaluation, I learned how to configure Neptune Analytics within Bedrock Knowledge Base, but I wasn't able to demonstrate its distinct advantages over standard vector search.
As of June 2026, GraphRAG in Bedrock Knowledge Base does not appear to automatically extract semantic relationships between entities (such as hosted_on or depends_on). Instead, the mechanism seems to focus on traversing shared entity relationships across different chunks. Therefore, to leverage a true, full-scale knowledge graph for retrieval, it looks like we need to manually ingest entity relationships into Neptune. I'd love to tackle that approach in my next challenge.

Trying Out AWS VPC Encryption Control

Kentaro Matsumoto — Sun, 22 Mar 2026 06:08:28 +0000

1. Introduction

While reviewing recent AWS feature updates, I came across an article about "VPC Encryption Control." It was released in November 2025 and is set to become a paid feature starting March 2026.
I was curious about how exactly it "enforces" encryption, so I decided to test its behavior myself.

2. What is VPC Encryption Control? (My Understanding)

Initially, I wondered: "Does this mean all traffic within the VPC must be encrypted? Will it detect if I'm using SSH/HTTPS (OK) versus Telnet/HTTP (NG) by inspecting packets?"
As it turns out, that’s not quite how it works. Instead, it monitors or enforces whether resources within the VPC are using Nitro-based EC2 instances or RDS that support transparent encryption at the AWS infrastructure layer.

3. What I Did

Created a VPC with VPC Encryption Control enabled (Monitor mode).
Set up VPC Flow Logs with specific fields required to identify whether traffic is encrypted.
Verified how the following traffic patterns are judged by Encryption Control:

#	SRC	DST	Protocol
1	Local PC	nginx(t3.micro)	http
2	Local PC	nginx(m7i.large)	http
3	Local PC	nginx(t3.micro)	https
4	Local PC	nginx(m7i.large)	https
5	in-VPC curl client(t3.micro)	nginx(t3.micro)	http
6	in-VPC curl client(m7i.large)	nginx(m7i.large)	http
7	in-VPC curl client(t3.micro)	nginx(t3.micro)	https
8	in-VPC curl client(m7i.large)	nginx(m7i.large)	https

Switched the VPC Encryption Control mode to Enforce mode.

4. Architecture Diagram

5. Procedure

5.1 Creating a VPC with Encryption Control

Create a VPC with VPC Encryption Control enabled (start with Monitor mode). This can be specified simply during the VPC creation process.
Confirm that the created VPC has an Encryption Control ID and is set to Monitor mode.

5.2 Creating Test Instances

Launch two instances with nginx installed (t3.micro and m7i.large).
Configure nginx with a server certificate to accept HTTPS (Reference: "Automatic SSL Certificate Renewal on EC2 using ACM Exported Certificates(in Japanese)").
Launch two instances for the curl client (t3.micro and m7i.large).
Note: Not all Nitro-based instances support automatic encryption. There is a specific list of supported instance types. For example, while t3 is Nitro-based, it is not supported for this feature.

5.3 Creating VPC Flow Logs

To determine if the traffic is judged as "encrypted," configure VPC Flow Logs using a custom format that includes the ${encryption-status} field.

5.4 Test Traffic and Results

Run curl from the local PC and the in-VPC instances to the nginx servers.
Example commands:

> curl http://x.x.x.x
> curl -k https://x.x.x.x (using -k to skip certificate validation when accessing via IP)

Results for the encryption-status field:
- 0: Not encrypted at the infrastructure layer.
- 1: Encrypted by the Nitro hardware.

#	SRC	DST	Protocol	Result
1	Local PC	nginx(t3.micro)	http	0
2	Local PC	nginx(m7i.large)	http	0
3	Local PC	nginx(t3.micro)	https	0
4	Local PC	nginx(m7i.large)	https	0
5	in-VPC curl client(t3.micro)	nginx(t3.micro)	http	0
6	in-VPC curl client(m7i.large)	nginx(m7i.large)	http	1
7	in-VPC curl client(t3.micro)	nginx(t3.micro)	https	0
8	in-VPC curl client(m7i.large)	nginx(m7i.large)	https	1

Key Takeaway:
- Only traffic between two supported Nitro instances is flagged as 1.
- Even if you use HTTPS, if the underlying infrastructure doesn't support the Nitro-level encryption, the VPC Encryption Control check does not consider it "encrypted."

5.5 Switching to Enforce Mode

To switch to Enforce mode, you must address any non-compliant resources. This includes the Internet Gateway and any non-compatible ENIs (like those belonging to the t3.micro).
By upgrading instances to m7i.large and setting exclusion rules for the Internet Gateway, you can successfully enable Enforce mode.

6. Reference Articles

Official AWS Blog: Provides a solid overview of the feature (in Japanese).

https://aws.amazon.com/jp/blogs/news/introducing-vpc-encryption-controls-enforce-encryption-in-transit-within-and-across-vpcs-in-a-region/

Deep Dive Verification: An article exploring what happens when you switch from Monitor to Enforce mode (in Japanese).

https://persol-serverworks.co.jp/blog/vpc/vpcvpc.html

7. Final Thoughts

While I don't see myself using this for my current systems anytime soon, I was impressed by the Nitro system's ability to transparently encrypt all inter-instance traffic. It's a powerful tool for high-compliance environments.

Trying out Amazon CloudWatch Network Flow Monitor in EKS

Kentaro Matsumoto — Tue, 25 Nov 2025 14:20:25 +0000

1. Introduction

The Amazon CloudWatch Network Flow Monitor service, which can monitor the communication status between resources within AWS, was released in December 2024.
This time, we will confirm the setup procedure and usability of the EKS version (where the agent runs as a DaemonSet).

2. What We Did

Prepare a barebones EKS cluster.
Add the Network Flow Monitor (for EKS) add-on to the EKS cluster.
Configure the Network Flow Monitor "monitors".
Access an Nginx pod launched inside EKS from an external client and verify that Network Flow Monitor metrics are collected.
Introduce packet loss to one of the Nginx pods and confirm that the Network Flow Monitor metrics change.

3. Architecture Diagram

4. Configuration Steps

4.1 Pre-environment Setup

We will build a VPC and EKS cluster for this evaluation (detailed steps omitted). This time, we use the Management Console with mostly default settings.

The k8s version is 1.33.
We prepared two t3.medium worker nodes as the node group.
The add-on "Amazon EKS Pod Identity Agent" was added (necessary for Network Flow Monitor; automatically added by default).
- The pod status after environment construction is as follows:

[ec2-user@ip-10-0-0-60 mysample]$ kubectl get pod -A -o wide
NAMESPACE      NAME                              READY   STATUS    RESTARTS   AGE   IP            NODE                                             NOMINATED NODE   READINESS GATES
external-dns   external-dns-754cf78755-ks8nc     1/1     Running   0          19h   10.0.10.131   ip-10-0-10-159.ap-northeast-3.compute.internal   <none>           <none>
kube-system    aws-node-bwshx                    2/2     Running   0          19h   10.0.11.7     ip-10-0-11-7.ap-northeast-3.compute.internal     <none>           <none>
kube-system    aws-node-ckvdl                    2/2     Running   0          19h   10.0.10.159   ip-10-0-10-159.ap-northeast-3.compute.internal   <none>           <none>
kube-system    coredns-bdbfddcf5-54sbq           1/1     Running   0          19h   10.0.10.13    ip-10-0-10-159.ap-northeast-3.compute.internal   <none>           <none>
kube-system    coredns-bdbfddcf5-zvdr9           1/1     Running   0          19h   10.0.10.115   ip-10-0-10-159.ap-northeast-3.compute.internal   <none>           <none>
kube-system    eks-node-monitoring-agent-ltzc9   1/1     Running   0          19h   10.0.10.159   ip-10-0-10-159.ap-northeast-3.compute.internal   <none>           <none>
kube-system    eks-node-monitoring-agent-nckx7   1/1     Running   0          19h   10.0.11.7     ip-10-0-11-7.ap-northeast-3.compute.internal     <none>           <none>
kube-system    eks-pod-identity-agent-jz6kc      1/1     Running   0          19h   10.0.11.7     ip-10-0-11-7.ap-northeast-3.compute.internal     <none>           <none>
kube-system    eks-pod-identity-agent-khq8q      1/1     Running   0          19h   10.0.10.159   ip-10-0-10-159.ap-northeast-3.compute.internal   <none>           <none>
kube-system    kube-proxy-569w9                  1/1     Running   0          19h   10.0.10.159   ip-10-0-10-159.ap-northeast-3.compute.internal   <none>           <none>
kube-system    kube-proxy-cm94v                  1/1     Running   0          19h   10.0.11.7     ip-10-0-11-7.ap-northeast-3.compute.internal     <none>           <none>
kube-system    metrics-server-fdccf8449-2b2sj    1/1     Running   0          19h   10.0.10.110   ip-10-0-10-159.ap-northeast-3.compute.internal   <none>           <none>
kube-system    metrics-server-fdccf8449-5584h    1/1     Running   0          19h   10.0.10.55    ip-10-0-10-159.ap-northeast-3.compute.internal   <none>           <none>

4.2 Adding the Network Flow Monitor Add-on

We add the Network Flow Monitor add-on using the Management Console. The official procedure is "Install the EKS AWS Network Flow Monitor Agent add-on."

From the Add-ons section of the constructed EKS cluster, select "Get more add-ons."

Select the AWS Network Flow Monitor Agent.

Create the necessary IAM role to be attached to the Network Flow Monitor Agent pods by selecting "Create Recommended Role."

Create the IAM role with the default settings and configure it as the role to be attached to the pods.

After being added as an add-on, confirm that it is running as a DaemonSet:

[ec2-user@ip-10-0-0-60 mysample]$ kubectl get daemonsets -A
NAMESPACE                     NAME                             DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
amazon-network-flow-monitor   aws-network-flow-monitor-agent   2         2         2       2            2           kubernetes.io/os=linux   75s
kube-system                   aws-node                         2         2         2       2            2           <none>                   19h
kube-system                   dcgm-server                      0         0         0       0            0           kubernetes.io/os=linux   19h
kube-system                   eks-node-monitoring-agent        2         2         2       2            2           kubernetes.io/os=linux   19h
kube-system                   eks-pod-identity-agent           2         2         2       2            2           <none>                   19h
kube-system                   kube-proxy                       2         2         2       2            2           <none>                   19h
[ec2-user@ip-10-0-0-60 mysample]$ kubectl get pod -A -o wide
NAMESPACE                     NAME                                   READY   STATUS    RESTARTS   AGE   IP            NODE                                             NOMINATED NODE   READINESS GATES
amazon-network-flow-monitor   aws-network-flow-monitor-agent-7v24v   1/1     Running   0          64s   10.0.11.7     ip-10-0-11-7.ap-northeast-3.compute.internal     <none>           <none>
amazon-network-flow-monitor   aws-network-flow-monitor-agent-rpqr6   1/1     Running   0          64s   10.0.10.159   ip-10-0-10-159.ap-northeast-3.compute.internal   <none>           <none>
... (other pods)

4.3 Configuring the Network Flow Monitor "Monitors"

We create three "monitors": for the entire VPC, for the AZ-3a side of the VPC, and for the AZ-3b side of the VPC.

From CloudWatch - Flow Monitors, select "Create Monitor."

Create a monitor targeting the entire EKS VPC.

Create a monitor selecting the AZ-3a subnet of the EKS VPC (and similarly for AZ-3b).

4.4 Preparing Nginx

Preparing Nginx with tc

To introduce packet loss to a pod later, we prepare an Nginx container image that can use the tc (Traffic Control) command and register it in ECR (steps omitted). The Dockerfile is as follows:

# Use the official Nginx image as the base image
FROM nginx:alpine

# Install the iproute2 package, which includes the tc command
RUN apk update && apk add iproute2

# Start nginx when the container launches
CMD ["nginx", "-g", "daemon off;"]

Deploying Nginx

We deploy Nginx so that one pod runs on each of the two worker nodes and expose the HTTP port externally. The NET_ADMIN capability is required to run the tc command. We use AntiAffinity to prevent both pods from launching on the same worker node.

kind: Deployment
metadata:
  name: mynginx-with-tc-deployment
spec:
  replicas: 2
  selector:
    matchLabels:
      app: mynginx-with-tc
  template:
    metadata:
      labels:
        app: mynginx-with-tc
    spec:
      containers:
      - name: mynginx-with-tc-container
        image: xxxxxxxxxxxx.dkr.ecr.ap-northeast-3.amazonaws.com/mksamba/mynginx-with-tc-repo:latest
        ports:
        - containerPort: 80
        securityContext:
          capabilities:
            add: ["NET_ADMIN"]
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - mynginx-with-tc
            topologyKey: "kubernetes.io/hostname"
---
apiVersion: v1
kind: Service
metadata:
  name: mynginx-with-tc-service
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing

spec:
  type: LoadBalancer
  selector:
    app: mynginx-with-tc
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80

Confirm that pods are running on each worker node:

[ec2-user@ip-10-0-0-60 mysample]$ kubectl apply -f mynginx-with-tc.yaml 
deployment.apps/mynginx-with-tc-deployment created
service/mynginx-with-tc-service created

[ec2-user@ip-10-0-0-60 mysample]$ kubectl get pod -A -o wide | grep nginx
default                       mynginx-with-tc-deployment-68cb4fff79-qjw9q   1/1     Running   0          71s   10.0.10.8     ip-10-0-10-159.ap-northeast-3.compute.internal   <none>           <none>
default                       mynginx-with-tc-deployment-68cb4fff79-tfc8s   1/1

4.5 Accessing Nginx from an External Client

We access Nginx via the CLB (Classic Load Balancer) about 10,000 times using curl from the Internet. Traffic is distributed to the two pods by the CLB.

#!/bin/bash
# Specify the target URL
URL="http://xxxxxxxxxx.ap-northeast-3.elb.amazonaws.com"

# Loop
for ((i=1; i<=10000; i++))
do
  echo "Request #$i"
  curl -o /dev/null -s -w "%{http_code}\n" "$URL"
done

4.6 Introducing Packet Loss to Nginx

We introduce a 3% packet loss to only one pod (the AZ-3b side).

# Identify the pod name to configure
[ec2-user@ip-10-0-0-60 ~]$ kubectl get pod -A -o wide |grep mynginx
default                       mynginx-with-tc-deployment-68cb4fff79-qjw9q   1/1     Running   0          27m   10.0.10.8     ip-10-0-10-159.ap-northeast-3.compute.internal   <none>           <none>
default                       mynginx-with-tc-deployment-68cb4fff79-tfc8s   1/1     Running   0          27m   10.0.11.191   ip-10-0-11-7.ap-northeast-3.compute.internal     <none>           <none>

# Introduce packet loss to the AZ-3b side pod
[ec2-user@ip-10-0-0-60 ~]$ kubectl exec -it mynginx-with-tc-deployment-68cb4fff79-tfc8s -- tc qdisc add dev eth0 root netem loss 3%

# (Reference) Command to revert the packet loss setting
[ec2-user@ip-10-0-0-60 ~]$ kubectl exec -it mynginx-with-tc-deployment-68cb4fff79-tfc8s -- tc qdisc del dev eth0 root

4.7 Checking the Network Flow Monitor "Monitors"

We check the monitor values during normal operation and when packet loss is introduced to one pod.
Around 21:20PM is normal operation, and around 21:40PM is the traffic with packet loss introduced.

VPC-wide Monitor: Around 21:40PM, retransmissions are occurring.

AZ-3a Monitor: Traffic is half of the VPC-wide total, but since the pod is normal, there are no retransmissions.

AZ-3b Monitor: Traffic is half of the VPC-wide total, and around 21:40PM, a large amount of retransmissions occurs due to the packet loss in the pod.

In this example, using the Network Flow Monitor, we can proceed with investigation like this: "Increased retransmissions across the entire VPC" -> "No issue on the AZ-3a side" -> "Retransmissions only on the AZ-3b side" -> "Network health Indicator for AZ-3b is Healthy, so it's not an AWS infrastructure issue" -> "Perhaps an anomaly within the user's scope of responsibility, such as the AZ-3b worker node or pod?"

5. Impressions

The setup process was extremely easy, simply adding the add-on via the Management Console.
This time, we confirmed the metric difference by introducing packet loss to generate retransmissions, but we want to consider how this service can improve our level of monitoring going forward.