1. Introduction
- My initial understanding was that OpenSearch or S3 Vectors would be the go-to databases when building a RAG system with Bedrock Knowledge Base. However, after hearing that GraphRAG might be a better fit for certain use cases, I decided to dive into GraphRAG using Neptune Analytics to explore the differences.
2. What I Did
- Prepared a list of documents where using GraphRAG is likely to improve search and retrieval accuracy.
- Created two Bedrock Knowledge Bases (an S3 Vectors version and a Neptune Analytics version) using the exact same source documents.
- Submitted queries to both and compared the accuracy and quality of the responses between the S3 Vectors and Neptune Analytics versions.
3. Architecture Diagram
4. Procedure
4.1 Preparing the Document List
- I had Gemini generate a list of documents that share relationships with one another, and saved them to S3. The following five documents were used:
doc_01_billing_service.txt
Internal Service Architecture: Billing Application
The company's core Customer Billing Service (Service-ID: BILL-PROD) is designed as a microservice to handle all subscription renewals. To process payments and retrieve ledger data in real-time, this Billing Application establishes a persistent, high-throughput connection to the primary transaction database engine, known in our inventory as "DB-Alpha-9".
doc_02_database_hosting.txt
Database Infrastructure Inventory
The transaction database "DB-Alpha-9" is configured with a high-availability active-passive clustering setup. The primary active node of DB-Alpha-9 is hosted on and powered by the physical hardware server labeled "Rack-Host-Mercury" located in our Oregon Data Center (Zone-A).
doc_03_hardware_rack.txt
Data Center Hardware Layout
The physical server "Rack-Host-Mercury" is mounted in Rack 12 of Zone-A. Power delivery to Rack-Host-Mercury is managed by the Intelligent Power Distribution Unit (PDU) identified as "PDU-West-03". Additionally, all network traffic for this server passes through the primary top-of-rack network switch "Switch-Nexus-X".
doc_04_maintenance_schedule.txt
Urgent Infrastructure Maintenance Bulletin
This weekend, the network operations team will perform an urgent hardware replacement. Due to recurring port failures, the top-of-rack network switch "Switch-Nexus-X" will be shut down and replaced with a newer model. This operation is scheduled for Sunday, June 14, at 02:00 AM UTC and will result in a temporary network blackout for all connected hardware under Switch-Nexus-X.
doc_05_unrelated_pdu_info.txt
Facility Power Grid Upgrades
The backup power distribution unit "PDU-West-04" is undergoing routine load testing. Please note that "PDU-West-03" (which powers adjacent server racks) is running at normal capacity and is not scheduled for any maintenance this month.
-
The key relationships within the document list are structured as follows:
- The service BILL-PROD depends on the database DB-Alpha-9.
- The database DB-Alpha-9 is hosted on Rack-Host-Mercury.
- Rack-Host-Mercury is located in the Oregon Data Center Zone-A.
- Rack-Host-Mercury relies on PDU-West-03.
- Rack-Host-Mercury relies on Switch-Nexus-X.
- Switch-Nexus-X is scheduled for maintenance on June 14.
- PDU-West-03 is operating normally.
- PDU-West-04 is undergoing load testing.
The table below shows which files contain these keywords.
- For a question like "What services are affected by the maintenance on June 14?", the system needs to understand the relationships between the documents. The goal is for it to trace the connections—from Maintenance to Switch-Nexus-X, then to Rack-Host-Mercury, DB-Alpha-9, and finally BILL-PROD—and correctly conclude with the answer: "BILL-PROD."
4.2 Creating the Bedrock Knowledge Bases
I created two types of Bedrock Knowledge Bases. (Detailed step-by-step procedures are omitted here; only the key parameters are listed).
-
S3 Vectors Version
- Data source:
- S3 (The bucket containing the five document files)
- Parsing strategy: Amazon Bedrock default parser
- Chunking strategy: Default chunking
- Data storage and processing
- Embedding model: Titan Text Embedding V2
- Vector store: Amazon S3 Vectors
-
Neptune Analytics Version
- Data source:
- S3 (The bucket containing the five document files)
- Parsing strategy: Amazon Bedrock default parser
- Chunking strategy: Default chunking
- Data storage and processing
- Embedding model: Titan Text Embedding V2
- Vector store: Amazon Neptune Analytics (GraphRAG)
- Model for graph construction: Claude 4.5 Haiku
4.3 Validation
4.3.1 Initial Validation
- I started by asking a few standard questions to observe the differences in behavior.
- While I ran these tests directly in the AWS Management Console, they are equivalent to executing the following AWS CLI command. The LLM used is Claude 4.6 Sonnet, and the maximum number of retrieved chunks (numberOfResults) is set to 10.
aws bedrock-agent-runtime retrieve-and-generate
--input "{\"text\": \"What user-facing services or applications will be affected by the maintenance scheduled on Sunday, June 14?\"}"
--retrieve-and-generate-configuration "{\"type\": \"KNOWLEDGE_BASE\", \"knowledgeBaseConfiguration\": {\"knowledgeBaseId\": \"XXXXXXXXXX\", \"modelArn\": \"arn:aws:bedrock:ap-northeast-1:XXXXXXXXXXXXX:inference-profile/global.anthropic.claude-sonnet-4-6\", \"retrievalConfiguration\": {\"vectorSearchConfiguration\": {\"numberOfResults\": 10}}}}"
--region ap-northeast-1
| Questions | S3 Vectors | Neptune Analytics |
|---|---|---|
| What user-facing services or applications will be affected by the maintenance scheduled on Sunday, June 14? | Result: OK The maintenance on Sunday, June 14 involves shutting down the network switch "Switch-Nexus-X," which will cause a temporary network blackout for all hardware connected to it. One of those connected servers is "Rack-Host-Mercury."[1] [2] Rack-Host-Mercury is the physical server hosting the primary active node of the transaction database "DB-Alpha-9."[3]Since DB-Alpha-9 will lose network connectivity during the maintenance window, the Customer Billing Service (BILL-PROD) — which relies on a persistent, real-time connection to DB-Alpha-9 to process subscription renewals and retrieve ledger data — will be directly impacted and likely unable to function during that period.[4] |
Result: OK The maintenance scheduled for Sunday, June 14 involves replacing the top-of-rack network switch "Switch-Nexus-X," which will cause a temporary network blackout for all hardware connected to it.[1]One of the servers connected to Switch-Nexus-X is "Rack-Host-Mercury," which handles all its network traffic through that switch.[2]Rack-Host-Mercury hosts the primary active node of the transaction database "DB-Alpha-9."[3] DB-Alpha-9 is the database that the core Customer Billing Service (Service-ID: BILL-PROD) relies on for processing payments and retrieving ledger data in real-time. Therefore, the Customer Billing Service — responsible for all subscription renewals — will be directly impacted by this maintenance window.[4] |
| Is there any maintenance scheduled for the power unit PDU-West-03? | Result: OK PDU-West-03 is not scheduled for any maintenance this month. It is currently running at normal capacity.[1] |
Result: OK There is no maintenance scheduled for PDU-West-03 this month. It is currently running at normal capacity.[1] |
- There is no difference in the answers provided by S3 Vectors and Neptune Analytics; both answered correctly. In both Knowledge Bases, one file maps to one chunk, meaning there are 5 source chunks total. Because the maximum chunk retrieval limit was set to 10, both systems were able to fetch all 5 chunks to construct their answers, resulting in identical outcomes.
4.3.2 Restricting the Maximum Chunk Retrieval Limit
To accurately answer the question "What user-facing services or applications will be affected by the maintenance scheduled on Sunday, June 14?", documents #01 through #04 are required (document #05 is irrelevant and not needed).
I changed the maximum chunk retrieval limit to 4 to verify whether each Knowledge Base could select the appropriate chunks, and to see how the final answers would look based on that selection. (Executed the command from section 4.3.1 with numberOfResults changed from 10 to 4).
| Questions | S3 Vectors | Neptune Analytics |
|---|---|---|
| What user-facing services or applications will be affected by the maintenance scheduled on Sunday, June 14? | [Result: OK] [Retrieved Docs: #01, #03, #04, #05] On Sunday, June 14 at 02:00 AM UTC, the network switch "Switch-Nexus-X" will be shut down and replaced due to recurring port failures. This will cause a temporary network blackout for all hardware connected to it.The physical server "Rack-Host-Mercury" routes all its network traffic through Switch-Nexus-X, meaning it will lose network connectivity during the maintenance window. The primary transaction database "DB-Alpha-9" is hosted on Rack-Host-Mercury (based on the infrastructure layout). The Customer Billing Service (Service-ID: BILL-PROD) maintains a persistent, high-throughput connection to DB-Alpha-9 to process payments and retrieve ledger data in real-time. Therefore, the Billing Application — which handles all subscription renewals — is the key user-facing service expected to be disrupted during this maintenance window. |
[Result: NG] [Retrieved Docs: #03, #04, #05] The physical server "Rack-Host-Mercury" will be affected by the Sunday June 14 maintenance. Its network traffic runs through "Switch-Nexus-X", which is being shut down and replaced that night at 02:00 AM UTC, causing a temporary network blackout for all hardware connected to it. The power supply to Rack-Host-Mercury (via PDU-West-03) is not affected, as PDU-West-03 is running at normal capacity and has no maintenance scheduled this month. The outage will be network-only. |
S3 Vectors selected documents #01, #03, #04, and #05. Although it selected an unnecessary chunk (#05) and missed document #02, it managed to infer the relationship from the other documents and (somehow) arrived at the correct answer.
Neptune Analytics selected documents #03, #04, and #05. Because it selected an unnecessary chunk (#05) and dropped a chunk absolutely vital for the answer (#01), the final response became inaccurate. Furthermore, despite the maximum retrieval count being set to 4, it only fetched 3 chunks. (The exact reason for only retrieving 3 chunks is unclear, but I speculate that chunks falling below a certain score threshold are excluded from retrieval).
Note that you can verify which chunks were selected—and their priority order—for each Knowledge Base using the section below.
aws bedrock-agent-runtime retrieve
--knowledge-base-id "XXXXXXXX"
--retrieval-query "{\"text\": \"What user-facing services or applications will be affected by the maintenance scheduled on Sunday, June 14?\"}"
--retrieval-configuration "{\"vectorSearchConfiguration\": {\"numberOfResults\": 4}}"
--region ap-northeast-1
| Priority | S3 Vectors | Neptune Analytics |
|---|---|---|
| 1 | #04 (0.485) | #03 (1.898) |
| 2 | #05 (0.385) | #05 (1.612) |
| 3 | #01 (0.356) | #04 (1.050) |
| 4 | #03 (0.354) | None |
- Direct comparison is not possible because the score ranges for S3 Vectors and Neptune Analytics differ (the exact calculation methods are unknown).
4.3.3 Verifying Node and Edge Registration in Neptune Analytics
- My current understanding is that with Neptune Analytics, the node and edge traversal results within the GraphDB might be influencing how chunks are selected.
- Therefore, I will verify exactly how nodes and edges have been registered within the Neptune Analytics instance created for the Bedrock Knowledge Base.
aws neptune-graph execute-query
--graph-identifier g-XXXXXXXXX
--query-string "MATCH (n)-[r]->(m) RETURN n, type(r), m LIMIT 50" \
--language open_cypher
--region ap-northeast-1
outfile.json
- Here are the results from a visualization script (built with kiro) using the JSON file (list of nodes and edges) retrieved via the command above.
-
When you select Neptune Analytics as the vector DB in Bedrock Knowledge Base, the following nodes and edges are automatically created:
- Nodes
- Chunk: Chunks created for each source document (5 total). Example: doc_01.
- Entity: Proper nouns, etc. (13 total). Example: DB-Alpha-9, Rack-Host-Mercury.
- DocumentID: The source document (5 total), represented as a roughly 64-byte hash string.
- Edges
- CONTAINS: Maps which Entity is contained in which Chunk (18 total). Example: doc_01 (Chunk) ---> DB-Alpha-9 (Entity).
- FROM: Maps which Chunk was created from which source document (5 total). Example: doc_01 (Chunk) ---> xxxxxxxxxxxxxxxx (DocumentID).
Essentially, the system only registers which keywords appear in which documents. It did not automatically register semantic relationships between Entities (such as "DB-Alpha-9 is hosted on Rack-Host-Mercury"), which is what I had optimistically expected. Because of this, I speculate that chunk selection is not based on semantic relationships, but rather on traversing the connections between chunks that share the same Entities.
5. Takeaways / Conclusion
- Through this evaluation, I learned how to configure Neptune Analytics within Bedrock Knowledge Base, but I wasn't able to demonstrate its distinct advantages over standard vector search.
- As of June 2026, GraphRAG in Bedrock Knowledge Base does not appear to automatically extract semantic relationships between entities (such as hosted_on or depends_on). Instead, the mechanism seems to focus on traversing shared entity relationships across different chunks. Therefore, to leverage a true, full-scale knowledge graph for retrieval, it looks like we need to manually ingest entity relationships into Neptune. I'd love to tackle that approach in my next challenge.
Top comments (0)