DEV Community: Alexey Vidanov

Language Aggregation in OpenSearch: Selecting One Document Per Group by Language Preference

Alexey Vidanov — Wed, 26 Nov 2025 11:04:54 +0000

Multilingual content is common in documentation systems, product catalogs, and knowledge bases. When the same item exists in several languages, search results often become cluttered with multiple versions of the same document.

A typical requirement is to return one document per content group, chosen using a language preference order such as de > en > fr.

This blog post presents a practical pattern for handling language aggregation. The approach is part of the open-source OpenSearch project and is fully supported in Amazon OpenSearch Service, making it suitable for both self-managed clusters and AWS-managed environments.

The Problem

If an article exists in German, English, and French, a standard search will return all three. You want:

One hit per crossLanguageGroup
The language with the highest user preference
Deterministic, predictable selection

Simple deduplication does not work because you must apply a ranking rule across the group.

Solution Overview

The solution relies on three capabilities:

Field Collapse Groups all translations of the same document.
Scripted Sort Applies an explicit language ranking.
Keyword Fields Enable efficient sorting and scripting on language arrays.

Workflow

Figure: Multi-language document search workflow using collapse functionality. The process reduces 6 duplicate documents across German, English, and French to 3 results by grouping cross-language versions and applying language preference ranking.

Index Setup

PUT tmp_multi_lang
{
  "mappings": {
    "properties": {
      "crossLanguageGroup": {
        "type": "text",
        "fields": { "keyword": { "type": "keyword" }}
      },
      "languages": {
        "type": "text",
        "fields": { "keyword": { "type": "keyword" }}
      },
      "title": {
        "type": "text",
        "fields": { "keyword": { "type": "keyword" }}
      },
      "content": { "type": "text" }
    }
  }
}

Notes:

crossLanguageGroup stores the logical ID shared by all translations of the same item. Every language variant uses the same value, so collapse can group them reliably.
languages is an array of ISO language codes (e.g., ["de", "en"]). Using an array lets the script evaluate multiple languages if needed.
.keyword fields are essential because text fields are analyzed. The analyzer splits or lowercases values, which breaks exact matching and makes sorting impossible.
The .keyword subfield stores the raw, untouched value, enabling:
- deterministic sorting
- exact matches (e.g., term queries)
- using values inside Painless scripts

Without .keyword, collapse and scripted sorting would not work correctly.

Sample Data

Load with POST tmp_multi_lang/_bulk using NDJSON:

{ "index": {} }
{ "crossLanguageGroup": "abc123", "languages": ["de"], "title": "Willkommen", "content": "Dies ist eine Einführung. Source: Reply" }
{ "index": {} }
{ "crossLanguageGroup": "abc123", "languages": ["en"], "title": "Welcome", "content": "This is an introduction. Source: Reply" }
{ "index": {} }
{ "crossLanguageGroup": "abc123", "languages": ["fr"], "title": "Bienvenue", "content": "Ceci est une introduction. Source: Reply" }

{ "index": {} }
{ "crossLanguageGroup": "xyz789", "languages": ["en"], "title": "Search", "content": "How to search in OpenSearch. Source: Reply" }
{ "index": {} }
{ "crossLanguageGroup": "xyz789", "languages": ["fr"], "title": "Recherche", "content": "Comment chercher dans OpenSearch. Source: Reply" }

{ "index": {} }
{ "crossLanguageGroup": "def456", "languages": ["de"], "title": "Produktübersicht", "content": "Dies ist eine Produktbeschreibung." }
{ "index": {} }
{ "crossLanguageGroup": "def456", "languages": ["en"], "title": "Product Overview", "content": "This is a product description." }

The Core Query

POST tmp_multi_lang/_search
{
  "query": {
    "match": { "content": "Reply" }
  },
  "collapse": {
    "field": "crossLanguageGroup.keyword"
  },
  "sort": [
    {
      "_script": {
        "type": "number",
        "order": "asc",
        "script": {
          "lang": "painless",
          "params": {
            "lang_order": { "de": 0, "en": 1, "fr": 2 }
          },
          "source": """
            int best = 100;
            def order = params.lang_order;
            if (doc.containsKey('languages.keyword')) {
              for (def l : doc['languages.keyword']) {
                if (order.containsKey(l)) {
                  int ord = (int) order.get(l);
                  if (ord < best) { best = ord; }
                }
              }
            }
            return best;
          """
        }
      }
    },
    { "_score": "desc" }
  ]
}

How It Works

Query Stage

Matches all documents containing "Reply".

Collapse Stage

Groups documents by crossLanguageGroup.keyword.

Script Sort Stage

Iterates the languages array
Checks each language in the priority map
Selects the lowest value (best match)
Uses 100 as fallback

Tie Breaking

If two documents share the same priority, _score decides.

Example Result

abc123/de, abc123/en, abc123/fr, xyz789/en, xyz789/fr, klm654/de

After collapse with de > en > fr:

abc123/de, xyz789/en, klm654/de

Why This Works

Keyword fields expose doc values, making sorting fast and predictable.
Scripted sort applies a strict language hierarchy, not a soft boost.
Collapse guarantees exactly one document per crossLanguageGroup.
Painless correctly handles multi-value arrays like languages.

Everything works together to deliver deterministic, language-aware selection.

Common Errors and Fixes

1. "Text fields are not optimised for operations"

Use .keyword:

doc['languages.keyword']

2. "unknown field [lang]"

lang must be inside the script object.

3. Casting errors

Use explicit casting:

int ord = (int) order.get(l);

4. "Illegal list shortcut value [values]"

Iterate normally:

for (def l : doc['languages.keyword']) { ... }

Alternative Approach: Score-Based Selection

This method uses query-time boosts to encourage certain languages rather than enforcing a strict order. Each language gets a different weight: German +3, English +2, French +1.

When OpenSearch calculates the score, documents that match higher-boosted languages naturally rise to the top.

After scoring, collapse picks the highest-scoring document per crossLanguageGroup.

What this means in practice

If a group has de, en, and fr, the German version usually wins because it has the highest boost.
But if the English document has stronger text relevance, its score may exceed the German one.
The boosts add to the full-text score, so the effect is soft preference, not a strict ranking.

Good fit: simple setups where speed matters and minor inconsistencies are acceptable.

Not ideal: cases requiring deterministic de > en > fr without exceptions.

POST tmp_multi_lang/_search
{
  "query": {
    "bool": {
      "must": { "match": { "content": "Reply" } },
      "should": [
        { "term": { "languages.keyword": { "value": "de", "boost": 3 } } },
        { "term": { "languages.keyword": { "value": "en", "boost": 2 } } },
        { "term": { "languages.keyword": { "value": "fr", "boost": 1 } } }
      ]
    }
  },
  "collapse": { "field": "crossLanguageGroup.keyword" },
  "sort": [ { "_score": "desc" } ]
}

Pros: fast and simple

Cons: boosts are additive, not strict priority

Conclusion

Language-aware document aggregation in OpenSearch is solved cleanly by combining collapse, a Painless script-based sort, and keyword-backed language fields. Script sorting provides reliable, deterministic selection, while score-based boosting offers a faster but less strict alternative.

This pattern is useful anywhere multilingual data creates noise in search results. By grouping documents, applying a clear language hierarchy, and keeping sorting deterministic, teams can deliver cleaner UX across documentation portals, product catalogs, and knowledge bases. It also works well with personalization and dynamic language preferences.

If you want to build cleaner multilingual search or explore how to apply language-aware ranking in Amazon OpenSearch Service, feel free to reach out.

At Reply, we help teams design scalable, predictable, and user-centred search workflows — from proof of concept to fully aligned production deployments.

Building Perceptual Color Similarity Search with Amazon OpenSearch Service

Alexey Vidanov — Thu, 09 Oct 2025 09:32:01 +0000

Introduction

Traditional keyword search fails for color matching. A customer searching for "burgundy" won't find "wine red" or "maroon," even though these colors are visually almost identical. The problem goes beyond vocabulary: human color perception is far richer than our limited naming system. While the human eye can distinguish millions of shades, we use only a few hundred common color names. Most colors exist in the unnamed spaces between "navy" and "royal blue," or "burgundy" and "crimson."

Simple RGB (Red, Green, Blue) distance calculations make this gap even wider. Two colors with nearly identical RGB values can appear very different, while visually similar ones may be far apart numerically. Because RGB describes how screens display color rather than how humans perceive it, it fails to recognize real-world similarities, especially when lighting or device conditions change.

To close this gap, we should switch from RGB to CIELAB, a color space designed to align with human vision. LAB describes color in terms of lightness and opponent color channels (green to red, blue to yellow), creating distances that reflect perceptual differences. This makes it ideal for comparing colors under varying lighting, shadows, or image quality.

We applied this approach in counterfeit detection. By indexing garments' colors in LAB and monitoring marketplace images, we detected suspicious listings where the perceptual distance ΔE exceeded a tuned threshold (ΔE > 15). Combined with metadata and text analysis, this reduced false positives and cut manual review workload in our proof of concept.

This article demonstrates how to build a production-ready perceptual color similarity search using Amazon OpenSearch Service with k-nearest neighbor (k-NN) capabilities and the CIELAB color space, a combination that enables systems to see color the way humans do.

Why RGB Distance Fails

RGB (Red, Green, Blue) is built for displaying color on screens, not for measuring how similar two colors look. Distances in RGB space often disagree with human perception.

Consider two pairs with the same RGB distance:

Example 1: Same distance, very different perception

Dark blue RGB(30, 30, 60) vs olive RGB(60, 60, 30)
Euclidean distance: 52
Human perception: colors are completely different (ΔE ≈ 25)

Example 2: Same distance, nearly identical perception

Dark red RGB(200, 100, 100) vs light red RGB(230, 130, 130)
Euclidean distance: 52
Human perception: colors are similar (ΔE ≈ 7)

The problem
Identical numerical distances can produce opposite visual outcomes. RGB distance does not predict how people see color differences because brightness and hue interactions matter far more than simple channel-wise arithmetic.

Why this happens
RGB treats red, green, and blue as independent, equally weighted axes. Human vision does not. Our eyes respond nonlinearly to brightness (greater sensitivity in darker ranges) and encode color through opponent channels (red vs green, blue vs yellow). As a result, equal RGB distances rarely correspond to equal perceptual differences.

The Solution: CIELAB Color Space

To align computer vision with human perception, we need a different color space. CIELAB (commonly written as LAB) is an international standard color space designed by the Commission Internationale de l'Éclairage to be perceptually uniform. In LAB, the same numerical distance corresponds to roughly the same perceived color difference, regardless of whether you're comparing dark blues, bright yellows, or muted grays. This perceptual uniformity makes LAB ideal for similarity search.

LAB Structure

LAB separates color into three components that mirror how human vision processes color:

L* (Lightness): 0 (black) to 100 (white), roughly aligned to perceived brightness
a*: green–red opponent channel; negative = green, positive = red (≈ −128 to +128)
b*: blue–yellow opponent channel; negative = blue, positive = yellow (≈ −128 to +128)

ΔE (Delta E): Measuring Perceptual Distance

In LAB space, the Euclidean distance between two colors is ΔE (Delta E):

ΔE76 = √[(L₂ - L₁)² + (a₂ - a₁)² + (b₂ - b₁)²]

Indicative interpretation based on empirical studies:

ΔE ≤ 1: Not perceptible under normal viewing
ΔE 1–2: Perceptible with close observation
ΔE 2–10: Noticeable; "similar but slightly different"
ΔE > 10: Clearly different

For most applications, ΔE76 (simple Euclidean distance) is sufficient. For precision-critical cases (e.g., cosmetics, paint), use ΔE2000, which compensates for known non-uniformities (notably in blue regions).

Architecture Overview

The pipeline extracts representative colors, converts them to LAB, and indexes vectors for fast similarity search:

Once colors are in LAB space, finding similar colors becomes a standard k-NN problem that OpenSearch's vector search capabilities handle efficiently.

Implementation

Step 1: RGB to LAB Conversion

First, extract a representative color (e.g., with OpenCV k-means clustering over product pixels, Amazon Rekognition features, or a masked region average for the product area). Then convert RGB to LAB using colormath:

from colormath.color_objects import sRGBColor, LabColor
from colormath.color_conversions import convert_color

def rgb_to_lab(r, g, b):
    """
    Convert RGB (0-255) to normalized LAB vector.
    Normalization keeps dimensions on comparable scales for k-NN.
    Without this, L* (0-100 range) would dominate distances.
    """
    rgb = sRGBColor(r, g, b, is_upscaled=True)
    lab = convert_color(rgb, LabColor)
    return [
        lab.lab_l / 100.0,   # L* [0,100] -> [0,1]
        lab.lab_a / 128.0,   # a* [-128,127] -> ~[-1,1]
        lab.lab_b / 128.0    # b* [-128,127] -> ~[-1,1]
    ]

# Example: Convert a burgundy coat color
lab_vector = rgb_to_lab(184, 33, 45)
print(lab_vector)  # [0.4036, 0.4555, 0.2576]

Multi-color products: For items with several prominent colors, either (a) index the dominant color (simple, smaller index), or (b) index the top N colors as separate docs sharing the same product_id (better recall; merge duplicates at read time).

Step 2: Create OpenSearch Index

Security note (production): Use VPC placement, IAM roles or fine-grained access control, and sign REST calls with AWS Signature Version 4.

PUT /product-colors
{
  "settings": {
    "index.knn": true,
    "index.number_of_shards": 2,
    "index.number_of_replicas": 1
  },
  "mappings": {
    "properties": {
      "product_id": { "type": "keyword" },
      "title": { "type": "text" },
      "color_name": { "type": "keyword" },
      "lab_vector": {
        "type": "knn_vector",
        "dimension": 3,
        "method": {
          "name": "hnsw",
          "engine": "lucene",
          "space_type": "l2",
          "parameters": {
            "ef_construction": 128,
            "m": 24
          }
        }
      },
      "brand": { "type": "keyword" },
      "price": { "type": "float" }
    }
  }
}

lab_vector uses space_type: "l2" (Euclidean distance), aligning with ΔE76. HNSW provides fast approximate nearest neighbors; tune m/ef_construction for your scale and accuracy needs.

Step 3: Index Products

from opensearchpy import OpenSearch, helpers

client = OpenSearch(endpoint, http_auth=(user, password))

actions = []
for product in products:
    lab_vec = rgb_to_lab(*product['rgb'])
    actions.append({
        "_index": "product-colors",
        "_id": product['id'],
        "_source": {
            "product_id": product['id'],
            "title": product['title'],
            "color_name": product.get('color_name', 'Unknown'),
            "lab_vector": lab_vec,
            "brand": product['brand'],
            "price": product['price']
        }
    })

success, errors = helpers.bulk(client, actions)
print(f"Indexed {success} documents")
if errors:
    print(f"Errors: {errors}")

Step 4: Query Similar Colors

Basic similarity:

POST /product-colors/_search
{
  "size": 20,
  "query": {
    "knn": {
      "lab_vector": {
        "vector": [0.54, 0.64, 0.52],
        "k": 50
      }
    }
  }
}

(Fetch k=50 candidates to improve recall, then return size=20 to keep payloads small.)

Combine color similarity with business filters to ensure relevance:

POST /product-colors/_search
{
  "size": 20,
  "query": {
    "bool": {
      "must": [
        {
          "knn": {
            "lab_vector": {
              "vector": [0.54, 0.64, 0.52],
              "k": 50
            }
          }
        }
      ],
      "filter": [
        { "term": { "brand": "Premium Outerwear Co." } },
        { "range": { "price": { "lte": 500 } } }
      ]
    }
  }
}

Step 5: Optional ΔE2000 Re-Ranking

Use ΔE2000 when tiny shade differences matter (cosmetics, paint, textiles). For general e-commerce, ΔE76 is typically sufficient and faster.

import colorspacious

def rerank_with_delta_e2000(query_lab_vec, candidates, top_n=10):
    """Re-rank candidates using ΔE2000 for maximum perceptual accuracy."""
    query_lab = [
        query_lab_vec[0] * 100.0,  # L* [0,100]
        query_lab_vec[1] * 128.0,  # a* [-128,127]
        query_lab_vec[2] * 128.0   # b* [-128,127]
    ]

    scored = []
    for cand in candidates:
        lab_vec = cand['lab_vector']
        cand_lab = [
            lab_vec[0] * 100.0,
            lab_vec[1] * 128.0,
            lab_vec[2] * 128.0
        ]

        delta_e = colorspacious.deltaE(query_lab, cand_lab, input_space="CIELab")
        scored.append((delta_e, cand))

    scored.sort(key=lambda x: x[0])
    return [cand for _, cand in scored[:top_n]]

Real-World Use Cases

Fashion E-Commerce: Alternative Product Recommendations

Index each product's dominant color in LAB and use a moderate ΔE threshold (up to ~8) to include related shades (wine, maroon, oxblood). Combine with size/brand/category filters to keep results relevant.

Cosmetics: Precise Shade Matching

Use tight ΔE thresholds (< 2) plus ΔE2000 re-ranking. Optionally filter by undertone (warm/cool/neutral). This reduces returns and builds trust.

Brand Protection: Counterfeit Detection

Detect subtle color deviations in logos/branding. Index genuine logo LAB vectors and monitor marketplace listings for significant deviations; flag when ΔE > 15 for review. This approach reduced manual review workload by ~40% in a PoC and complements image/text analysis pipelines.

import numpy as np
from skimage.color import deltaE_ciede2000

def rerank_with_delta_e2000(query_lab_vec, candidates, top_n=10):
    """Re-rank candidates using true ΔE2000 for maximum perceptual accuracy."""
    if not candidates:
        return []

    # Prepare query
    q = np.array([query_lab_vec[0]*100.0, query_lab_vec[1]*128.0, query_lab_vec[2]*128.0], dtype=np.float64)

    # Build candidate array (n,3)
    cand_arr = np.array([
        [c['lab_vector'][0]*100.0, c['lab_vector'][1]*128.0, c['lab_vector'][2]*128.0]
        for c in candidates
    ], dtype=np.float64)

    # Compute ΔE2000 for all candidates at once
    q_rep = np.repeat(q[np.newaxis, :], cand_arr.shape[0], axis=0)
    delta_es = deltaE_ciede2000(q_rep, cand_arr)

    # Sort and return top_n
    idx = np.argsort(delta_es)[:top_n]
    return [candidates[i] for i in idx]

Best Practices

Implementation

Standardize photography (D65 ~6500K) and camera settings.
Work in LAB; avoid raw RGB similarity.
Handle backgrounds (segmentation/cropping to product pixels).
Choose color strategy: dominant color vs. top-N colors per item.

Performance & Scale

Start with ΔE76; add ΔE2000 only if user tests require it.
Combine with business filters (category, brand, price, size).
Tune HNSW (m, ef_construction, and ef_search).

Security & Operations

Secure the domain (VPC, IAM/FGAC, TLS, SigV4).
Alarms for p95 latency and memory pressure.
Iterate using CTR, conversion, complaints, and latency telemetry.

Validation

User tests to calibrate ΔE thresholds per domain.
A/B pilots before full rollout; monitor CTR, conversion, bounce, returns.

Bottom Line

Building perceptual color similarity search is about aligning technology with how humans actually see. Using CIELAB vectors and k-NN search in Amazon OpenSearch Service bridges that gap, allowing systems to understand color differences the way people do. Whether in fashion, cosmetics, or brand protection, it enables intuitive, human-centric experiences that go far beyond simple RGB filters.

If you are exploring how to make your product search perceptually aware or want to prototype an OpenSearch-based similarity engine, feel free to reach out.

At Reply, we help organizations design intelligent, scalable, and vision-aligned search solutions from proof of concept to production.

How to Use Amazon OpenSearch Service Index Aliases with Knowledge Bases in Amazon Bedrock

Alexey Vidanov — Wed, 30 Jul 2025 13:31:42 +0000

Many teams start experimenting with Amazon Bedrock Knowledge Bases using the default setup. It works fine — until it doesn’t.

Once your workloads stabilize, you’ll likely want:

To optimize the mapping (e.g., adjust analyzers or add new fields)
To change shard counts for scaling
To version your data and test new schema ideas safely

Without index aliases, making these changes requires downtime or recreating the KB — an annoying and error-prone process.

Index aliases solve this by decoupling Bedrock from the physical index. You keep the Bedrock configuration pointing to a stable name (bedrock_index), while swapping the backend index version (bedrock_index_v1 → bedrock_index_v2) invisibly.

OpenSearch Vector Storage Options (At a Glance)

Zoom image will be displayed

What Are Index Aliases and Why Use Them?

An index alias is a logical pointer to one or more real indices in OpenSearch. You configure Bedrock to use a fixed alias name (e.g., bedrock_index), while the actual data resides in versioned indices (bedrock_index_v1, bedrock_index_v2, ...).

Benefits of Using Aliases:

Zero-Downtime Schema Changes: Swap backend index without reconfiguring Bedrock
Instant Rollbacks: Revert to previous index in seconds
Blue/Green Deployments: Test new index versions behind the same alias
Simplified Access Controls: Apply policies to a single alias instead of multiple indices
Lifecycle Management: Route hot/cold data behind one consistent alias
Cleaner Code and Integrations: External tools or apps always talk to the same alias

**Performance note:** Aliases introduce negligible latency. Read/write operations perform the same as direct index access, unless multiple indices are targeted.

Alias Swap

Step-by-Step Guide

Implementing index aliases for Amazon Bedrock Knowledge Bases with Amazon OpenSearch Service requires a few careful setup steps — but once done, you gain flexibility, versioning, and zero-downtime upgrades.

This guide walks you through:

the required permissions and access policies,
how to configure OpenSearch correctly, and
how to use aliases with existing or new Knowledge Bases.

Whether you’re retrofitting aliases into a running system or designing for future-proofing from day one, these instructions will help you avoid disruptions and enable smooth schema evolution.

Prerequisites

Before starting, make sure your environment meets these conditions:

IAM Permissions: The Bedrock service role must have explicit permissions to interact with your OpenSearch domain and indices. Use the following policy as a template:

"Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "es:ESHttpGet",
                "es:ESHttpPost", 
                "es:ESHttpPut",
                "es:ESHttpDelete"
            ],
            "Resource": [
                "arn:aws:es:<region>:<accountId>:domain/<domainName>/<indexName>/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "es:DescribeDomain"
            ],
            "Resource": [
                "arn:aws:es:<region>:<accountId>:domain/<domainName>"
            ]
        }
    ]
}

Public OpenSearch Domain: Bedrock Knowledge Bases do not yet support VPC access. Ensure your domain is public and reachable from Bedrock.
OpenSearch Access Policy: Your OpenSearch domain must allow access from the Bedrock role. Example policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<accountId>:role/<BedrockServiceRole>"
            },
            "Action": [
                "es:ESHttpGet",
                "es:ESHttpPost",
                "es:ESHttpPut", 
                "es:ESHttpDelete",
                "es:DescribeDomain"
            ],
            "Resource": [
                "arn:aws:es:<region>:<accountId>:domain/<domainName>",
                "arn:aws:es:<region>:<accountId>:domain/<domainName>/*"
            ]
        }
    ]
}

Replace , , , , and with your actual values.

Alias Integration Scenarios

Once the IAM and access policies are in place, you’re ready to apply index aliases. There are two main paths depending on your current state:

If you already have a working Bedrock Knowledge Base, follow Scenario A to transition to aliases.
If you’re starting fresh, Scenario B shows how to set it up the right way from the beginning.

A. Using Aliases with an Existing Knowledge Base

Identify the current Bedrock index (e.g., bedrock_index).
Create a new versioned index with your updated schema and settings:

PUT bedrock_index_v2
{
  "settings": { "number_of_shards": 1, "number_of_replicas": 1, "knn": true },
  "mappings": {
    "properties": {
      "bedrock-knowledge-base-default-vector": {
        "type": "knn_vector",
        "dimension": 1024,
        "method": { "engine": "faiss", "name": "hnsw", "space_type": "l2" }
      },
      "AMAZON_BEDROCK_TEXT": { "type": "text" },
      "AMAZON_BEDROCK_METADATA": { "type": "text", "index": false }
    }
  }
}

3. Reindex your data from the old index into the new one:

POST _reindex
{
  "source": { "index": "bedrock_index" },
  "dest": { "index": "bedrock_index_v2" }
}

**Validation tip:**

GET _cat/aliases/bedrock_index?v
GET bedrock_index/_search?size=0

4. Switch the alias and remove the original index:

DELETE bedrock_index
POST _aliases
{
  "actions": [
    { "add": { "index": "bedrock_index_v2", "alias": "bedrock_index" } }
  ]
}

B. Setting Up a New Knowledge Base from Scratch

If you haven’t created the Knowledge Base yet, you can start clean with the alias approach. This gives you full flexibility from day one.

Create a temporary placeholder index to satisfy the Bedrock setup wizard:

PUT bedrock_index
{
  "settings": { "number_of_shards": 1, "number_of_replicas": 1, "knn": true },
  "mappings": {
    "properties": {
      "bedrock-knowledge-base-default-vector": {
        "type": "knn_vector",
        "dimension": 1024,
        "method": { "engine": "faiss", "name": "hnsw", "space_type": "l2" }
      },
      "AMAZON_BEDROCK_TEXT": { "type": "text" },
      "AMAZON_BEDROCK_METADATA": { "type": "text", "index": false }
    }
  }
}

2. Create your production index with the intended schema and settings:

PUT bedrock_index_v1
{ /* use desired schema here */ }

3. Swap the alias to point to the real index:

DELETE bedrock_index
POST _aliases
{
  "actions": [
    { "add": { "index": "bedrock_index_v1", "alias": "bedrock_index" } }
  ]
}

Schema Evolution Workflow (Using Aliases)

Use this pattern to apply schema changes without downtime:

Create a new versioned index Example: bedrock_index_v2 with updated mappings or settings
Reindex the data Copy documents from the current index to the new one using _reindex
Test and validate Run sample queries, check document counts, and confirm relevance
Update the alias Point bedrock_index alias to the new index using _aliases

POST _aliases
{
 "actions": [
 { "remove": { "index": "bedrock_index_v1", "alias": "bedrock_index" } },
 { "add": { "index": "bedrock_index_v2", "alias": "bedrock_index" } }
 ]
}

5. Clean up old indices Delete outdated versions like bedrock_index_v1(optional but recommended)

Error Handling & Troubleshooting

Even with careful planning, issues can arise during reindexing or alias management. Here’s how to address common problems:

Alias Update Fails

Ensure the alias name isn’t already assigned to another index
Make alias updates atomic using the _aliases API (remove+add in one request)
Confirm you have write permissions for the domain and target indices

Missing or Mismatched Data

Compare document counts across indices using GET /<index>/_count
Re-run _reindex with a filtered query to catch missed documents
Watch for document ID collisions or field mapping mismatches

**Pro tip:** Always validate the final setup with:

GET _cat/aliases?v
GET bedrock_index/_search?size=0

Reindex Operation Fails

Use GET _tasks to check task status and diagnose errors
Run reindex asynchronously using wait_for_completion=false for better control and retry logic
Check OpenSearch logs or CloudWatch for throttling or mapping issues

To make the _reindex request asynchronous, use the ?wait_for_completion=false query parameter. This allows the task to run in the background, and you can later track it using the returned task ID.

Asynchronous Reindex Example

POST _reindex?wait_for_completion=false
{
  "source": { "index": "bedrock_index" },
  "dest": { "index": "bedrock_index_v2" }
}

Response

{
  "task": "tUV03FsmR8Kkz5mF6J9xxxx:12345"
}

Check Status

GET _tasks/tUV03FsmR8Kkz5mF6J9xxxx:12345

You can also cancel it if needed:

POST _tasks/tUV03FsmR8Kkz5mF6J9xxxx:12345/_cancel

Rollback Procedure

If something goes wrong after an alias switch, rolling back is simple — provided you’ve kept the old index.

Retain previous index versions Always keep earlier versions (e.g., bedrock_index_v1) until validation is complete.
Repoint the alias If issues arise, restore the alias to the previous version:

POST _aliases
{
 "actions": [
 { "remove": { "index": "bedrock_index_v2", "alias": "bedrock_index" } },
 { "add": { "index": "bedrock_index_v1", "alias": "bedrock_index" } }
 ]
}

3. Verify rollback success

GET _cat/aliases?v
GET bedrock_index/_search?q=test&size=5

Pro Tips

Use versioned index names like bedrock_index_v1, bedrock_index_v2 to track schema evolution
Automate reindexing and alias switching in your CI/CD pipeline
Always validate with:

GET _cat/aliases?v
GET bedrock_index/_search?size=0

During migration, consider setting:

"index.blocks.write": true

"index.blocks.read_only_allow_delete": true

to prevent unintended writes to old indices.

Bottom Line

Until Amazon Bedrock natively supports index aliases, using OpenSearch aliases is the best way to enable continuous schema evolution with zero downtime. For anything beyond quick prototypes or minimal workloads, a managed OpenSearch domain with versioned indices and alias controloffers better cost-efficiency, observability, and long-term flexibility.

If you’re unsure how to structure your Bedrock Knowledge Base or want to explore advanced OpenSearch patterns, feel free to drop me a message.

At Reply, we help organizations design scalable, secure, and future-ready AI architectures — whether you’re just getting started or optimizing production workloads.

Building Custom Script Plugins in Amazon OpenSearch Service: A Technical Deep Dive

Alexey Vidanov — Tue, 17 Jun 2025 08:22:47 +0000

Amazon OpenSearch Service now supports custom plugins, allowing advanced users to extend the search engine’s functionality beyond its out-of-the-box features. In this deep dive, we focus on the newest plugin type – Script Plugins – and explore how to create one, how they differ from built-in scripts, and best practices for developing and deploying them. This guide provides a tutorial-style walkthrough with detailed technical insights.

What Are Custom Plugins?

OpenSearch plugins are modular extensions that run within the OpenSearch cluster, enabling custom functionality such as analyzers, queries, and scoring logic. While self-managed OpenSearch (and historically Elasticsearch) has long supported these plugins, Amazon OpenSearch Service (AOS) did not allow user-developed plugins—until late 2024.

That changed with the release of version 2.15, which introduced support for custom plugins in the managed service. This opened up new possibilities for developers to tailor AOS to meet specific application needs.

Timeline of Plugin Support in Amazon OpenSearch Service

Version 2.15 (Late 2024) – Custom plugin support launched with initial focus on AnalysisPlugin and SearchPlugin.
May 2025 – ScriptPlugin support was added, enabling advanced use cases such as custom scoring, filtering, and field transformations within queries.

Currently Supported Plugin Types in AOS

AnalysisPlugin – Add custom analyzers, tokenizers, or filters to extend text analysis.
SearchPlugin – Create custom query types, scoring logic, suggesters, or aggregations.
MapperPlugin – Define custom field types and control how data is indexed and stored.
ScriptPlugin (since 2.15) – Embed custom scripting engines to implement complex query-time logic.

⚠️ As of mid-2025, other plugin types—such as IngestPlugin, ActionPlugin, and EnginePlugin—are not supported in Amazon OpenSearch Service.

Script Plugins: Core Concepts

What Is a Script Plugin?

In OpenSearch, scripts (written in the built-in Painless scripting language) are often used in queries for custom scoring, filtering, or field transformations. A script plugin allows you to go beyond what Painless scripts can do by adding new scripting logic in Java or even introducing entirely new scripting languages to OpenSearch. As the Tinder engineering team put it, a script plugin is essentially a run() function that takes query parameters and a document (“lookup”) as input and produces a relevance score (or decision) as output. In other words, a script plugin lets you inject custom code into the scoring process of the search engine.

Script Plugins vs. Painless Scripts

Script plugins offer several advantages over standard Painless inline scripts:

Richer Logic – You can implement complex algorithms and leverage Java libraries or external frameworks. (Painless is sandboxed and limited to basic operations.)
New Scripting Languages – You aren’t limited to Painless; a plugin can define a new script language or domain-specific language for OpenSearch queries.
Performance – Custom script engines are written in Java and compiled, which can yield better performance than interpreted Painless scripts for heavy computations.
Greater Control – Script plugins run inside the OpenSearch JVM with broader privileges. This gives you more power (e.g. access to low-level APIs or optimized data structures) than the sandboxed environment of Painless. (Of course, with this power comes the responsibility to ensure safety and stability.)

When to Choose Script Plugins

Scenario	Script Plugin	Painless Script	Application Layer
Performance	✅ Best (compiled Java)	⚠️ Moderate	❌ Higher latency
Complex Logic	✅ Full Java capabilities	⚠️ Limited	✅ Most flexible
Deployment	⚠️ Requires deployment	✅ No deployment	✅ No deployment
Updates	⚠️ Requires redeployment	✅ Easy to update	✅ Easy to update
External Services	❌ Not allowed	❌ Not allowed	✅ Full access
Resource Usage	✅ Optimized	⚠️ Moderate	❌ Higher overhead

Before implementing a script plugin, consider these alternatives:

Painless Scripts: For simpler use cases, offering a good balance of flexibility and performance with no deployment required.
Application Layer: When you need maximum flexibility or access to external services, though it comes with higher latency.
Built-in Features: OpenSearch's built-in features like function score queries, runtime fields, and script fields might already provide what you need.

Limitations and Considerations for Script Plugins in Amazon OpenSearch Service

Before using script plugins in Amazon OpenSearch Service, be aware of the following constraints:

No External API Calls

Script plugins can't access external services or HTTP endpoints. This sandboxing ensures security and performance stability.

Version Compatibility

Only specific OpenSearch versions support custom plugins:

Supported: 2.15, 2.17
Not supported: 2.19 (in our tests in June 2025, plugin validation failed on AWS-managed clusters)

Blue/Green Deployment Required

Plugin installation triggers a blue/green deployment. The cluster is recreated behind the scenes. There is no downtime, but installation can take time. Plan accordingly in production.

Feature Limitations

Custom plugins disable several AWS-managed features:

Cross-Cluster Search/Replication
Remote Reindexing
Auto-Tune
Multi-AZ with Standby
AWS-hosted OpenSearch Dashboards (requires self-hosting)

Performance Impact

Script logic runs per document at query time and may increase latency or resource usage.

Developing a Custom Script Plugin (Step-by-Step)

In this section, we’ll walk through creating a custom script plugin for OpenSearch. Our example will be a “Hello World” script plugin with a GenAI-powered scoring function. This plugin will demonstrate:

Custom Scoring Logic – A scoring algorithm that considers multiple factors (product rating, price, stock availability, recency of updates, etc.) to adjust relevance scores.
Parameterized Configuration – The ability to adjust scoring weights and thresholds at query time via parameters (so you can fine-tune the behavior without changing the code).
Built-in Optimizations – Efficient calculations, input validation, and error handling to minimize performance overhead and ensure stability.

Development Environment Setup

For this example, we have a sample project available on GitHub that contains the full plugin implementation. You can use it as a starting point for your own plugin development:

# Clone the example repository
git clone https://github.com/vidanov/opensearch-script-plugin-hello-world.git
cd opensearch-script-plugin-hello-world

(Ensure you have a Java 17 JDK and Gradle available, as OpenSearch 2.x plugins use Java 17.)

Project Structure and Organization

The project follows a typical OpenSearch plugin layout. Key files and directories include:

genai-script-plugin-with-ai/
├── src/
│   ├── main/java/com/example/
│   │   └── HelloWorldScriptPlugin.java    # Main plugin implementation (Java)
│   ├── main/resources/
│   │   └── plugin-descriptor.properties   # Plugin metadata (name, version, type)
│   └── test/java/com/example/
│       └── HelloWorldScriptPluginTest.java # Unit tests for the plugin logic
├── build.gradle                           # Gradle build configuration for OpenSearch
└── README.md                              # Documentation and usage instructions

This structure is generated by the OpenSearch plugin build tools. The Java class HelloWorldScriptPlugin.java is our primary focus – it defines the plugin and the custom script engine.

Core Implementation

Our plugin class needs to extend the base Plugin class and implement the ScriptPlugin interface provided by OpenSearch. This requires us to supply a custom Script Engine. Essentially, the script engine is where we define the logic of our new scripting language. Below is a key part of the implementation:

public class HelloWorldScriptPlugin extends Plugin implements ScriptPlugin {
    @Override
    public ScriptEngine getScriptEngine(Settings settings, Collection<ScriptContext<?>> contexts) {
        return new HelloWorldScriptEngine();
    }
}

In this snippet, we override getScriptEngine(...) to return an instance of our custom HelloWorldScriptEngine. This engine (implemented as an inner class or separate class) registers a new script language – in our case called "hello_world" – with OpenSearch. The script engine is responsible for compiling script source code and producing a ScoreScript that OpenSearch can execute for each document during queries.

How the script engine works: Inside HelloWorldScriptEngine, we define how to handle different script contexts. For a score script, our engine provides a factory that uses the parameters and document fields to calculate a score. For example, if the script source is "custom_score", our engine’s ScoreScript will read the document’s fields (rating, price, stock, etc.) and the provided params (thresholds, boosts, penalties) and compute a final score. All of this logic is written in Java, giving us full flexibility in how scoring is done. (You could also implement other script functions or additional script source names, e.g. different scoring strategies, within the same plugin.)

Parameterized Scoring Implementation

One of the most powerful features of script plugins in Amazon OpenSearch Service is the ability to parameterize the scoring logic. Instead of hard-coding thresholds and weights, the plugin can read parameters from the query at runtime.

This makes your scoring configurable, testable, and adaptive — ideal for scenarios like A/B testing, personalization, or multi-tenant ranking logic.

Why Use Parameterized Scoring?

Dynamic Tuning at Query Time
No Plugin Redeploy Required
Multiple Strategies via One Plugin
Support for A/B Testing and Experiments

How It Works (With Code Example)

In your GenAIScoreScriptFactory, parameters are parsed using helpers like pDouble() and pString():

double ratingThreshold = pDouble(params, "rating_threshold", 4.5);
double priceThreshold  = pDouble(params, "price_threshold", 100.0);
double ratingWeight    = pDouble(params, "rating_weight", 0.4);
String ratingField     = pString(params, "rating_field", "rating");

These values are passed at query time. You can modify them without changing the plugin code.

Example: Passing Parameters in a Query

{
  "query": {
    "script_score": {
      "query": { "match_all": {} },
      "script": {
        "source": "weighted_score",
        "lang": "hello_world",
        "params": {
          "rating_field": "avg_rating",
          "price_field": "discounted_price",
          "rating_weight": 0.5,
          "price_weight": 0.2,
          "max_price": 500.0
        }
      }
    }
  }
}

In this example:

We invoke the weighted_score strategy inside the plugin.
We override field names and scoring weights at query time.

Switching Between Scoring Strategies

The plugin supports different scoring strategies (weighted_score, custom_score, popularity_score) based on the script source:

@Override
public double execute(ExplanationHolder explanation) {
    if ("weighted_score".equals(scriptSource)) {
        return weightedScore();
    } else if ("popularity_score".equals(scriptSource)) {
        return popularityScore();
    } else {
        return customScore(); // fallback
    }
}

You can switch strategies with:

"source": "popularity_score"

No need to rebuild or redeploy — simply change the script source in the query.

Using Amazon Q Developer to Create and Implement a Java Score Script Plugin

If you're building custom scoring logic for Amazon OpenSearch Service, you don’t have to start from scratch. Amazon Q Developer can generate the entire Java class for your plugin — including parameterized scoring logic, plugin structure, and runtime selection of different strategies — from a single, well-crafted prompt.

Step 1: Define the Logic in Plain English

Start by describing your goal clearly. For example:

"I want to create a scoring plugin that boosts well-rated and cheap products, penalizes out-of-stock items, and includes a popularity score based on views, sales, and review count. All thresholds and weights should be configurable via query parameters."

Step 2: Use a Single Prompt in Q Developer

You can paste the following prompt into Q Developer to generate the entire plugin code:

Create a Java ScoreScript plugin for Amazon OpenSearch Service (Java 11 compatible) named `HelloWorldScriptPlugin`.

The plugin should:

1. Support a strategy called `popularity_score` with the following logic:
   - Normalize `views`, `sales`, `review_count`, and `rating`
   - Use logarithmic scaling for `views`, `sales`, and `review_count`
   - Use: log(value + 1) / log(max_value + 1)
   - Normalize `rating` by dividing by 5.0

2. Allow configurable weights via params:
   - `views_weight`, `sales_weight`, `reviews_weight`, `rating_weight`
   - Provide default weights (e.g., 0.25 for each)

3. Compute the final score as the weighted sum of the normalized values.

4. Parse parameters using helper method `pDouble(params, key, defaultValue)`

5. Extract document field values using `docDouble(field, defaultValue)`.

6. Add a fallback strategy `custom_score` with simplified logic: multiply three boosts based on rating, price, and stock.

7. Add support for passing `scriptSource` as a string (e.g. "popularity_score") to select between scoring strategies.

Generate the full plugin, including the `Plugin`, `ScriptPlugin`, `ScoreScript.Factory`, and `ScoreScript` logic.

Step 3: What You'll Get from Q Developer

Q Developer will typically generate:

A plugin class implementing ScriptPlugin
A custom ScriptEngine with support for ScoreScript
A factory that reads parameters and selects logic
A ScoreScript that implements:
- customScore() logic with boost/penalty
- popularityScore() logic using weighted normalized values
- execute() method with strategy selection logic
Helper methods for safe parameter and field access

Step 4: Adjust and Compile

After you get the code:

Review field names and adjust if needed.
Place the code inside a Gradle-based plugin scaffold.
Ensure Java 17 and OpenSearch 2.x compatibility.
Use the OpenSearch Gradle plugin to build your .zip package.

You can then deploy this plugin to your Amazon OpenSearch Service cluster.

Installation and Operations

Once your custom script plugin is developed and tested, the next step is to deploy it to an Amazon OpenSearch Service domain. Deploying a plugin on AWS involves preparing the plugin as a zip package, uploading it, and then instructing the OpenSearch Service to install it on your domain. Here we outline the requirements and steps for a successful deployment.

Prerequisites and Requirements

Before deploying a custom plugin, ensure your target OpenSearch domain meets the following requirements (these are mandated by AWS for custom plugins):

OpenSearch Version 2.15 or 2.17 – Custom plugins are supported only on versions 2.15+ (and remember, not on 2.19 yet).
Node-to-node encryption enabled – Your domain must have node-to-node encryption turned on.
Encryption at rest enabled – The domain must have encryption of data at rest.
HTTPS enforced – Only HTTPS access is allowed (no plaintext HTTP).
TLS security policy – The domain should use a modern TLS security policy (e.g. Policy-Min-TLS-1-2-PFS-2023-10 or newer).

# Upload to S3
aws s3 cp build/distributions/hello-world-genai-script-plugin.zip s3://your-bucket/plugins/

# Create package
aws opensearch create-package \
  --package-name hello-world-genai-script-plugin \
  --package-type ZIP-PLUGIN \
  --package-source S3BucketName=genai-plugin-bucket,S3Key=plugins/hello-world-genai-script-plugin.zip \
  --engine-version OpenSearch_2.15 \
  --region <YOUR_AWS_REGION>

# Wait till the package is validated, associate
aws opensearch associate-package \
    --package-id <PACKAGE_ID> \
    --domain-name <OPENSEARCH_DOMAIN_NAME> \
    --region <YOUR_AWS_REGION>

# Verify
aws opensearch list-packages-for-domain --domain-name <OPENSEARCH_DOMAIN_NAME>

Plugin installation triggers blue/green deployment—no downtime but takes time.

You can filter the custom plugins in the AWS management console

Plugin usage examples

Example: Basic Script Score Query

To illustrate, consider an e-commerce product search scenario. We want to boost products that are highly rated, reasonably priced, in stock, and recently updated. We have deployed our HelloWorldScriptPlugin which defines a script language "hello_world" with a script function called "custom_score". Here’s how a search query might use this custom script with parameters:


# Let us create an example product index

PUT products_test
{
  "settings": {
    "index": {
      "number_of_shards": 1,
      "number_of_replicas": 0
    }
  },
  "mappings": {
    "properties": {
      "name": {
        "type": "text"
      },
      "rating": {
        "type": "float"
      },
      "price": {
        "type": "float"
      },
      "stock": {
        "type": "integer"
      },
      "last_updated": {
        "type": "date"
      },
      "views": {
        "type": "integer"
      },
      "sales": {
        "type": "integer"
      }
    }
  }
}

# Let us add some products

POST products_test/_bulk?refresh=true
{"index":{}}
{"name":"Alpha Wireless Headphones","rating":4.6,"price":45.0,"stock":12,"last_updated":"2025-05-20","views":1000,"sales":150}
{"index":{}}
{"name":"Beta Noise-Cancelling Headphones","rating":4.9,"price":120.0,"stock":5,"last_updated":"2025-05-05","views":5000,"sales":400}
{"index":{}}
{"name":"Gamma Budget Earbuds","rating":3.9,"price":25.0,"stock":30,"last_updated":"2025-04-15","views":250,"sales":50}
{"index":{}}
{"name":"Delta Premium Over-Ear","rating":4.3,"price":220.0,"stock":0,"last_updated":"2025-04-01","views":3000,"sales":250}
{"index":{}}
{"name":"Epsilon Sport Earbuds","rating":4.1,"price":60.0,"stock":8,"last_updated":"2025-05-30","views":1800,"sales":300}

# And test the first query

GET products_test/_search
{
  "query": {
    "function_score": {
      "query": {
        "match": {
          "name": "wireless headphones"
        }
      },
      "script_score": {
        "script": {
          "lang": "hello_world",
          "source": "custom_score",
          "params": {
            "rating_threshold": 4.0,
            "rating_boost": 1.5,
            "price_threshold": 50.0,
            "cheap_boost": 1.3,
            "expensive_penalty": 0.8,
            "out_of_stock_penalty": 0.3,
            "base_multiplier": 2.0,
            "fallback_score": 0.5,
            "recency_factor": 0.1,
            "popularity_weight": 0.7,
            "price_weight": 0.3
          }
        }
      }
    }
  }
}

In this query, we search for products with descriptions matching “wireless headphones,” then apply a script_score to modify the relevance score of each result using our plugin’s logic. We pass a number of parameters to custom_score that control how the scoring works. Here’s what each parameter means:

Rating Threshold & Boost:
- rating_threshold – The minimum rating (e.g. average customer review) for a product to be considered “highly rated” and receive a boost. In our example, 4.0 stars.
- rating_boost – The multiplier to apply if the product’s rating exceeds the threshold. (1.5x in this case, meaning highly-rated products get a 1.5× score boost from the rating factor.)
Price Parameters:
- price_threshold – A price cutoff to distinguish “cheap” vs “expensive” products (here $50).
- cheap_boost – Multiplier for products priced under the threshold (1.3x, giving cheaper items a boost).
- expensive_penalty – Multiplier for products over the threshold (0.8x, slightly penalizing pricier items).
Stock Parameter:
- out_of_stock_penalty – Multiplier to apply if an item is out of stock (0.3x in the example, significantly reducing the score for items that aren’t available to purchase).
Scoring Weights:
- popularity_weight – Weight (relative importance) of the item’s popularity in the overall score calculation (e.g. 0.7).
- price_weight – Weight of the price factor in the overall score (e.g. 0.3). (These weights might be used inside the script to combine factors like popularity vs price impact. In our simple example, they could control a weighted sum, but how they’re applied depends on the script’s code.)
Recency Factor:
- recency_factor – A decay factor for recency (e.g. 0.1). This could be used to give a small boost to newer or recently updated products, or conversely to decay older items’ scores over time.
Base Multiplier:
- base_multiplier – An overall score multiplier applied at the end of the calculation (in our case 2.0, meaning after all other factors the score is doubled). This can be useful to calibrate the output of the script to a desired range or importance relative to the original query score.
Fallback Score:
- fallback_score – A default score to return if the script cannot compute a meaningful score for a document (for example, if required fields are missing or an error occurs). Here it’s 0.5. Using a fallback ensures that an error in script execution doesn’t completely drop the document from results; it still gets a baseline score.

These parameters correspond to how we wrote the script logic in the plugin. For instance, the plugin might check each document’s rating field against rating_threshold to decide whether to apply rating_boost. It likely multiplies factors like rating boost, price boost/penalty, and stock penalty together (as we implemented) and then multiplies by base_multiplier. The fallback_score would be returned if any exception or missing data prevents the normal calculation.

Advanced Scoring Strategies

The real power of parameterized scripts is that you can adjust the scoring to different scenarios by simply changing the parameters. You might even store and reuse parameter sets for various contexts. For example:

Holiday Season: During a holiday shopping season, you might want to aggressively boost highly-rated products(assuming reviews matter more during gift shopping) and also raise the price threshold (people may spend more on gifts). You could use parameters like:

GET products_test/_search
{
  "query": {
    "script_score": {
      "query": {
        "match_all": {}
      },
      "script": {
        "lang": "hello_world",
        "source": "custom_score",
        "params": {
          "rating_threshold": 4.0,
          "rating_boost": 2.0,
          "price_threshold": 100.0,
          "cheap_boost": 1.5,
          "expensive_penalty": 0.9,
          "out_of_stock_penalty": 0.1
        }
      }
    }
  }
}

Parameter Explanations:

rating_threshold: 4.0 — Only highly rated items get boosted.
rating_boost: 2.0 — Extra score for items above the threshold.
price_threshold: 100.0 — Defines "cheap" items during promotions.
cheap_boost: 1.5 — Boost cheaper items more.
expensive_penalty: 0.9 — Slight penalty for costly products.
out_of_stock_penalty: 0.1 — Heavy penalty if the item is unavailable.

In this holiday configuration, we doubled the rating boost and increased the cheap boost, while being more lenient on expensive items (0.9 penalty is a mild reduction) because shoppers might splurge more.

Clearance Sale: For a clearance sale scenario, you might want to heavily favor cheaper items and don’t require as high a rating (since clearance items might not all be top-rated). A parameter set could be:

GET products_test/_search
{
  "query": {
    "script_score": {
      "query": {
        "match_all": {}
      },
      "script": {
        "lang": "hello_world",
        "source": "custom_score",
        "params": {
          "rating_threshold": 3.5,
          "rating_boost": 1.2,
          "price_threshold": 25.0,
          "cheap_boost": 2.0,
          "expensive_penalty": 0.5,
          "out_of_stock_penalty": 0.2
        }
      }
    }
  }
}

Explanation of Parameters:

rating_threshold: 3.5 – Includes more moderately rated products.
rating_boost: 1.2 – Smaller positive impact for meeting rating.
price_threshold: 25.0 – Marks very cheap items.
cheap_boost: 2.0 – Strong push for clearance deals.
expensive_penalty: 0.5 – Heavy penalty for high-cost items.
out_of_stock_penalty: 0.2 – Medium penalty for unavailable items.

Here, anything above $25 is considered expensive and heavily penalized (0.5 multiplier), encouraging cheaper items to rise to the top. Highly-rated isn’t as important (threshold 3.5 and only 1.2x boost), reflecting that during clearance, price and availability might matter more.

By adjusting parameters in this way, you can reuse the same plugin for very different ranking behaviors. Enterprise architects can define a few parameter sets (perhaps stored in the application or a config file) for various situations (seasonal promotions, different markets, etc.), and developers can apply them as needed in queries.

References

Alexey Vidanov - A simple “Hello World” script plugin for OpenSearch Template in Github for the Amazon OpenSearch Service managed domain, written in Java. A great starting point if you want to learn how to create and integrate custom script plugins into your OpenSearch cluster.
Amitai Stern – “Taking the Leap: My First Steps in OpenSearch Plugins” (Logz.io Blog) – Introduction to building a simple OpenSearch REST plugin, with prerequisites like Java and Gradle and step-by-step examples of a “Hello World” plugin.
Amazon AWS – “Amazon OpenSearch Service now supports Custom Plugins” (Nov 21, 2024) – AWS announcement of custom plugin support in the managed service, including the motivation for custom plugins and the scope of supported plugin types.
OpenSearch Project – “https://opensearch.org/blog/plugins-intro/” (Dec 2, 2021) – OpenSearch official blog post explaining the plugin architecture, how plugins are installed and loaded, and the role of the Security Manager and plugin policy files.
OpenSearch Forum – “Set up communication with external service in OpenSearch plugin” (Discussion, May 2025) – A community discussion highlighting the challenges of making external network calls from within a plugin (SecurityManager restrictions and potential workarounds).
OpenSearch Plugin Template (GitHub) – The official OpenSearch plugin template repository, useful as a starting point for new plugins. It contains the boilerplate code and files needed for a basic plugin project.

Document Versioning in Amazon OpenSearch Service: OpenSearch as the Source of Truth. Part 3

Alexey Vidanov — Fri, 11 Apr 2025 20:39:05 +0000

In our previous discussion, we emphasized using a primary database as the source of truth, with OpenSearch serving as a search layer. However, certain scenarios necessitate managing document versioning directly within OpenSearch. This article explores strategies for handling document versioning in OpenSearch.

1. Two-Indices Approach

One effective method for managing document versioning involves using two separate indices:

1. Immutable Index:

Purpose: Stores every document version as an immutable record, providing a complete audit trail.
Advantage: Ensures that no version is overwritten, which is crucial for compliance and historical analysis.

2. Search Interface Index:

Purpose: Contains only the latest version of each document.
Advantage: Optimized for fast retrieval and efficient queries, as it reduces the amount of data to search through.

Trade-Off: While this dual-index method simplifies compliance and auditability, it significantly increases data storage and indexing operations. Maintaining two indices means higher ingestion costs, increased storage consumption, and more complex query execution, as both indices must remain synchronized.

2. Single-Index Approach for Versioned Documents in OpenSearch

When handling immutable documents with versioning in OpenSearch, a key challenge is ensuring search results reflect only the latest document versions while preserving older content for historical reference. Instead of modifying indices or adding flags like is_latest, we can achieve this with a single optimized query that:

Finds documents where the search term appears in either the latest (searchableText) or previous versions (oldVersionsText).
Excludes outdated documents where the term appears only in oldVersionsText.
Ensures that only the latest document per relationId is returned.

Index Structure and Data Handling

Index Name: test_index

Stored Fields:

relationId (keyword) – Groups multiple versions of a document.
searchableText (text) – Stores the most recent searchable content.
oldVersionsText (text) – Stores previous versions of the content.
update_time (date) – Timestamp of the document's last update.

How Data is Managed:

Document Updates: When a document is updated, a new version is inserted. The previous version’s content is moved to oldVersionsText.
Determining Latest Version: The update_time field is used to identify the most recent version.

Important Consideration: Storing older versions in every document increases the index size significantly. Over time, this can impact performance and storage costs. This method, while effective in some scenarios, introduces a multi-step query, which may become a performance bottleneck at scale.

Why a Refined Query is Necessary

If we only search in searchableText, we may miss relevant results because the latest version might not contain the search term, while an older version does.

For example:

A document initially contains “OpenSearch performance optimization” in searchableText.
Later, the document is updated to “OpenSearch advanced techniques”, moving the previous text to oldVersionsText.
A search for “performance optimization” would only find the outdated document unless we refine the query.

Optimized Query: How It Works

Searches in searchableText and oldVersionsText.
Ensures that if the search term appears only in oldVersionsText, the outdated document is excluded.
Retrieves only the most recent version of each document.

Step-by-Step Guide

Step 1: Create the Index

PUT test_index
{
  "mappings": {
    "properties": {
      "relationId": {"type": "keyword"},
      "latestContent": {"type": "text"},
      "oldVersionsText": {"type": "text"},
      "update_time": {"type": "date"}
    }
  }
}

Step 2: Insert Sample Documents

POST test_index/_bulk
{"index": {"_id": "1"}}
{"relationId": "doc1", "latestContent": "OpenSearch advanced techniques", "oldVersionsText": ["OpenSearch performance optimization"], "update_time": "2025-03-12T12:00:00Z"}
{"index": {"_id": "2"}}
{"relationId": "doc2", "latestContent": "OpenSearch index tuning", "oldVersionsText": [], "update_time": "2025-03-12T13:00:00Z"}
{"index": {"_id": "3"}}
{"relationId": "doc1", "latestContent": "OpenSearch performance optimization", "oldVersionsText": [], "update_time": "2025-03-11T10:00:00Z"}

Step 3: Execute the Optimized Query

This query ensures that:

The search term appears in searchableText or oldVersionsText.
Documents where the term appears only in oldVersionsText are excluded.
Only the latest document version per relationId is returned.

GET test_index/_search
{
  "query": {
    "bool": {
      "should": [
        { "match": { "latestContent": "performance optimization" } },
        { "match": { "oldVersionsText": "performance optimization" } }
      ],
      "minimum_should_match": 1,
      "must_not": {
        "bool": {
          "must": [
            { "match": { "oldVersionsText": "performance optimization" } },
            { "bool": { "must_not": { "match": { "latestContent": "performance optimization" } } } }
          ]
        }
      }
    }
  },
  "sort": [{ "update_time": "desc" }]
}

How This Query Works

Dual-Field Coverage: The should clause ensures that a document is considered if it contains the term "performance optimization" in either the latest content (latestContent) or in the older versions (oldVersionsText). This guarantees that we capture any document that might be relevant regardless of which field holds the term.
Exclusion of Outdated Matches: The must_not clause is crucial—it specifically excludes documents where the term appears only in oldVersionsText. This means that if a document's latest version does not contain the search term, even if an older version does, that document will not be returned. The inner structure checks for documents matching in oldVersionsText but missing a match in latestContent. Only those documents are filtered out.
Sorting by Update Time: The sort parameter orders the results by update_time in descending order, ensuring that the most recent versions are prioritized.

The Key Points

Retrieves all relevant documents — Ensures we don’t miss documents where the term appears in both searchableText and oldVersionsText.
Prevents returning outdated documents alone — If the term appears only in an old version, we exclude it.
No need for **is_latest** flags or index modifications – Simplifies indexing by handling filtering at the query level.
Balances accuracy and efficiency — Uses OpenSearch’s filtering capabilities without extra processing.

Considerations and Trade-Offs

Index Size Impact: Storing previous versions in oldVersionsTextincreases the index size over time. If document updates are frequent, this may require a cleanup strategy.
Query Complexity: This approach involves multiple steps in query execution (searching in both fields, filtering, and sorting), which could lead to performance
Scalability: For high-update environments or large-scale deployments, consider periodic cleanup strategies or even alternative architectures (e.g., the two-indices approach) to maintain performance.

Conclusion

Managing document versioning directly within OpenSearch is inherently complex. While OpenSearch can serve as the source of truth for versioned documents, it isn’t the optimal standalone solution for all production environments. There’s no one-size-fits-all answer; as many experienced consultants say, “it depends.” By deeply understanding the trade-offs, you can select and tailor the approach that best fits your specific use case.

This refined single-index strategy, leveraging the optimized query above, provides a powerful means to retrieve only the latest relevant document versions while still maintaining a comprehensive history of changes.

Document Versioning in OpenSearch: Database as the Source of Truth. Part 2

Alexey Vidanov — Fri, 11 Apr 2025 20:37:19 +0000

Best Approach: Database as the Source of Truth & OpenSearch as a Search Layer

Introduction

A key consideration in this strategy is document versioning. OpenSearch is not designed to maintain a history of document versions, and its handling of updates introduces important trade-offs. By leveraging a database for version control and OpenSearch for fast retrieval, applications can ensure both accuracy and performance.

Why Separate the Search Layer from the Database?

A database and OpenSearch serve different purposes, and using them correctly results in a more efficient system:

Data integrity and versioning: A relational or NoSQL database ensures strict data consistency, transaction safety, and historical tracking. This is essential for applications where version control is required.
Search performance: OpenSearch optimizes full-text search and fast lookups but lacks strong consistency mechanisms and built-in version tracking.
Scalability: Keeping OpenSearch lightweight by only storing relevant indexed data makes scaling search clusters more manageable.
Backups and restoration: Since OpenSearch is not the source of truth, it can be entirely recreated from the database without requiring complex backup strategies.

How to Store and Organize Data Effectively

Versioning and OpenSearch’s Update Model

OpenSearch does not truly update documents in place. Instead, each update:

Creates a new document version.
Updates the index reference.
Deletes the older version asynchronously.

This means:

The latest version is always accessible through indexing mechanisms.
A slight delay in search availability is introduced, dependent on refresh_interval, cluster performance, and index size.
Storing multiple versions inside OpenSearch leads to unnecessary storage overhead and increased indexing complexity.

Best Practices for Versioning and Indexing

Store only the latest version of a document in OpenSearch.
Keep a full version history in the database to ensure traceability and compliance.
For real-time accuracy, use backend logic to verify OpenSearch results against the database before presenting data to the user.

Example: Using DynamoDB and a Lambda Indexer

A common approach for handling versioning and indexing efficiently is using Amazon DynamoDB as the primary database and an AWS Lambda function to update OpenSearch asynchronously.

DynamoDB as the Source of Truth:

Stores all document versions, maintaining full historical records.
Uses DynamoDB Streams to capture item modifications in real time.

2. Lambda Indexer for OpenSearch:

A Lambda function is triggered by DynamoDB Streams whenever an item is modified.
The function extracts the latest version and updates OpenSearch via the OpenSearch API.
Ensures OpenSearch only contains the most recent document, preventing unnecessary versioning overhead.

3. Handling Deletes and Expired Versions:

The Lambda function removes outdated versions from OpenSearch while retaining historical versions in DynamoDB.
Ensures efficient query performance without cluttering OpenSearch with redundant versions.

Example Code for a Lambda Indexer

import json
import boto3
from opensearchpy import OpenSearch, RequestsHttpConnection
from requests_aws4auth import AWS4Auth

# Configuration: update these with your details.
region = 'your-region'  # e.g., 'us-east-1'
host = 'your-opensearch-domain'  # e.g., 'search-mydomain.us-east-1.es.amazonaws.com'
index_name = 'your-index-name'

# Set up AWS authentication for SigV4 signing.
credentials = boto3.Session().get_credentials()
awsauth = AWS4Auth(
    credentials.access_key,
    credentials.secret_key,
    region,
    'es',
    session_token=credentials.token
)

# Initialize the OpenSearch client.
client = OpenSearch(
    hosts=[{'host': host, 'port': 443}],
    http_auth=awsauth,
    use_ssl=True,
    verify_certs=True,
    connection_class=RequestsHttpConnection
)

def lambda_handler(event, context):
    for record in event["Records"]:
        if record["eventName"] in ["INSERT", "MODIFY"]:
            document = record["dynamodb"]["NewImage"]
            doc_id = document["id"]["S"]
            data = {
                "id": doc_id,
                "title": document["title"]["S"],
                "content": document["content"]["S"],
                "timestamp": document["timestamp"]["S"]
            }
            response = client.index(index=index_name, id=doc_id, body=data)
            print("Updated document:", response)
        elif record["eventName"] == "REMOVE":
            doc_id = record["dynamodb"]["Keys"]["id"]["S"]
            response = client.delete(index=index_name, id=doc_id)
            print("Deleted document:", response)

Handling Real-Time Accuracy

OpenSearch’s eventual consistency model means changes are not immediately available for search.
If exact real-time accuracy is required, consider implementing backend logic that cross-checks OpenSearch results against the database.
The trade-off is complexity versus performance: OpenSearch provides ultra-fast queries, but perfect real-time accuracy requires extra processing steps.

Example Scenarios for Reducing Update Frequency

Reducing the number of updates to OpenSearch can significantly improve performance. Here are some real-world strategies:

Shop Inventory Search: Instead of storing the exact number of available products in OpenSearch, categorize availability into broader ranges like:

“Out of Stock”
“Limited Stock”
“Moderate Stock”
“Plentiful”

This reduces the frequency of updates and indexing workload.

Dynamic Pricing Optimization: Instead of storing the exact price of each item, group prices into predefined buckets that allow efficient filtering:

50 → Represents prices between 0-5
100 → Represents prices between 50-100
200 → Represents prices between 100-200
500 → Represents prices between 200-500

This method significantly reduces indexing load while maintaining the ability to perform efficient range-based searches in OpenSearch. Filtering documents based on these predefined price groups is computationally inexpensive and does not require constant reindexing when prices fluctuate.

Example: OpenSearch Index Mapping and Data Storage

Index Mapping:

{
  "mappings": {
    "properties": {
      "id": { "type": "keyword" },
      "title": { "type": "text" },
      "content": { "type": "text" },
      "timestamp": { "type": "date" },
      "stock_level": { "type": "keyword" },
      "price_range": { "type": "integer" }
    }
  }
}

Storing a Document:

{
  "id": "12345",
  "title": "High-Performance Laptop",
  "content": "A powerful laptop with 16GB RAM and 512GB SSD.",
  "timestamp": "2024-03-17T12:00:00Z",
  "stock_level": "Moderate Stock",
  "price_range": 200
}

Benefits of This Approach

Minimizes Indexing Overhead: Price changes do not require frequent document updates.
Efficient Filtering: OpenSearch can efficiently retrieve documents based on predefined price ranges without additional computation.
Scalability: Suitable for large datasets with frequently changing prices and inventory levels.

Structuring Data for Performance and Scalability

OpenSearch benefits from a flat, denormalized structure:

Avoid deeply nested objects that require complex queries.
Eliminate the need for multiple joins across indices by storing relevant information in a single index document.
Keeping data denormalized reduces indexing complexity and improves search performance.

Backup and Restoration Strategies

A key advantage of this approach is that OpenSearch can be entirely recreated from the database:

If an OpenSearch cluster is lost, documents can be reindexed from the database without risk of data loss.
This minimizes the need for frequent OpenSearch snapshots, simplifying disaster recovery and reducing operational costs.

Key Benefits of This Approach

Improved Data Consistency: The database remains the single source of truth.
Optimized Performance: OpenSearch is leaner, avoiding unnecessary writes and updates.
Scalability: OpenSearch clusters remain manageable as they only store relevant indexed data.
Simplified Maintenance: Easier disaster recovery since OpenSearch can be rebuilt from the database.
Better Version Control: The database maintains a full history of document versions, while OpenSearch serves only the latest, reducing storage bloat and complexity.

This method is strongly recommended for applications that demand precise version control and rapid search functionality.

The subsequent sections explore alternative strategies where OpenSearch itself must manage document versioning.

Understanding Document Versioning in OpenSearch. Part 1

Alexey Vidanov — Fri, 11 Apr 2025 20:35:37 +0000

What is Document Versioning?

Document versioning refers to the practice of tracking and managing multiple versions of a document over time. In many applications, document changes need to be recorded rather than overwritten, ensuring historical integrity and compliance with regulations. Versioning is critical in industries such as finance, healthcare, legal, and content management, where keeping an accurate record of past document states is essential for audits, accountability, and compliance.

Understanding Versioning and Immutable Storage

Versioning Approaches in Traditional Systems

Traditional databases and content management systems typically handle versioning through methods such as:

Row-based historical tracking: Storing each document version with timestamps and unique identifiers in database tables.
Event sourcing: Capturing all changes as immutable events in an append-only log.
Snapshot and delta storage: Storing periodic full snapshots with incremental changes recorded between versions.

However, Amazon OpenSearch Service does not natively support these traditional versioning mechanisms.

Immutable Storage and Compliance in OpenSearch

Immutable document storage, where records cannot be modified or deleted, is essential for compliance with regulations such as GDPR, HIPAA, and SOC 2. Immutable storage ensures data integrity, auditability, and tamper resistance, particularly important for regulated sectors like healthcare and finance.

OpenSearch Compliance Capabilities on AWS

Amazon OpenSearch Service offers multiple compliance-supporting features:

Fine-Grained Access Control: Role-Based Access Control (RBAC) and AWS IAM integration restrict modifications to authorized users only.
Audit Logging: Built-in audit logs integrated with AWS CloudTrail provide comprehensive records for audit purposes.
Managed Snapshots: Automated incremental backups allow immutable point-in-time data recovery.
Compliance Certifications: Officially audited and certified compliance with standards including HIPAA, SOC 2, ISO, and FedRAMP.

Applications leveraging OpenSearch typically adopt a “write-once” approach, preserving historical records unmodified after creation.

AWS Services Complementing OpenSearch for Versioning and Compliance

AWS provides integrated solutions that complement OpenSearch by enhancing document versioning, data retention, and compliance:

Amazon S3 with Versioning:

Stores each version of documents, ensuring robust data integrity.
Provides lifecycle policies for automatic management of document versions.
Integrates with AWS Backup for efficient archival and retention.

Amazon DynamoDB for Immutable Data Storage:

Maintains historical records with strong consistency through timestamped entries.
Serves as an authoritative source of truth, seamlessly integrated with OpenSearch for indexing and retrieval.

AWS Backup and AWS Audit Manager:

Automates comprehensive backups, supporting compliance with regulations such as GDPR, HIPAA, and SOC 2.

By combining OpenSearch or Amazon OpenSearch Service with AWS managed storage, security, and compliance services, organizations achieve secure, compliant, and efficient document versioning and storage.

OpenSearch as a Search Layer, Not a Versioning System

OpenSearch is designed primarily as a powerful distributed search and analytics engine rather than a document version control system. It includes a built-in _version field intended primarily for optimistic concurrency control. This built-in versioning mechanism increments a version number upon each document update but does not retain historical document states.

Applications needing comprehensive audit trails, compliance tracking, or historical data retrieval must implement custom versioning approaches. The recommended best practice is to utilize OpenSearch strictly as a high-performance search and analytics layer while maintaining the authoritative historical data in a separate, persistent database designed specifically for version control.

Why OpenSearch is Not Optimized for Versioning

Unlike traditional databases, OpenSearch follows a distributed architecture that makes version tracking challenging:

Eventual Consistency: Updates are indexed asynchronously, meaning that documents may not appear updated in search results immediately.
Sharding Complexity: Data is split across multiple shards, making atomic updates and transactions difficult to implement at scale.
Optimized for Read Performance: OpenSearch is built for fast, scalable search operations, not transactional integrity.
No Native Version History: The _version field only tracks the latest version, with no capability to retrieve past document states.

For example, if an application tracks legal contracts or medical records, simply relying on OpenSearch’s _version field would not provide a verifiable audit history—previous versions would be irreversibly lost.

Using the Cloud for Compliance and Versioning

For organizations that need regulatory compliance, security, and versioning best practices, leveraging cloud services is the most effective approach. Cloud providers like AWS offer managed solutions that help achieve compliance while maintaining performance and scalability.

Understanding OpenSearch’s `_version` Field

While OpenSearch assigns each document a _version, this does not function like traditional version control systems such as Git or database transaction logs. Instead, _version is used to prevent conflicts when multiple clients attempt to update the same document.

How `_version` Works

A document is indexed for the first time → _version = 1
A client updates the document → _version increments (_version = 2)
Another update occurs → _version = 3
However, previous versions are overwritten, not stored.

If two clients try to update the same document simultaneously, OpenSearch can reject changes that do not match the expected _version. This ensures concurrent updates do not overwrite each other, but it does not provide a way to retrieve historical versions.

Example: Updating a Document

PUT /my_index/_doc/1
{
  "title": "First Version",
  "content": "This is the first version of the document."
}

PUT /my_index/_doc/1
{
  "title": "Updated Version",
  "content": "This is the updated version of the document."
}

After the second PUT request, the original version is completely replaced. The _version number increases, but the old data is lost.

What happens if you try to retrieve version 1? Unlike databases that store historical states, OpenSearch only retains the latest version. Querying for an old version (e.g., GET /my_index/_doc/1?version=1) will not work—only the most recent document is available.

Conclusion: Key Takeaways from Document Versioning in OpenSearch

In this article, we explored the challenges and solutions for document versioning in OpenSearch. Key takeaways include:

OpenSearch is not a versioning system; it is optimized for search, not for maintaining historical records.
The built-in _version field is only for optimistic concurrency control and does not store historical versions.
Applications requiring audit trails, compliance, and historical trackingshould maintain a separate source of truth, such as a database or object storage.
AWS provides robust tools for compliance, including Amazon S3 with versioning, DynamoDB, OpenSearch Service with IAM control, and AWS Backup.
The best strategy for OpenSearch versioning depends on the use case.

With this foundation, we are now ready to explore specific strategies for managing document versions in OpenSearch. Stay tuned for the next article: **Using a Database as the Source of Truth: Best Practices for OpenSearch Integration.**

API Gateway and Lambda Throttling with Terraform. Part 2

Alexey Vidanov — Mon, 14 Oct 2024 12:26:26 +0000

In the previous post we covered the basics of setting up throttling for your API Gateway and Lambda functions. In this follow-up, we’ll take it to the next level, adding budget controls, time-based throttling adjustments, and AWS WAF security integration to safeguard your API while optimizing both performance and cost-efficiency.

1. Budget and Billing Alerts

Managing cloud expenses is essential to avoiding surprises, especially when usage spikes unexpectedly. AWS Budgets allows you to set up monthly spending limits and receive alerts before overspending occurs.

Example: Setting up Budget Alerts

resource "aws_budgets_budget" "monthly_budget" {
  name         = "monthly-budget-${var.environment}"
  budget_type  = "COST"
  time_unit    = "MONTHLY"
  limit_amount = "1000"
  limit_unit   = "USD"

  notification {
    comparison_operator        = "GREATER_THAN"
    threshold                  = 80
    threshold_type             = "PERCENTAGE"
    notification_type          = "FORECASTED"
    subscriber_email_addresses = ["alert@example.com"]
  }

  notification {
    comparison_operator        = "GREATER_THAN"
    threshold                  = 100
    threshold_type             = "PERCENTAGE"
    notification_type          = "ACTUAL"
    subscriber_email_addresses = ["alert@example.com"]
  }
}

You can also configure alerts to trigger responses in other AWS services, such as SNS topics or CloudWatch Alarms, to better manage your Lambda functions during traffic spikes.

# SNS Topic for Alerts
resource "aws_sns_topic" "budget_alerts_topic" {
  name = "budget-alerts-topic-${var.environment}"
}

# Subscribe an email address to the SNS topic
resource "aws_sns_topic_subscription" "email_subscription" {
  topic_arn = aws_sns_topic.alarms.arn
  protocol  = "email"
  endpoint  = var.alarms_email
}

# Example CloudWatch Alarm for Lambda Throttling (to control costs)
resource "aws_cloudwatch_metric_alarm" "lambda_throttles" {
  alarm_name          = "lambda-throttles-${aws_lambda_function.example.function_name}"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = 2
  metric_name         = "Throttles"
  namespace           = "AWS/Lambda"
  period              = 300
  statistic           = "Sum"
  threshold           = 5
  alarm_description   = "Alert for Lambda throttling due to budget constraints"
  alarm_actions       = [aws_sns_topic.budget_alerts_topic.arn]

  dimensions = {
    FunctionName = aws_lambda_function.example.function_name
  }
}

2. Time-Based Throttling Adjustments

For high-traffic applications, one-size-fits-all throttling can be inefficient. You can dynamically adjust API Gateway throttling limits during peak and off-peak hours using AWS EventBridge and Lambda.

Example: Time-Based Throttling with Lambda and EventBridge

Below is the Python code that dynamically adjusts API Gateway throttling limits:

import boto3
import os

def handler(event, context):
    api_gateway = boto3.client('apigateway')

    api_id = os.environ['API_ID']
    stage = os.environ['STAGE']
    burst_limit = int(os.environ['BURST_LIMIT'])
    rate_limit = int(os.environ['RATE_LIMIT'])

    # Update method settings with new throttling limits
    api_gateway.update_stage(
        restApiId=api_id,
        stageName=stage,
        patchOperations=[
            {
                'op': 'replace',
                'path': '/*/*/throttling/burstLimit',
                'value': str(burst_limit)
            },
            {
                'op': 'replace',
                'path': '/*/*/throttling/rateLimit',
                'value': str(rate_limit)
            }
        ]
    )

    return {
        'statusCode': 200,
        'body': f'Throttling updated: Burst={burst_limit}, Rate={rate_limit}'
    }

Lambda Creation in Terraform

To automate the creation of the Lambda function for adjusting throttling limits, use the following Terraform configuration:

variable "api_throttling_configs" {
  type = map(object({
    burst_limit = number
    rate_limit  = number
  }))
  default = {
    peak = {
      burst_limit = 5000
      rate_limit  = 2000
    }
    off_peak = {
      burst_limit = 1000
      rate_limit  = 500
    }
  }
}


resource "aws_lambda_function" "adjust_throttling_peak" {
  function_name = "adjust_throttling_peak_${var.environment}"
  handler       = "adjust_throttling.handler"
  runtime       = "python3.11"
  role          = aws_iam_role.lambda_exec.arn
  filename      = "adjust_throttling_peak.zip"

  environment {
    variables = {
      API_ID      = aws_api_gateway_rest_api.example.id
      STAGE       = aws_api_gateway_stage.example.stage_name
      BURST_LIMIT = var.api_throttling_configs.peak.burst_limit
      RATE_LIMIT  = var.api_throttling_configs.peak.rate_limit
    }
  }
}

resource "aws_lambda_function" "adjust_throttling_off_peak" {
  function_name = "adjust_throttling_off_peak_${var.environment}"
  handler       = "adjust_throttling.handler"
  runtime       = "python3.11"
  role          = aws_iam_role.lambda_exec.arn
  filename      = "adjust_throttling_off_peak.zip"

  environment {
    variables = {
      API_ID      = aws_api_gateway_rest_api.example.id
      STAGE       = aws_api_gateway_stage.example.stage_name
      BURST_LIMIT = var.api_throttling_configs.off_peak.burst_limit
      RATE_LIMIT  = var.api_throttling_configs.off_peak.rate_limit
    }
  }
}

Here's a more understandable version:

Automating Time-Based Adjustments

To automatically adjust settings based on the time of day (for example, peak hours vs. off-peak hours), you can use EventBridge Rules to trigger Lambda functions at specific times. This way, your system can automatically adjust to handle more traffic during peak times and reduce capacity during off-peak hours.

In this example, we set up rules that run at 8:00 AM (peak hours) and 8:00 PM (off-peak hours). Here's how it works:

Peak Hours Rule: This triggers at 8:00 AM to increase capacity during busy times.
Off-Peak Hours Rule: This triggers at 8:00 PM to reduce capacity during slower times.

Here’s how to set it up:

# Define a rule for peak hours (8:00 AM)
resource "aws_cloudwatch_event_rule" "peak_hours_rule" {
  name                = "peak-hours-rule"
  schedule_expression = "cron(0 8 * * ? *)"  # Adjust to your peak hours
}

# Define a rule for off-peak hours (8:00 PM)
resource "aws_cloudwatch_event_rule" "off_peak_hours_rule" {
  name                = "off-peak-hours-rule"
  schedule_expression = "cron(0 20 * * ? *)"  # Adjust to your off-peak hours
}

# Set the target for peak hours (a Lambda function to handle peak traffic)
resource "aws_cloudwatch_event_target" "peak_hours_target" {
  rule      = aws_cloudwatch_event_rule.peak_hours_rule.name
  target_id = "adjust_throttling_peak"
  arn       = aws_lambda_function.adjust_throttling_peak.arn
}

# Set the target for off-peak hours (a Lambda function to handle lower traffic)
resource "aws_cloudwatch_event_target" "off_peak_hours_target" {
  rule      = aws_cloudwatch_event_rule.off_peak_hours_rule.name
  target_id = "adjust_throttling_off_peak"
  arn       = aws_lambda_function.adjust_throttling_off_peak.arn
}

This setup automatically changes your system’s capacity based on time. At 8:00 AM, the system prepares for heavy traffic, and at 8:00 PM, it scales down for lighter usage.

3. Adding AWS WAF for Extra Protection

Integrating AWS WAF (Web Application Firewall) into your API Gateway helps protect it from malicious traffic, such as bots or potential DDoS (Distributed Denial of Service) attacks. It acts as a security layer that filters out harmful requests before they reach your API.

Example: Blocking IPs and Controlling Bots with WAF

This example shows how to set up a WAF rule that protects your API Gateway by blocking bot traffic:

resource "aws_wafv2_web_acl" "api_gateway_acl" {
  name  = "api-gateway-waf-acl"
  scope = "REGIONAL"  # Use "REGIONAL" to protect your regional API Gateway

  # Default action to allow all requests unless a rule blocks them
  default_action {
    allow {}
  }

  # Enable visibility so you can monitor activity using CloudWatch metrics
  visibility_config {
    cloudwatch_metrics_enabled = true
    metric_name                = "apiGatewayAcl"
    sampled_requests_enabled   = true
  }

  # Bot Protection Rule to block bad bot traffic
  rule {
    name     = "BotProtectionRule"  # Name of the rule
    priority = 2  # Order of execution for this rule (lower number = higher priority)

    # No override action means the default allow/block settings apply
    override_action {
      none {}
    }

    # Managed rule group to use AWS's pre-built bot control rules
    statement {
      managed_rule_group_statement {
        name        = "AWSManagedRulesBotControlRuleSet"
        vendor_name = "AWS"  # AWS provides these bot protection rules
      }
    }

    # Enable visibility for this specific rule as well
    visibility_config {
      cloudwatch_metrics_enabled = true
      metric_name                = "botProtection"
      sampled_requests_enabled   = true
    }
  }
}

This setup adds protection to your API by detecting and blocking bot traffic, while also giving you the ability to monitor suspicious activity via CloudWatch metrics.

Conclusion

By incorporating Budget Alerts, Time-Based Throttling Adjustments, and WAF Integration, you're ensuring that your API is not only efficient and cost-effective but also secure. These advanced practices complement each other, providing a well-rounded solution for managing traffic, costs, and security in a scalable AWS environment.

Next Steps

Monitoring and Fine-Tuning: After deploying these configurations, it's crucial to monitor their performance. Use CloudWatch metrics and logs to verify that throttling adjustments and security rules are working as expected.
Performance Testing: Perform load testing during both peak and off-peak hours to ensure your dynamic throttling setup aligns with your traffic patterns.
Further Optimization: Explore other AWS services like AWS Shield for advanced DDoS protection and AWS X-Ray for tracing API Gateway requests to identify potential bottlenecks.

By continually monitoring and adjusting, you’ll be able to maintain an API infrastructure that scales with traffic, optimizes costs, and maintains high levels of security.

Amazon API Gateway and AWS Lambda Throttling with Terraform. Part 1

Alexey Vidanov — Fri, 11 Oct 2024 09:53:29 +0000

In today's cloud-native world, effectively managing API and serverless function performance is crucial for building reliable and cost-effective applications. This guide explores advanced throttling techniques for AWS API Gateway and Lambda using Terraform, incorporating best practices from the AWS Well-Architected Framework and real-world implementation patterns.

Why Throttling Matters

When building cloud-native applications, it's easy to focus on the business logic and forget key infrastructure components such as throttling and usage limits. However, this oversight can lead to unpredictable performance, increased costs, and even service outages. Setting the right throttling limits ensures that your applications:

Adapt to varying load patterns and sudden traffic spikes
Protect backend services from overload and cascading failures
Optimize costs across different environments while maintaining service quality
Provide meaningful monitoring and alerting for proactive management
Support different user tiers and business requirements effectively
Enable graceful degradation during high-load scenarios

In real-world projects, teams often realize the importance of throttling limits only after encountering performance or cost issues. It can be challenging to set proper limits without historical data, but this guide provides you with the framework to get those metrics in place, simplifying budgeting and operational planning. Furthermore, proper throttling configuration serves as a critical defense mechanism against denial-of-service attacks, whether intentional or accidental.

Implementation Overview

Let's dive into a comprehensive throttling implementation that addresses these needs using Terraform. We'll build a solution that's both flexible and production-ready.

1. Dynamic Configuration Management

One of the key challenges in multi-environment setups is managing environment-specific configurations. In the example below, we set up environment-specific throttling limits for API Gateway:

variable "environment" {
  type        = string
  description = "Environment name (e.g., dev, staging, prod)"
}

variable "api_throttling_configs" {
  type = map(object({
    burst_limit = number
    rate_limit  = number
  }))
  default = {
    dev = {
      burst_limit = 1000
      rate_limit  = 500
    }
    prod = {
      burst_limit = 5000
      rate_limit  = 1000
    }
  }
}

This setup allows you to manage different limits for development and production environments. For example, the dev environment has lower limits, ensuring that your resources are not overwhelmed during testing, while prod has higher limits to handle real-world traffic.

2. Lambda Configuration

AWS Lambda has a default maximum concurrency limit of 1,000 concurrent executions per account per region. You can control concurrency at the function level to avoid unplanned spikes and protect backend services. Here's how to set concurrency limits for Lambda:

resource "aws_lambda_function" "example" {
  function_name = "my_lambda_function_${var.environment}"
  ... // other lambda function configurations
  reserved_concurrent_executions = lookup(var.lambda_concurrency_limits, var.environment, 100)
}

By setting reserved_concurrent_executions, we control how many instances of the Lambda function can run simultaneously, protecting backend services from excessive traffic. The default value in this example is 100 concurrent executions, but this can be adjusted depending on the environment.

Real-World Observation

A common oversight in development is forgetting to configure these limits early on. Teams often realize the importance of limiting Lambda concurrency when they see unexpected bills or performance degradation. This configuration helps prevent such scenarios by establishing clear boundaries from the start.

3. Comprehensive Monitoring

Monitoring is critical for ensuring that throttling is functioning as expected. Use Amazon CloudWatch alarms to proactively track throttling metrics:

resource "aws_cloudwatch_metric_alarm" "lambda_throttles" {
  alarm_name          = "lambda-throttles-${aws_lambda_function.example.function_name}"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = 2
  metric_name         = "Throttles"
  namespace           = "AWS/Lambda"
  period              = 300
  statistic           = "Sum"
  threshold           = 5
  alarm_description   = "Lambda function throttling detected"
  alarm_actions       = [aws_sns_topic.alerts.arn]

  dimensions = {
    FunctionName = aws_lambda_function.example.function_name
  }
}

4. Amazon API Gateway Configuration

Amazon API Gateway can experience heavy loads during peak traffic times. To prevent backend services from being overwhelmed, we use throttling limits:

resource "aws_api_gateway_method_settings" "example" {
  rest_api_id = aws_api_gateway_rest_api.example.id
  stage_name  = aws_api_gateway_stage.example.stage_name
  method_path = "*/*"

  settings {
    metrics_enabled         = true
    logging_level          = "INFO"
    data_trace_enabled     = true
    throttling_burst_limit = lookup(var.api_throttling_configs[var.environment], "burst_limit", 1000)
    throttling_rate_limit  = lookup(var.api_throttling_configs[var.environment], "rate_limit", 500)
  }
}

5. Monitoring with Amazon CloudWatch Dashboards

Once the throttling is configured, it's important to visualize the data to make informed decisions. Amazon CloudWatch dashboards offer a great way to monitor both Amazon API Gateway and AWS Lambda throttling:

resource "aws_cloudwatch_dashboard" "throttling_monitoring" {
  dashboard_name = "throttling-monitoring-${var.environment}"

  dashboard_body = jsonencode({
    widgets = [
      {
        type   = "metric"
        width  = 12
        height = 6
        properties = {
          metrics = [
            ["AWS/ApiGateway", "ThrottleCount", "ApiName", aws_api_gateway_rest_api.example.name],
            ["AWS/Lambda", "Throttles", "FunctionName", aws_lambda_function.example.function_name]
          ]
          period = 300
          stat   = "Sum"
          region = var.aws_region
          title  = "Throttling Overview"
        }
      }
    ]
  })
}

6. Cost Management with Usage Plans

Amazon API Gateway usage plans are a practical way to control costs and ensure that API consumers do not abuse the service:

resource "aws_api_gateway_usage_plan" "tiered" {
  name = "tiered-usage-plan-${var.environment}"

  api_stages {
    api_id = aws_api_gateway_rest_api.example.id
    stage  = aws_api_gateway_stage.example.stage_name
  }

  quota_settings {
    limit  = 10000
    period = "MONTH"
  }

  throttle_settings {
    burst_limit = lookup(var.api_throttling_configs[var.environment], "burst_limit", 1000)
    rate_limit  = lookup(var.api_throttling_configs[var.environment], "rate_limit", 500)
  }

  tags = {
    Environment = var.environment
    CostCenter  = "API-Gateway"
  }
}

Without proper throttling in place, API Gateway costs can spiral out of control. Budgeting becomes much easier when you set usage plans early and monitor actual consumption via dashboards.

7. Budget and Billing Alerts

An important aspect of managing API Gateway and Lambda throttling is cost control. By setting up budget and billing alerts, you can monitor and track usage costs to avoid unexpected charges. Here’s how you can approach it:

AWS Budgets: Set a monthly budget for API Gateway and Lambda usage, and configure notifications to alert you when costs exceed a certain threshold. This allows proactive management of expenses and ensures that your application remains cost-efficient.
Cost Anomaly Detection: Enable AWS Cost Anomaly Detection to spot unusual usage patterns that may indicate misconfigurations or unexpected traffic spikes, helping you address cost-related issues promptly.

These measures, combined with your throttling configurations, provide a robust approach to managing both application performance and cost efficiency.

Testing and Validation

Testing your throttling configurations ensures reliability in production:

Load Testing: Simulate high traffic to verify the throttling limits are being respected, including edge cases and boundary conditions.
Scenario Testing: Test burst traffic and sustained load to validate both limits and system resilience, particularly focusing on recovery patterns.
Monitoring Validation: Ensure your CloudWatch alarms are firing during test scenarios and verify the accuracy of metrics collection.

Best Practices for Production

1. Regular Review

Continuously monitor usage trends and adjust throttling settings as your traffic patterns evolve
Periodically review cost implications of your throttling configurations
Analyze throttling patterns to identify potential optimization opportunities
Consider seasonal variations in traffic when setting limits

2. Documentation

Maintain detailed runbooks for handling throttling-related incidents
Document any configuration changes, including justifications, in a version-controlled manner
Keep a historical record of throttling adjustments and their impacts

3. Compliance

Perform regular audits of your throttling configurations to ensure they meet compliance and security standards
Document throttling decisions as part of your compliance framework
Ensure throttling mechanisms align with SLA commitments

Further Considerations and Possible Ways to Go

As you refine your throttling strategy, here are some additional techniques you can consider:

Budget and Billing Alerts: Set up budget limits and enable cost anomaly detection to avoid unexpected charges.
Time-Based Throttling Adjustments: Use AWS EventBridge to adjust throttling limits during peak hours vs. off-hours to optimize resource allocation.
WAF Integration: Add an extra layer of security by integrating AWS WAF for IP-based throttling, blocking suspicious IP addresses.
Request Validation: Ensure that API requests conform to expected formats to reduce the chances of invalid requests causing backend overload.
Dead Letter Queues (DLQs): Ensure that throttled requests are not lost by sending them to DLQs for later reprocessing.

See the follow up Amazon API Gateway and AWS Lambda Throttling with Terraform. Part 2

Conclusion

Advanced throttling is a critical aspect of modern cloud applications. By implementing these patterns with Terraform, you can create a robust, scalable, and maintainable throttling solution that protects your applications while optimizing costs. The key is to approach throttling as a dynamic system that requires ongoing attention and refinement, rather than a set-and-forget configuration.

Key Reminders:

Adapt and Review: Continuously evaluate and adjust throttling configurations based on real-world usage patterns
Monitor and Alert: Track throttling metrics for actionable insights and maintain comprehensive dashboards
Security and Compliance: Maintain rigorous security checks and documentation while ensuring throttling aligns with business requirements
Performance Balance: Strike the right balance between protection and performance to avoid over-throttling

With the configurations and best practices outlined in this guide, you can ensure your applications are prepared to handle varying traffic loads while maintaining predictable performance and cost efficiency. Remember that throttling is not just about limiting requests – it's about creating a resilient system that can gracefully handle any load condition while protecting your infrastructure and budget.

Comprehensive Guide to AWS AI/ML Services: The Ultimate Decision Maker’s Playbook

Alexey Vidanov — Wed, 09 Oct 2024 11:50:35 +0000

Looking to integrate artificial intelligence and machine learning into your business strategy? AWS has an ever-growing suite of AI/ML services, which can sometimes feel overwhelming to navigate. But don’t worry—this playbook is designed to provide a clear, simple guide to help you find the right tools for your business. Explore how AWS AI/ML services can empower your business and simplify your processes—no expert knowledge needed!

Easy-to-Start Innovations in AWS AI

Amazon PartyRock – Rapid AI Prototyping

Built on Amazon Bedrock, PartyRock is revolutionizing how we experiment with AI. This no-code environment enables quick application development using pre-trained foundation models, making AI accessible to everyone.

Key Features:

Zero coding required
Built-in integration with Bedrock foundation models
Support for Retrieval Augmented Generation (RAG)
Rapid prototyping capabilities

Use Cases:

Testing AI models in a business context
Prototyping AI-driven customer solutions
Automating simple business tasks

Business Benefits:

Speed up innovation cycles
Reduce time to market for AI applications
Democratize AI development for non-technical teams

Amazon Q – Your AI-Powered Assistant

Amazon Q is a generative AI-powered assistant designed to deliver insights, content generation, and task automation across various business functions. It provides immediate, contextual responses by pulling data from your company’s repositories, such as Amazon S3, Google Drive, or Amazon Kendra. It can handle complex queries using natural language and is available in flavors tailored for different use cases.

Flavors of Amazon Q:

Amazon Q Business Supports marketing, HR, and other departments to automate tasks such as generating summaries, creating content, and streamlining workflows.
Amazon Q Developer Offers AI-powered assistance for software development, from generating code to scanning for vulnerabilities and even modernizing legacy code.
Amazon Q in QuickSight Provides insights directly within Amazon QuickSight, allowing business analysts to use natural language to generate dashboards and analyze data.

Amazon Bedrock: Enterprise-Grade Generative AI

Amazon Bedrock provides seamless access to a diverse set of foundation models, empowering businesses to build and scale custom AI solutions with ease. Whether you need advanced text generation, image creation, or specialized AI models, Bedrock offers a flexible, enterprise-ready platform to meet your needs.

Key Features:

Access to popular AI models through a single API
Integration with enterprise data and applications
Customizable models for specific business use cases

Use Cases:

Automating content generation
Enhancing data analytics and reporting
Creating visual content for marketing and design

Business Benefits:

Speed up innovation with cutting-edge AI capabilities
Personalize customer interactions with tailored solutions
Maintain robust security and privacy standards at scale

Amazon Bedrock supports leading models from providers such as AI21 Labs, Anthropic, Mistral, Stability AI, Cohere, and Amazon’s own models, allowing you to choose the best fit for your text and image generation needs. Whether it’s creating high-quality text, summarizing information, or generating detailed visuals, Bedrock gives you the flexibility to harness generative AI for a wide range of business applications.

AWS AI/ML Service Categories

1. Ready-to-Use AI Services

These services are designed to be easy to implement with minimal setup, allowing businesses to quickly incorporate AI into their operations. The table below breaks down key AI services, their purposes, and how long they typically take to implement.

Service	Purpose & What it does	Popular use cases	Implementation Time (minimum)	Complexity
Amazon Polly	Converts text into lifelike speech.	Voice-enabled apps, content narration, accessibility tools.	Hours	Low
Amazon Rekognition	Analyzes images and videos for objects, people, and activities.	Security facial recognition, content moderation, media analysis.	Hours	Low
Amazon Transcribe	Converts speech to text from audio and video files.	Customer call transcription, video subtitles, automated note-taking.	Hours	Low
Amazon Comprehend	Natural language processing to analyze text and extract insights.	Sentiment analysis, document categorization, entity extraction.	Hours	Low
Amazon Kendra	Provides intelligent search powered by machine learning.	Enterprise search, customer support portals, research databases.	Days	Low-Medium
Amazon Textract	Extracts text and data from scanned documents, tables, and forms.	Automated document processing, invoice extraction, identity verification.	Days	Low-Medium
Amazon Fraud Detector	Detects fraudulent online activities using machine learning.	Fraud detection in payments, account takeovers, identity theft prevention.	Days	Low-Medium
Amazon Translate	Real-time translation of text between languages.	Website localization, multilingual customer service.	Hours	Low
Amazon Personalize	Provides personalized recommendations based on user behavior.	E-commerce product recommendations, content streaming suggestions.	Days	Medium
Amazon Lex	Builds conversational interfaces using voice and text (chatbots).	Customer support chatbots, virtual assistants, voice apps for contact centers.	Days	Low-Medium

2. ML Development Platforms

For organizations requiring custom ML solutions:

Platform	Purpose & What it does	Best For	Implementation Time (minimum)	Complexity
Amazon SageMaker	Provides an end-to-end platform for developing, training, and deploying ML models, with full control and advanced MLOps tools.	ML teams requiring complete control	Weeks	High
Amazon Bedrock	Enables deployment of foundation models, including custom model fine-tuning and integration with APIs and knowledge bases.	Teams building GenAI applications	Days-Weeks	Medium
Amazon Canvas	Offers a no-code platform for creating ML models through a visual interface, integrating business data for fast predictions.	Business analysts	Days	Low

Amazon SageMaker SageMaker is a comprehensive platform that enables ML teams to develop, train, and deploy models with full control over every aspect of the machine learning lifecycle. Unlike Amazon Bedrock, which is focused on deploying and fine-tuning pre-built foundation models, SageMaker allows for the creation of highly customized ML solutions. With built-in algorithms, auto ML capabilities, distributed training, and advanced MLOps tools, it’s ideal for teams looking to build bespoke models from scratch or with complex training data. SageMaker also offers granular model monitoring, giving ML teams detailed insights into model performance post-deployment. If an organization needs full control over the entire ML pipeline and prefers a deep, hands-on approach to model development, SageMaker is the platform of choice. In contrast, Bedrock simplifies the deployment of foundation models, making it more suited for teams that prioritize quick results with less customization.

Amazon Canvas For teams without a strong technical background in machine learning, Amazon Canvas offers a no-code solution that allows business analysts to create and deploy models through a simple, visual interface. While SageMaker and Bedrock are designed for technical teams, Canvas focuses on democratizing ML by making it accessible to non-developers. Its automated model creation and integration with business data sources make it an excellent tool for quickly generating insights without writing code. In comparison to Bedrock, Canvas lacks the advanced features for model fine-tuning and complex training but offers a faster way to start generating predictions for business use cases. For organizations looking to empower business users with ML capabilities without involving the technical overhead of SageMaker or the foundational focus of Bedrock, Canvas is the ideal choice.

3. Specialized AI Applications

AWS offers purpose-built solutions for specific use cases:

Amazon Augmented AI (Amazon A2I) for human review workflows
Amazon Comprehend Medical for healthcare insights
Amazon DevOps Guru for application optimization
Various Lookout services for anomaly detection

Integration with the AWS Ecosystem

AWS AI/ML services are not standalone solutions but are seamlessly integrated into the broader AWS ecosystem, allowing businesses to manage, scale, and secure their AI workloads efficiently. Whether you’re storing data, ensuring network connectivity, or monitoring system performance, AWS has the capabilities to support every aspect of your IT needs.

Storage Solutions

Amazon S3 A scalable object storage ideal for storing large datasets such as training data and model outputs.
Amazon EFS (Elastic File System A managed file storage that allows shared access to data across multiple AI instances.
Amazon FSx Provides specialized file systems for workloads requiring high-performance storage, including FSx for Lustre for data-heavy AI tasks.
Amazon Glacier Ideal for long-term, low-cost archival of data used for AI model history or compliance.

Database Solutions

Amazon Aurora A highly available relational database, perfect for storing structured data required by AI models.
Amazon Neptune A graph database suited for complex relationships, such as powering recommendation engines or social network analysis.
Amazon DynamoDB A fully managed NoSQL database for high-performance, low-latency data retrieval.
Amazon OpenSearch Service Leverage OpenSearch as a backend for real-time data search and analytics, critical for retrieval-augmented generation (RAG) systems.

Networking

Amazon VPC (Virtual Private Cloud) Create secure, isolated cloud environments for your AI applications. Connect on-premises resources using
AWS Direct Connect, enabling hybrid cloud setups where your AI/ML systems can securely communicate with existing infrastructure.

Monitoring & Logging

Amazon CloudWatch Monitor your AI applications and infrastructure in real-time. Track system performance, set alarms, and gain insights to improve application health and reduce downtime.
AWS X-Ray Trace requests as they travel through your AI services, making it easier to diagnose bottlenecks or errors.

Security & Compliance

AWS IAM (Identity and Access Management) Implement fine-grained security controls to manage access to your AI/ML services. With AWS Key Management Service (KMS) and built-in encryption, your data is secured both in transit and at rest. Compliance with GDPR, HIPAA, and other standards
AWS Shield and WAF (Web Application Firewall) Protect your AI-driven web applications from DDoS attacks, ensuring availability and reliability.

On-Premise Integration

AWS supports hybrid cloud solutions, making it easy to integrate with your existing IT infrastructure. Services like AWS Outposts allow you to run AWS AI services on-premise, while AWS Snowball and Snowcone enable secure data migration to the cloud.

In essence, AWS has the capability to host virtually every IT service your business needs, whether in the cloud or integrated with your existing infrastructure. By leveraging its vast ecosystem, you can ensure that your AI solutions are scalable, secure, and highly efficient.

Cost Considerations

AWS AI/ML services follow a pay-as-you-go model:

Ready-to-use services: Generally cents to dollars per use
ML platforms: Hundreds to thousands monthly
Custom infrastructure: Higher costs based on scale

Real-World Case Studies

Bayer AG tecRacer helped Bayer AG to implement an open platform for AI/ML development.
Hapag-Lloyd Hapag-Lloyd leveraged Amazon Q Business to automate responses to internal procedure queries, significantly reducing support time. With response times as fast as 1–3 seconds, Hapag-Lloyd aims to redirect support staff to higher-value tasks, enhancing efficiency and employee satisfaction.
LG AI Research: LG AI Research adopted Amazon SageMaker to accelerate the development of AI models for various applications. By utilizing SageMaker’s powerful tools, LG significantly reduced the time required for training models, enabling faster innovation and the deployment of cutting-edge AI solutions across multiple industries.
Slack: Slack integrated Amazon SageMaker JumpStart to power its native generative AI features, ensuring secure and scalable AI-driven solutions. This enabled Slack to offer enhanced user experiences by quickly delivering AI-powered insights, automating tasks, and providing personalized assistance, all while maintaining high levels of security and compliance.

Getting Started Guide

For Beginners:

Start with Amazon PartyRock for quick experiments
Utilize ready-to-use services like Polly and Rekognition
Explore Amazon Q Business for immediate business insights

For Advanced Users:

Leverage Amazon Bedrock for custom AI applications
Use SageMaker for end-to-end ML development
Implement specialized services for specific use cases

Conclusion

Remember to start small, experiment frequently, and scale based on your success. AWS's flexible pricing model allows you to grow your AI capabilities alongside your business needs.

At tecRacer, we specialize in helping businesses unlock the full potential of AWS AI/ML services. With years of experience across various industries, we have successfully implemented real-world AI/ML solutions that accelerate innovation and deliver tangible business results.

Let us help you transform your business with AWS AI/ML services today.

Performance Boost: 10 Expert Tips for Optimizing Your Amazon OpenSearch Service Cluster

Alexey Vidanov — Wed, 02 Oct 2024 08:43:24 +0000

By implementing these recommendations, you can maximize the potential of your Amazon OpenSearch Service domain, delivering an improved search experience while optimizing costs and maintaining security. Let's explore these expert tips to supercharge your OpenSearch cluster.

This guide updates our 10 expert tips for 2024, grouped into key areas: Hardware, Indexing, Monitoring, Sharding, and Query Optimization. We’ll also discuss why keeping your OpenSearch version up to date is crucial for unlocking performance improvements.

Let's get started.

Hardware

Leverage Graviton3 and OR1 Instances for Better Performance The new Graviton3-based instances—C7g (compute-optimized), M7g (general-purpose), R7g (memory-optimized), and R7gd (with local SSD storage)—offer significant performance boosts over their predecessors. They deliver up to 30% better compute performance and improved energy efficiency, making them ideal for a variety of OpenSearch workloads.
- C7g is ideal for compute-heavy search operations and analytics workloads.
- M7g suits a balanced mix of compute, memory, and storage needs, perfect for general-purpose OpenSearch domains.
- R7g and R7gd are designed for memory-intensive use cases, with R7gd also offering local SSD storage for high-speed access to frequently used data.
- Use OR1 Instances for Cost-Effective Long-Running Workloads The OR1 family is designed to handle long-running, steady-state workloads like log ingestion and monitoring at a lower cost. While not as powerful as the Graviton3-based instances, OR1 instances are optimized for cost savings and long-term log storage, balancing performance with reduced pricing for operational workloads.
Start big Remember, it's easier to measure the excess capacity in an overpowered cluster than the deficit in an underpowered one, so it's recommended to start with a larger cluster than you think you need, then test and scale down to an efficient cluster that has the extra resources to ensure stable operations during periods of increased activity.

Indexing

Use bulk ingest requests and employ multi-threading Bulk requests are more efficient than individual index requests. For example, a single thread can index 1000 small documents per second, but with bulk requests, it can index 100,000 to 250,000 documents per second. The bulk API is a powerful tool for indexing multiple documents in a single request, reducing the overhead of individual indexing requests. The optimal bulk size varies depending on the use case, but a good starting point is between 5-15MB.

To enhance indexing throughput, employ multi-threading. This can be achieved using OpenSearch SDKs and libraries like opensearch-py. By creating 10-20 threads per node, you can significantly boost your indexing performance.

Optimize Minimize frequent updates: To maximize efficiency in OpenSearch, minimize frequent updates to the same document. This prevents the accumulation of deleted documents and large segment sizes. Instead, collect necessary updates in your application and selectively transmit them to OpenSearch, reducing overhead and improving performance. As an example, when storing stock information in the index, it's recommended to represent it using levels (e.g., available, low, not available) instead of numerical values. This approach ensures efficient storage and retrieval of stock data in OpenSearch.

Do not index everything: Disabling indexing for specific fields by setting "index": false in the field mapping can help optimize storage, improve indexing performance.

Tune your _source field :

The _source field in OpenSearch is a special field that holds the original JSON object that was indexed. This field is automatically stored for each indexed document and is returned by default in search results.
The primary advantage of the _source field is that it allows you to access the original document directly from the search results. This can be particularly useful for debugging purposes or for performing partial updates to documents.
However, storing the _source field does increase storage requirements. Each indexed document essentially gets stored twice: once in the inverted index for searching and once in the _source field.
If your use case doesn't require accessing the original document in search results, you can disable storing the _source field to save storage space. This can be done by setting "enabled": false in the _source field mapping.

Monitoring

Use CloudWatch Monitoring tools like Amazon CloudWatch can be used to track indexing performance and identify bottlenecks. Enabling Slow Logs can save a lot of time. Set up the recommended CloudWatch alarms for Amazon OpenSearch Service
Profile queries Profiling your OpenSearch queries can provide valuable insight into how your queries are being executed and where potential performance bottlenecks may be occurring. The Profile API in OpenSearch is a powerful tool for this purpose.
- To use the Profile API, simply append ?profile=true to your search queries. This will return a detailed breakdown of your query's execution, including information about how long each operation took and how the query was rewritten internally.
- The output of the Profile API is divided into sections for each shard that participated in the response. Within each shard section, you'll find details about the query and aggregation trees.
- The query tree shows how the query was executed across the inverted index, including the time taken by each term. The aggregation tree, on the other hand, shows how the aggregations were computed, including the time taken by each bucket.
- By analyzing this information, you can identify which parts of your query are taking the most time and adjust them accordingly. This could involve changing the structure of your query, adjusting your index mappings, or modifying your OpenSearch cluster configuration.
- Remember, profiling adds overhead to your queries, so it's best to use it sparingly and only in a testing or debugging environment.

Sharding

Find an optimal shard number and size: The ideal shard size in OpenSearch is typically between 10GB and 50GB for workloads where search latency is a key performance objective, and 30-50GB for write-heavy workloads such as log analytics. Large shards can make it difficult for OpenSearch to recover from failure, but having too many small shards can cause performance issues and out of memory errors.

The number of primary shards for an index should be determined based on the amount of data you have and your expected data growth. A general guideline is to try to keep shard size between 10–30 GiB for workloads where search latency is a key performance objective, and 30–50 GiB for write-heavy workloads such as log analytics.

Optimize shard locating: Overallocating shards can lead to wasted resources. On a given node, have no more than 25 shards per GiB of Java heap. For example, an m5.large.search instance has a 4-GiB heap, so each node should have no more than 100 shards. At that shard count, each shard is roughly 5 GiB in size, which is well below the recommended size range.

Search and Query Performance

Use filters: Filters are faster than queries because they don’t calculate relevance (_score). They simply include or exclude documents.
Use search templates: One effective way to boost your Amazon OpenSearch Service is by utilizing search templates. Search templates allow you to predefine and reuse complex search queries, reducing the processing time and improving search performance.

Bonus Tip: Regularly Update Your Cluster for Optimal Performance

Keeping your Amazon OpenSearch Service cluster up to date is one of the easiest ways to ensure peak performance. Each new version of OpenSearch introduces significant performance improvements, optimizations, and bug fixes. For example, OpenSearch 2.14 brings major enhancements to query speed, indexing efficiency, and resource management, which can significantly reduce costs while improving overall cluster responsiveness.

To ensure you're taking advantage of the latest improvements:

Plan Regular Updates: Schedule periodic updates for your OpenSearch clusters. AWS makes it easy to upgrade with minimal downtime using blue/green deployments.
Test in a Staging Environment: Before applying updates in production, always test new versions in a staging environment to ensure compatibility with your existing setup.
Leverage New Features: Take advantage of the latest features and optimizations in the newer versions, such as better memory management, faster queries, and enhanced index recovery processes.

Regularly updating your OpenSearch cluster will keep your environment running smoothly and allow you to leverage the latest innovations for improved performance.

Additional Reading

Keep in mind, these tips and improvements are just the starting point. The ultimate effectiveness of their application depends on your specific scenario and use case. While we've focused on the areas of Hardware, Indexing, Sharding, Monitoring, and Optimization, there are many more facets to consider. For instance, security, which is a critical component we haven't delved into here. Remember, your OpenSearch Service should be as unique as your needs.

If you require assistance in optimizing your OpenSearch Service deployment, tecRacer, an Amazon OpenSearch Service Delivery Partner, is here to provide expert guidance. Our team of professionals specializes in designing, deploying and securely managing OpenSearch Service infrastructures tailored to individual needs. Whether you need support in selecting the right instance types, fine-tuning indexing strategies, monitoring performance, or optimizing search and query operations, tecRacer can provide the expertise you need.

Photo of Alessandro Bianchi on Unsplash

Why and How to Migrate from Solr to Amazon OpenSearch Service in 2024

Alexey Vidanov — Mon, 30 Sep 2024 14:58:33 +0000

This guide is for organizations currently using Solr and considering a migration to Amazon OpenSearch Service. We will explore the benefits of migrating, the process, and the key challenges, offering insights from real-world migrations to help you make an informed decision.

Introduction: The Evolution of Enterprise Search

In today’s fast-paced digital landscape, having a scalable and easily manageable search infrastructure is essential for smooth user experiences and efficient operations. Apache Solr and Amazon OpenSearch Service are two robust platforms offering advanced search capabilities.

Though Apache Solr has been a trusted solution for years, there is growing interest in cloud-native platforms like Amazon OpenSearch Service.

This shift reflects a broader demand for managed services that simplify operations, making OpenSearch an attractive choice for organizations looking to modernize their search infrastructure.

Why Migrate?

Migrating from Solr to OpenSearch offers several compelling benefits, especially for AWS-centric organizations. The advantages of moving to Amazon OpenSearch Service include:

Simplified Management: OpenSearch is a fully managed service, reducing the burden of maintaining your search infrastructure.
Cloud-Native Scalability: OpenSearch’s integration with other AWS services allows for smooth scaling and performance optimization without additional effort.
Advanced Features: OpenSearch provides out-of-the-box solutions such as ML integrations, anomaly detection, real-time analytics, and powerful dashboards for intuitive data visualization—features that extend beyond the capabilities of Solr.
Security and Compliance: Amazon OpenSearch Service integrates with AWS Identity and Access Management (IAM) and offers encryption at rest and in transit, among other security features.

Migration Process: From Solr to Amazon OpenSearch

1. Assess Your Priorities, Train Your Team, and Choose a Migration Strategy

Before diving into technical details, it’s crucial to define your migration priorities and ensure your team is prepared for the transition to OpenSearch. Whether you're opting for a lift-and-shift migration or planning a modernization strategy, it’s essential that your team understands how to use and manage OpenSearch effectively.

Start by organizing training sessions or workshops to familiarize your team with OpenSearch’s features, query language, and management tools. This will help ensure a smoother transition and enable your team to take full advantage of OpenSearch’s capabilities from the outset. tecRacer offers tailored workshops that can help your team get up to speed quickly.

Once your team is prepared, you can decide between:

Minimal Downtime Migration: Opt for an active-active setup where both Solr and OpenSearch run in parallel, ensuring a smooth cutover with zero downtime.
Downtime-Tolerant Migration: If your application can tolerate some downtime, you can simplify and expedite the migration process.

Additionally, for cases where Solr isn’t the system of record, consider rebuilding indices directly in OpenSearch. This gives you the opportunity to redesign schemas, mappings, and search functionalities, allowing for a cleaner, modern implementation.

AWS recommends modernization for long-term scalability and flexibility. From our experience, modernization can be achieved rapidly, with most of the complexity typically residing in the application layer rather than the search engine.

2. Cluster Sizing and Cost Estimation

Once you’ve committed to modernization and your team is trained, the next step is to right-size your OpenSearch cluster. Start by assessing your current Solr setup in terms of performance and cost, then project these needs into OpenSearch. Consider how scaling requirements will affect performance and costs in a cloud-native environment.

After this assessment, plan a Proof of Concept (PoC) after completing the schema mapping and key configurations (discussed in step 5). This PoC will help validate your cluster sizing and performance estimates, ensuring you’re working with the right setup before the actual migration.

3. Core Functionality and Future Needs

Review the core functionalities of your existing search infrastructure. Identify which features are critical to your business and explore ways OpenSearch’s advanced capabilities can improve your search infrastructure. For example, OpenSearch offers vector search, semantic search, hybrid search, and more, which could open new opportunities for your organization.

Also, plan for future scaling and feature needs. tecRacer can offer workshops and training to help your team fully leverage OpenSearch’s potential.

At this point, focus on planning and be prepared to conduct the following tests after the initial setup:

Relevance and Accuracy: Ensuring that the migrated queries produce high-quality search results.
Search Latency: Measuring response times under typical and peak usage conditions.
User Experience (UX): Validating that the search results align with user expectations in terms of speed and relevance.

4. Proof of Concept (PoC), Sizing Decisions, and Final Testing

With your migration strategy in place, you can now perform a Proof of Concept (PoC). During the PoC, test different cluster configurations and query behaviors to ensure they align with your performance needs and cost projections. This phase allows you to fine-tune your cluster size and performance settings based on real-world workloads.

Key considerations for the PoC:

Performance Testing: Simulate typical and peak traffic scenarios to ensure your OpenSearch cluster scales efficiently and maintains the required performance levels.
Data Quality and UX Testing: Evaluate the quality of search results, ensuring accuracy and relevance.
Cost Efficiency: Ensure your cluster sizing and performance optimizations meet both operational and cost requirements, factoring in future growth.

Based on the PoC results, finalize your decision on whether the cluster is sized appropriately and whether OpenSearch can handle your workload effectively. If all criteria are met, you can confidently move forward with the migration.

5. Execution: Schema Mapping and Best Practices

Once you have the correct sizing from the PoC, the next step is to begin the actual migration process, starting with schema mapping. OpenSearch uses a more flexible, dynamic mapping system than Solr, allowing you to optimize field types and indexing behavior.

During this phase, focus on:

Schema-to-Mapping Translation: Convert your Solr schema into OpenSearch mappings, leveraging OpenSearch’s flexible field definitions.
Analyzers and Normalizers: Choose and configure the appropriate analyzers and normalizers for your data to ensure proper indexing and search functionality.
Index and Shard Management: Plan the number of indices and shards carefully to optimize performance and resource allocation.

Thoroughly test your mapping and indexing configurations to ensure they align with your business needs before migrating production data.

6. Security and Compliance Best Practices

As you set up the OpenSearch environment, make security a priority. OpenSearch integrates seamlessly with AWS Identity and Access Management (IAM), allowing granular permission control. Additionally, you can secure your cluster within a VPC and monitor it with Amazon CloudWatch.

Key steps include:

IAM Roles and Permissions: Implement the appropriate permissions for your OpenSearch cluster.
Encryption: Enable encryption for data both at rest and in transit.
Monitoring: Set up CloudWatch to monitor security and performance metrics.

As security configurations can vary between Solr and OpenSearch, thoroughly test your security settings to ensure compliance before migrating live data.

7. Final Migration and Continuous Operations Setup

With the PoC validated, schema mappings tested, and security in place, it’s time for the actual migration. Depending on your chosen strategy (active-active or downtime-tolerant), you’ll either phase in OpenSearch or switch over entirely.

During the final migration, you need to implement:

Workload Runbooks: Define procedures for handling day-to-day operations, covering search queries, index management, and performance monitoring.
Alarms and Monitoring: Set up alarms in CloudWatch to alert your team of any anomalies in performance or security.
Performance Tuning: Continuously monitor and fine-tune your OpenSearch cluster to ensure it operates efficiently, adjusting based on real-world workloads.
Backup and Disaster Recovery Plans: Establish regular snapshot schedules, and ensure that your disaster recovery plan is robust and regularly tested.

By ensuring that monitoring and automation are in place, you can maintain a smooth-running system post-migration. Regular performance and cost evaluations, as well as ongoing query and data quality improvements, will help keep your OpenSearch cluster optimized.

Through this structured process—starting with clear migration priorities, validating configurations via a PoC, training your team, and finally executing a carefully monitored migration—you can ensure a successful transition from Solr to OpenSearch with minimal disruption and maximum long-term benefits.

Terminology Comparison

Solr Term	OpenSearch Term	Comment
Collection	Index	Both represent a set of documents to be searched. Solr’s Collection is equivalent to an Index in OpenSearch.
Schema	Mapping	Both define the structure of documents. Solr uses a Schema, while OpenSearch uses Mapping to manage field types and document structure.
FieldType	Analyzer (and more)	Solr’s FieldType defines both data types and how fields are analyzed. OpenSearch’s Analyzer only handles text analysis.
DynamicField	Dynamic Mapping	Solr’s DynamicField and OpenSearch’s Dynamic Mapping both apply rules automatically to new fields that are not explicitly defined.
Query (Standard Syntax)	Query DSL (JSON-based)	Solr uses a standard syntax for queries, while OpenSearch uses a Query DSL, which is more flexible and expressed in JSON.

Schema and Query Translation: Key Differences

While Solr and OpenSearch both leverage Lucene, the two platforms handle schemas and queries differently, which presents challenges during migration.

Schema Translation

Solr uses a global schema for each collection, which simplifies management but can lead to limitations when different documents require varied settings. OpenSearch, on the other hand, employs index-based mappings, offering flexibility in handling diverse data types across multiple indices.

Query Translation

OpenSearch and Solr differ significantly in their querying approaches, but OpenSearch provides a wide range of powerful options, which you can explore in its API reference. For instance, the simple_query_string query has a syntax similar to Solr’s, making it an easy starting point for users less familiar with OpenSearch’s DSL. In many cases, it can perform well and simplify the migration process.

That said, we’ve observed with one of our customers that using simple_query_string in OpenSearch is not exactly the same as in Solr. While it may seem similar at first glance, there are key differences in behavior, such as handling term-specific boosts.

Here’s an example of where these differences come into play:

In Solr, you can easily boost individual terms in a multi-term query using the caret (^) symbol. But in OpenSearch’s simple_query_string, this functionality is not fully supported, and queries relying on heavy term-specific boosting may need more advanced solutions like the bool query.

Solr Query with Term-Specific Boosting:

q=title:(OpenSearch^2 AWS) OR description:(search engine)

In this Solr query, only the term OpenSearch is boosted within the title field, while AWS is left unboosted. This term-specific boosting within a multi-term query is not natively supported in OpenSearch.

Attempted OpenSearch Equivalent:

{
  "query": {
    "simple_query_string": {
      "fields": ["title", "description"],
      "query": "title:(OpenSearch^2 AWS) OR description:(search engine)"
    }
  }
}

While this query will run, the boost on OpenSearch (^2) will be ignored because simple_query_string in OpenSearch does not support boosting individual terms within a multi-word query.

To accurately replicate this behavior, you would need to use more advanced query types like bool or match, where you can apply specific boosts to each term individually.

OpenSearch `bool` Query with Term-Specific Boosting:

{
  "query": {
    "bool": {
      "should": [
        { 
          "match": { 
            "title": { 
              "query": "OpenSearch", 
              "boost": 2 
            } 
          } 
        },
        { 
          "match": { 
            "title": { 
              "query": "AWS" 
            } 
          } 
        },
        { 
          "match": { 
            "description": { 
              "query": "search engine" 
            } 
          } 
        }
      ]
    }
  }
}

Here, each term (OpenSearch and AWS) is treated separately, with explicit boosting applied only to OpenSearch. This approach is required because OpenSearch does not support inline term-specific boosts within a single multi-word query in the way Solr does.

Thus, while simple_query_string can be a helpful starting point for migrating basic queries from Solr, more complex queries—especially those involving term-specific boosting—require transitioning to OpenSearch's full Query DSL for accurate query behavior and control..

More Query Translation Examples

Basic Field Query

Solr: q=title:OpenSearch
OpenSearch:

   {
     "query": {
       "match": {
         "title": "OpenSearch"
       }
     }
   }

Range Query

Solr: q=price:[10 TO 100] AND popularity:[5 TO *]
OpenSearch:

   {
     "query": {
       "bool": {
         "must": [
           {
             "range": {
               "price": {
                 "gte": 10,
                 "lte": 100
               }
             }
           },
           {
             "range": {
               "popularity": {
                 "gte": 5
               }
             }
           }
         ]
       }
     }
   }

Fuzzy Search

Solr: q=title:OpenSerch~2
OpenSearch:

   {
     "query": {
       "fuzzy": {
         "title": {
           "value": "OpenSerch",
           "fuzziness": 2
         }
       }
     }
   }

Phrase Query with Slop

Solr: q="open source search"~3
OpenSearch:

   {
     "query": {
       "match_phrase": {
         "content": {
           "query": "open source search",
           "slop": 3
         }
       }
     }
   }

Use Case: MemoMeister's Solr Migration to Amazon OpenSearch

Freiraum GmbH, based in Stuttgart, manages the digital documentation platform MemoMeister. Recognizing that efficient enterprise search was critical for its customers, the company decided to migrate its search infrastructure from Solr on EC2 to Amazon OpenSearch Service.

Key Objectives of the Migration:

Reduce operational overhead through a fully managed service.
Improve search performance and scalability.
Leverage OpenSearch’s integration with other AWS services for enhanced resilience and ease of management.

The Proof of Concept (PoC) focused on testing OpenSearch in a Multi-AZ environment, ensuring the solution met high availability and performance benchmarks. The results demonstrated that OpenSearch not only met but exceeded expectations in terms of ease of use, resilience, and drastically reduced operational overhead.

Jan Schurkus, CEO bei Freiraum GmbH

During the Proof of Concept Project for the migration of Solr to Amazon OpenSearch Service, we were not only be able to assess the viability of this solution but also initiated the migration process itself. tecRacer helped us to untangle the knots in our minds and unleash the full potential of the service. The impulses they provided were crucial in developing a solution aligned with best practices. We work with a tecRacer on various cloud improvement projects simultaneously, which serves as building blocks for achieving greater success with our customers.

Migration Highlights:

Savings on Total Cost of Ownership (TCO): Operational overhead was reduced by 50%x, thanks to OpenSearch’s ease of management.
Cluster Stability: The stability provided by OpenSearch’s managed service ensured a seamless migration experience.
Optimized Search Capabilities: Despite initial concerns, OpenSearch offered all the necessary search functionalities, along with improved resource efficiency.

Overcoming Challenges in Migration

Migrating from Solr to OpenSearch can involve various challenges, including:

Schema Differences: Addressing the differences between Solr’s global schema and OpenSearch’s index-specific mappings.
Query Language Adjustments: Rewriting queries to use OpenSearch’s Query DSL.
Terminology Translation: Understanding and adapting the differences in terminology between Solr and OpenSearch.
Data Integrity: Ensuring that field mappings and data ingestion processes don’t lead to inconsistencies during the migration process.

Conclusion

Migrating from Solr to Amazon OpenSearch Service offers numerous long-term benefits, including simplified management, enhanced search functionalities, and better scalability. With the right tools and planning, the migration process can be seamless and rewarding. By adopting a proof of concept approach, organizations can ensure that they are set up for success, both in terms of technical performance and operational efficiency.

Whether you're looking to reduce operational overhead, improve search capabilities, or leverage AWS's ecosystem, migrating to OpenSearch is a forward-thinking choice for enterprises.