DEV Community

Cover image for How to connect to AWS OpenSearch or Elasticsearch clusters using python
Lucy Linder
Lucy Linder

Posted on • Originally published at blog.derlin.ch

2

How to connect to AWS OpenSearch or Elasticsearch clusters using python

Connecting to an OpenSearch (ES) service running in AWS using Python is painful. Most examples I find online either don't work or are outdated, leaving me constantly fixing the same issues. To save time and frustration, here’s a collection of working code snippets, up-to-date as of December 2024.


Connect using the opensearch-py library (OpenSearch + ElasticSearch)

This is my preferred way of connecting to an ES instance managed by AWS. It works for both ElasticSearch and OpenSearch clusters, and the authentication can take advantage of AWS profiles.

Install opensearch-py and boto3 (for authentication):

pip install opensearch-py boto3
Enter fullscreen mode Exit fullscreen mode

At the time of writing, this installs opensearch-py==2.8.0 and boto3==1.35.81.

Now, you can create a client using the following:

import boto3

from opensearchpy import (
    AWSV4SignerAuth,
    OpenSearch,
    RequestsHttpConnection,
)

es_host = "search-my-aws-esdomain-5k2baneoyj4vywjseocultv2au.eu-central-1.es.amazonaws.com"
aws_access_key = "AKIAXCUEGTAF3CV7GYKA"
aws_secret_key = "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"
region = "eu-central-1"

# Note: you can also use boto3.Session(profile_name="my-profile") or other ways
session = boto3.Session(
    aws_access_key_id=aws_access_key,
    aws_secret_access_key=aws_secret_key,
    region_name=region,
)

client = OpenSearch(
    hosts=[{"host": es_host, "port": 443}],
    http_auth=AWSV4SignerAuth(session.get_credentials(), region, "es"),
    connection_class=RequestsHttpConnection,
    use_ssl=True,
)
Enter fullscreen mode Exit fullscreen mode

Note that boto3.Session supports various ways of creating a session: using a profile, environment variables, and more. I will let you check it out!

Once you have it, check the connection using:

client.ping() # should return True
client.info() # use this to get a proper error message if ping fails
Enter fullscreen mode Exit fullscreen mode

To check indices:

# List all indices
client.cat.indices()
client.indices.get("*")

# Check the existence of an indice
client.indices.exists("my-index")
Enter fullscreen mode Exit fullscreen mode

Connect using the elasticsearch library (ElasticSearch only)

🔥 This only works for ElasticSearch clusters! Connecting to an OpenSearch cluster raises

UnsupportedProductError: The client noticed that the server is not Elasticsearch and we do not support this unknown product

elasticsearch >= 8

Most snippets are still referencing RequestsHttpConnection, a class that was removed in elasticsearch 8.X. If you were googling for the error cannot import name 'RequestsHttpConnection' from 'elasticsearch’, you are at the right place!

Install elasticsearch (this should install elastic-transport as well), and requests_aws4auth . The latter, based on requests, is required to handle authentication to AWS:

pip install elasticsearch requests-aws4auth
Enter fullscreen mode Exit fullscreen mode

At the time of writing, this installs elastic-transport==8.15.1, elasticsearch==8.17.0 and requests-aws4auth==1.3.1.

Now, you can create a client using the following:

from elastic_transport import RequestsHttpNode
from elasticsearch import Elasticsearch
from requests_aws4auth import AWS4Auth

es_endpoint = "search-my-aws-esdomain-5k2baneoyj4vywjseocultv2au.eu-central-1.es.amazonaws.com"
aws_access_key = "AKIAXCUEGTAF3CV7GYKA"
aws_secret_key = "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"
region = "eu-central-1"

es = Elasticsearch(
    f"https://{es_host}",
    http_auth=AWS4Auth(
        aws_access_key, 
        aws_secret_key, 
        region,
        "es",
    ),
    verify_certs=True,
    node_class=RequestsHttpNode,
)
Enter fullscreen mode Exit fullscreen mode

Once you have it, check the connection using:

es.ping() # should return True
es.info() # use this to get a proper error message if ping fails
Enter fullscreen mode Exit fullscreen mode

elasticsearch < 8

If you are still on an old version of elasticsearch:

pip install "elasticsearch<8" requests-aws4auth
Enter fullscreen mode Exit fullscreen mode

Currently elasticsearch==7.17.12, requests-aws4auth==1.3.1.

Now, you can create a client using the following:

from elasticsearch import Elasticsearch, RequestsHttpConnection
from requests_aws4auth import AWS4Auth

es_endpoint = "search-my-aws-esdomain-5k2baneoyj4vywjseocultv2au.eu-central-1.es.amazonaws.com"
aws_access_key = "AKIAXCUEGTAF3CV7GYKA"
aws_secret_key = "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"
region = "eu-central-1"

es = Elasticsearch(
    host=es_endpoint,
    http_auth=AWS4Auth(
        aws_access_key, aws_secret_key, region, "es"
    ),
    use_ssl=True,
    port=443,
    verify_certs=True,
    connection_class=RequestsHttpConnection,
)
Enter fullscreen mode Exit fullscreen mode

Check the connection:

es.ping() # should return True
es.info() # use this to get a proper error message if ping fails
Enter fullscreen mode Exit fullscreen mode

Speedy emails, satisfied customers

Postmark Image

Are delayed transactional emails costing you user satisfaction? Postmark delivers your emails almost instantly, keeping your customers happy and connected.

Sign up

Top comments (0)

Heroku

Build apps, not infrastructure.

Dealing with servers, hardware, and infrastructure can take up your valuable time. Discover the benefits of Heroku, the PaaS of choice for developers since 2007.

Visit Site

👋 Kindness is contagious

Explore a sea of insights with this enlightening post, highly esteemed within the nurturing DEV Community. Coders of all stripes are invited to participate and contribute to our shared knowledge.

Expressing gratitude with a simple "thank you" can make a big impact. Leave your thanks in the comments!

On DEV, exchanging ideas smooths our way and strengthens our community bonds. Found this useful? A quick note of thanks to the author can mean a lot.

Okay