DEV Community: A_Lucas

Embedding and Reranking with Alibaba Cloud AI Service

A_Lucas — Fri, 28 Feb 2025 03:26:46 +0000

Written by: Elastic Tomás Murúa

In this article, we'll cover how to integrate Alibaba Cloud AI features with Elasticsearch to improve relevance in semantic searches.

Alibaba Cloud AI Search is a solution that integrates advanced AI features with Elasticsearch tools, by leveraging the Qwen LLM/DeepSeek-R1 family to contribute with advanced models for inference and classification. In this article, we'll use descriptions of novels and plays written by the same author to test the Alibaba reranking and sparse embedding endpoints.

Steps

1）Configure Alibaba Cloud AI

2）Create Elasticsearch mappings

3）Index data into Elasticsearch

4）Query data

5）Bonus: Answering questions with completion

Configure Alibaba Cloud AI

Alibaba Cloud AI reranking and embeddings

Open inference Alibaba Cloud offers different services. In this example, we'll use the descriptions of popular books and plays by Agatha Christie to test Alibaba Cloud embeddings and reranking endpoints in semantic search.

The Alibaba Cloud AI reranking endpoint is a semantic reranking functionality. This type of reranking uses a machine learning model to reorder search results based on their semantic similarity to a query. This allows you to use out-of-the-box semantic search capabilities on existing full-text search indices.

The sparse embedding endpoint is a type of embedding where most values are zero, making relevant information more prominent.

Get Alibaba Cloud API Key

We need a valid API Key to integrate Alibaba with Elasticsearch. To get it, follow these steps:

1）Access the Alibaba Cloud portal from the Service Plaza section.

2）Go to the left menu API Keys as shown below.

3）Generate a new API Key.

Configure Alibaba Endpoints

We´ll first configure the sparse embedding endpoint to transform the text descriptions into semantic vectors:

Embeddings endpoint:

PUT _inference/sparse_embedding/alibabacloud_ai_search_sparse
{
    "service": "alibabacloud-ai-search",
    "service_settings": {
        "api_key": "<api_key>",
        "service_id": "ops-text-sparse-embedding-001",
        "host": "default-j01.platform-cn-shanghai.opensearch.aliyuncs.com",
        "workspace": "default"
    }
}

We´ll then configure the rerank endpoint to reorganize results.

Rerank Endpoint:

PUT _inference/rerank/alibabacloud_ai_search_rerank
{
    "service": "alibabacloud-ai-search",
    "service_settings": {
        "api_key": "<api_key>",
        "service_id": "ops-bge-reranker-larger",
        "host": "default-j01.platform-cn-shanghai.opensearch.aliyuncs.com",
        "workspace": "default"
    }
}

Now that the endpoints are configured, we can prepare the Elasticsearch index.

Create Elasticsearch mappings

Let's configure the mappings. For this, we need to organize both the texts with the descriptions as well as the model-generated vectors.

We'll use the following properties:

semantic_description: to store the embeddings generated by the model and run semantic searches.

description: we'll use a "text" type to store the novels and plays’ descriptions and use them for full-text search.

We'll include the copy_to parameter so that both the text and the semantic field are available for hybrid search:

PUT arts
{
  "mappings": {
    "properties": {
      "semantic_description": {
        "type": "semantic_text",
        "inference_id": "alibabacloud_ai_search_sparse"
      },
      "description": {
        "type": "text",
        "copy_to": "semantic_description"
      }
    }
  }
}

With the mappings ready, we can now index the data.

Index data into Elasticsearch

Here's the dataset with the descriptions that we'll use for this example. We'll index it using the Elasticsearch Bulk API.

POST arts/_bulk
{ "index": {} }
{ "description": " Black Coffee is a play by the British crime-fiction author Agatha Christie. In the play, a scientist discovers that someone in his household has stolen the formula for an explosive." }
{ "index": {} }
{ "description": "The Mousetrap is a murder mystery play by Agatha Christie. The play opened in London's West End in 1952 and ran continuously until 16 March 2020." }
{ "index": {} }
{ "description": "The Body in the Murder is a Miss Marple mystery novel published by Agatha Christie in 1942. The case involves the murder of two teenage girls who are similar in appearance." }
{ "index": {} }
{ "description": " Agatha Christie's last published novel before she passed, Curtain: Poirot's Last Case is also her indelible detective's last appearance. Poirot and Hastings return to the very same house from The Mysterious Affairs at Styles over 30 years later." }
{ "index": {} }
{ "description": " Death on the Nile is Agatha Christie's most daring travel mystery novel. The tranquillity of a cruise along the Nile is shattered by the discovery that Linnet Ridgeway has been shot through the head." }
{ "index": {} }
{ "description": " The Murder of Roger Ackroyd was Agatha Christie’s first book to be published by William Collins in the spring of 1926. William Collins became part of HarperCollins and are still Christie’s publishers today." }

Note that the first two documents, “Black Coffee” and “The Mousetraps” are plays while the others are novels.

Query data

To see the different results we can get, we'll run different types of queries, starting with semantic query, then applying reranking, and finally using both. We'll use the same question "Which novel was written by Agatha Christie?" expecting to get the three documents that explicitly say novel, plus the one that says book. The two plays should be the last results.

Semantic search

We'll begin querying the semantic_text field to ask: "Which novel was written by Agatha Christie?" Let's see what happens:

GET /arts/_search
{
  "_source": {
    "includes": [
      "description"
    ]
  },
  "query": {
    "semantic": {
      "field": "semantic_description",
      "query": "Which novel was written by Agatha Christie?"
    }
  }
}

Response:

{
  "took": 1246,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 6,
      "relation": "eq"
    },
    "max_score": 0.1759066,
    "hits": [
      {
        "_index": "arts",
        "_id": "rdJ4-ZMB36zj9EVTnMgJ",
        "_score": 0.1759066,
        "_source": {
          "description": " Death on the Nile is Agatha Christie's most daring travel mystery novel. The tranquillity of a cruise along the Nile is shattered by the discovery that Linnet Ridgeway has been shot through the head."
        }
      },
      {
        "_index": "arts",
        "_id": "rNJ4-ZMB36zj9EVTnMgJ",
        "_score": 0.17499167,
        "_source": {
          "description": " Agatha Christie's last published novel before she passed, Curtain: Poirot's Last Case is also her indelible detective's last appearance. Poirot and Hastings return to the very same house from The Mysterious Affairs at Styles over 30 years later."
        }
      },
      {
        "_index": "arts",
        "_id": "q9J4-ZMB36zj9EVTnMgJ",
        "_score": 0.16319725,
        "_source": {
          "description": "The Body in the Murder is a Miss Marple mystery novel published by Agatha Christie in 1942. The case involves the murder of two teenage girls who are similar in appearance."
        }
      },
      {
        "_index": "arts",
        "_id": "qtJ4-ZMB36zj9EVTnMgJ",
        "_score": 0.15506727,
        "_source": {
          "description": "The Mousetrap is a murder mystery play by Agatha Christie. The play opened in London's West End in 1952 and ran continuously until 16 March 2020."
        }
      },
      {
        "_index": "arts",
        "_id": "qdJ4-ZMB36zj9EVTnMgJ",
        "_score": 0.14572844,
        "_source": {
          "description": " Black Coffee is a play by the British crime-fiction author Agatha Christie. In the play, a scientist discovers that someone in his household has stolen the formula for an explosive."
        }
      },
      {
        "_index": "arts",
        "_id": "rtJ4-ZMB36zj9EVTnMgJ",
        "_score": 0.13951442,
        "_source": {
          "description": " The Murder of Roger Ackroyd was Agatha Christie’s first book to be published by William Collins in the spring of 1926. William Collins became part of HarperCollins and are still Christie’s publishers today."
        }
      }
    ]
  }
}

In this case, the response prioritized most of the novels, but the document that says “book” appears last. We can still further refine the results with reranking.

Refining results with Reranking

In this case, we'll use a _inference/rerank request to assess the documents we got in the first query and improve their rank in the results.

POST _inference/rerank/alibabacloud_ai_search_rerank
{
  "query": "Which novel was written by Agatha Christie?",
  "input": [
    "Black Coffee is a play by the British crime-fiction author Agatha Christie. In the play, a scientist discovers that someone in his household has stolen the formula for an explosive.",
    "The Mousetrap is a murder mystery play by Agatha Christie. The play opened in London's West End in 1952 and ran continuously until 16 March 2020.",
    " The Body in the Murder is a Miss Marple mystery novel published by Agatha Christie in 1942. The case involves the murder of two teenage girls who are similar in appearance.",
    " Agatha Christie's last published novel before she passed, Curtain: Poirot's Last Case is also her indelible detective's last appearance. Poirot and Hastings return to the very same house from The Mysterious Affairs at Styles over 30 years later.",
    " Death on the Nile is Agatha Christie's most daring travel mystery novel. The tranquillity of a cruise along the Nile is shattered by the discovery that Linnet Ridgeway has been shot through the head.",
    " The Murder of Roger Ackroyd was Agatha Christie’s first book to be published by William Collins in the spring of 1926. William Collins became part of HarperCollins and are still Christie’s publishers today."
  ]
}

Response:

{
  "rerank": [
    {
      "index": 3,
      "relevance_score": 0.91086304
    },
    {
      "index": 4,
      "relevance_score": 0.8409133
    },
    {
      "index": 2,
      "relevance_score": 0.76838577
    },
    {
      "index": 5,
      "relevance_score": 0.2295352
    },
    {
      "index": 0,
      "relevance_score": 0.13846178
    },
    {
      "index": 1,
      "relevance_score": 0.06620602
    }
  ]
}

The response here shows that both plays are now at the bottom of the results.

Semantic search and reranking endpoint combined

Using a retriever, we'll combine the semantic query and reranking in just one step:

POST /arts/_search
{
  "_source": {
    "includes": ["description"]
  },
  "retriever": {
    "text_similarity_reranker": {
      "retriever": {
        "standard": {
          "query": {
            "semantic": {
              "field": "semantic_description",
              "query": "Which novel was written by Agatha Christie?"
            }
          }
        }
      },
      "field": "description",
      "rank_window_size": 10,
      "inference_id": "alibabacloud_ai_search_rerank",
      "inference_text": "Which novel was written by Agatha Christie?"
    }
  }
}

Response:

  "took": 1568,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 6,
      "relation": "eq"
    },
    "max_score": 0.91086304,
    "hits": [
      {
        "_index": "arts",
        "_id": "rNJ4-ZMB36zj9EVTnMgJ",
        "_score": 0.91086304,
        "_source": {
          "description": " Agatha Christie's last published novel before she passed, Curtain: Poirot's Last Case is also her indelible detective's last appearance. Poirot and Hastings return to the very same house from The Mysterious Affairs at Styles over 30 years later."
        }
      },
      {
        "_index": "arts",
        "_id": "rdJ4-ZMB36zj9EVTnMgJ",
        "_score": 0.8409133,
        "_source": {
          "description": " Death on the Nile is Agatha Christie's most daring travel mystery novel. The tranquillity of a cruise along the Nile is shattered by the discovery that Linnet Ridgeway has been shot through the head."
        }
      },
      {
        "_index": "arts",
        "_id": "q9J4-ZMB36zj9EVTnMgJ",
        "_score": 0.76838577,
        "_source": {
          "description": "The Body in the Murder is a Miss Marple mystery novel published by Agatha Christie in 1942. The case involves the murder of two teenage girls who are similar in appearance."
        }
      },
      {
        "_index": "arts",
        "_id": "rtJ4-ZMB36zj9EVTnMgJ",
        "_score": 0.2295352,
        "_source": {
          "description": " The Murder of Roger Ackroyd was Agatha Christie’s first book to be published by William Collins in the spring of 1926. William Collins became part of HarperCollins and are still Christie’s publishers today."
        }
      },
      {
        "_index": "arts",
        "_id": "qdJ4-ZMB36zj9EVTnMgJ",
        "_score": 0.13846178,
        "_source": {
          "description": " Black Coffee is a play by the British crime-fiction author Agatha Christie. In the play, a scientist discovers that someone in his household has stolen the formula for an explosive."
        }
      },
      {
        "_index": "arts",
        "_id": "qtJ4-ZMB36zj9EVTnMgJ",
        "_score": 0.06620602,
        "_source": {
          "description": "The Mousetrap is a murder mystery play by Agatha Christie. The play opened in London's West End in 1952 and ran continuously until 16 March 2020."
        }
      }
    ]
  }
}

The results here differ from the semantic query. We can see that the document with no exact match for "novel" but that says “book” (The Murder of Roger Ackroyd) appears higher than in the first semantic search. Both plays are still the last results, just like with reranking.

Bonus: Answering questions with completion

With embeddings and reranking we can satisfy a search query, but still, the user will see all the search results and not the actual answer.

With the examples provided, we are one step away from a RAG implementation, where we can provide the top results + the question to an LLM to get the right answer.

Fortunately, Alibaba Cloud AI Service also provides an endpoint service we can use to achieve this purpose.

Let’s create the endpoint

Completion Endpoint:

1）Create Completion Endpoints with Alibaba Cloud Qwen LLM

PUT _inference/completion/alibabacloud_ai_search_completion
{
    "service": "alibabacloud-ai-search",
    "service_settings": {
        "host" : "default-j01.platform-cn-shanghai.opensearch.aliyuncs.com",
        "api_key": "<api_key>",
        "service_id": "ops-qwen-turbo",
        "workspace" : "default"
    }
}

2）We can also create it with DeepSeek-R1

PUT _inference/completion/alibabacloud_ai_search_completion_deepseek_r1
{
    "service": "alibabacloud-ai-search",
    "service_settings": {
        "host" : "default-j01.platform-cn-shanghai.opensearch.aliyuncs.com",
        "api_key": "{{API_KEY}}",
        "service_id": "deepseek-r1",
        "workspace" : "default"
    }
}

And now, send the results and question from the previous query:

Query with Alibaba Cloud Qwen LLM

POST _inference/completion/alibabacloud_ai_search_completion
{
  "input": """
    Answer the following question using the context provided:
    QUESTION: Which novel was written by Agatha Christie?
    CONTEXT:
    DOCUMENT1
    Black Coffee is a play by the British crime-fiction author Agatha Christie. In the play, a scientist discovers that someone in his household has stolen the formula for an explosive.
    DOCUMENT2
    The Mousetrap is a murder mystery play by Agatha Christie. The play opened in London's West End in 1952 and ran continuously until 16 March 2020.
    DOCUMENT3
    The Body in the Murder is a Miss Marple mystery novel published by Agatha Christie in 1942. The case involves the murder of two teenage girls who are similar in appearance.
    DOCUMENT4
    Agatha Christie's last published novel before she passed, Curtain: Poirot's Last Case is also her indelible detective's last appearance. Poirot and Hastings return to the very same house from The Mysterious Affairs at Styles over 30 years later.
    DOCUMENT5
    Death on the Nile is Agatha Christie's most daring travel mystery novel. The tranquillity of a cruise along the Nile is shattered by the discovery that Linnet Ridgeway has been shot through the head."
    DOCUMENT6
    The Murder of Roger Ackroyd was Agatha Christie’s first book to be published by William Collins in the spring of 1926. William Collins became part of HarperCollins and are still Christie’s publishers today.
    ANSWER:
    """
}

Response:

{
  "completion": [
{
      "result": "Agatha Christie wrote several novels, including \"The Body in the Murder,\" \"Curtain: Poirot's Last Case,\" \"Death on the Nile,\" and \"The Murder of Roger Ackroyd.\""
    }
  ]
}

Query with Alibaba Cloud DeepSeek-R1

POST _inference/completion/alibabacloud_ai_search_completion_deepseek_r1?timeout=180s
{
  "input": "<|system|>
    You are an AI assistant.</s>
    <|user|>
    CONTEXT:
    Black Coffee is a play by the British crime-fiction author Agatha Christie. In the play, a scientist discovers that someone in his household has stolen the formula for an explosive;

    The Mousetrap is a murder mystery play by Agatha Christie. The play opened in London's West End in 1952 and ran continuously until 16 March 2020;

    The Body in the Murder is a Miss Marple mystery novel published by Agatha Christie in 1942. The case involves the murder of two teenage girls who are similar in appearance;
    Agatha Christie's last published novel before she passed, Curtain: Poirot's Last Case is also her indelible detective's last appearance. Poirot and Hastings return to the very same house from The Mysterious Affairs at Styles over 30 years later;

    Death on the Nile is Agatha Christie's most daring travel mystery novel. The tranquillity of a cruise along the Nile is shattered by the discovery that Linnet Ridgeway has been shot through the head;
    The Murder of Roger Ackroyd was Agatha Christie’s first book to be published by William Collins in the spring of 1926. William Collins became part of HarperCollins and are still Christie’s publishers today;

    QUESTION: 
    Which novela were written by Agatha Christie?</s>
    <|assistant|>"
}

Result:

{
  "completion": [
    {
      "result": """<think>
Okay, let's see. The user is asking which novels were written by Agatha Christie based on the given context. First, I need to go through each item in the context and determine if it's a novel. The user mentioned "novela," which I think is Spanish for "novel," so they're asking about novels, not plays or other works.
Looking at the context entries one by one:
1. **Black Coffee** is described as a play by Christie. So that's a play, not a novel. Exclude.
2. **The Mousetrap** is a murder mystery play, opened in London's West End. Definitely a play, not a novel. Exclude.
3. **The Body in the Murder** is listed as a Miss Marple mystery novel published in 1942. Wait, the title here might be a bit off. Agatha Christie wrote a novel called "The Body in the Library," which is a Miss Marple story from 1942. Maybe the user made a typo. Assuming it's "The Body in the Library," then yes, that's a novel. But the title given is "The Body in the Murder," which I don't recall. Need to check if that's a real title or a mistake. However, since the context says it's a Miss Marple novel published in 1942, I'll proceed with that, even if the title is slightly wrong. So include as a novel.
4. **Curtain: Poirot's Last Case** is mentioned as her last published novel before she passed. So that's a novel. Include.
5. **Death on the Nile** is described as a travel mystery novel. That's a novel. Include.
6. **The Murder of Roger Ackroyd** was her first book published by William Collins. That's a novel. Include.
So the novels listed here are: The Body in the Murder (assuming typo), Curtain, Death on the Nile, and The Murder of Roger Ackroyd. However, "The Body in the Murder" might actually be "The Body in the Library," which is the correct title. But since the user provided that exact title, I should list it as given, even if there's an error. Alternatively, note the possible typo.
Also, check if there are other works mentioned. The other entries are plays. So the answer should list the four novels mentioned in the context, being careful with the title accuracy.
</think>
The novels written by Agatha Christie mentioned in the context are:  
1. **The Body in the Murder** (likely a typo for *The Body in the Library*, a Miss Marple novel published in 1942).  
2. **Curtain: Poirot's Last Case** (her final published novel featuring Hercule Poirot).  
3. **Death on the Nile** (a travel mystery novel set on a Nile cruise).  
4. **The Murder of Roger Ackroyd** (her breakthrough novel published in 1926).  
*Note*:  
- *Black Coffee* and *The Mousetrap* are plays, not novels.  
- If "The Body in the Murder" is intended to refer to *The Body in the Library*, the latter is the correct title of Christie's 1942 Miss Marple novel."""
    }
  ]
}

Conclusion

Integrating Alibaba Cloud AI Search with Elasticsearch allows us to easily access completion, embedding, and reranking models to incorporate them into our search pipeline.

We can use the reranking and embedding endpoints, either separately or together, with the help of a retriever.

We can also introduce the completion endpoint to finish up a RAG end-to-end implementation.

Ready to start your journey with elasticsearch on Alibaba Cloud? Explore our tailored Cloud solutions and services to take the first step towards transforming your data into a visual masterpiece.

Please Click here, Embark on Your 30-Day Free Trial

Original text：Embeddings and reranking with Alibaba Cloud AI Service - Elasticsearch Labs

Building Real-Time Anomaly Detection Systems with Alibaba Cloud Elasticsearch ML Modules

A_Lucas — Thu, 27 Feb 2025 06:36:09 +0000

Overview of Alibaba Cloud Elasticsearch ML Modules

Key Features for Real-Time Anomaly Detection

Alibaba Cloud Elasticsearch ML Modules offer powerful tools for detecting anomalies in real time. These modules use both unsupervised and supervised machine learning techniques to analyze your data effectively. Unsupervised learning identifies unusual patterns without prior training, making it ideal for detecting anomalies in logging or financial transactions. Supervised learning, on the other hand, trains models on labeled data to classify new information and predict outcomes using regression and classification algorithms.

These features allow you to process large volumes of logging data efficiently. You can uncover hidden patterns and address potential issues before they disrupt your system. By leveraging these capabilities, you enhance your ability to maintain robust IT operations.

Observability Capabilities in Elasticsearch

Observability plays a crucial role in anomaly detection. Alibaba Cloud Elasticsearch integrates advanced observability tools to help you monitor and analyze your system. The AI Assistant uses RAG technology and large language models to automate data analysis and visualization. It transforms observability data into actionable insights, enabling you to identify root causes of errors quickly.

This assistant also monitors real-time anomalies and potential failures. It analyzes logging data to detect security threats and proposes defense strategies. These observability capabilities ensure your cloud-native system remains secure and efficient.

Why Choose Alibaba Cloud for aiops and Anomaly Detection

Alibaba Cloud stands out as a leader in aiops and anomaly detection. Its Elasticsearch ML Modules combine cloud-native technologies with advanced machine learning to deliver unparalleled performance. The platform supports seamless integration with your IT infrastructure, enabling you to process logging data in real time.

With Alibaba Cloud, you gain access to cutting-edge tools for observability and aiops. These tools empower you to automate monitoring, reduce downtime, and improve system reliability. By choosing Alibaba Cloud, you ensure your IT operations remain resilient and future-ready.

Key Components of a Real-Time Anomaly Detection System

Data Ingestion and Preprocessing

Data ingestion and preprocessing form the foundation of any real-time anomaly detection system. You need to collect data from diverse sources such as website logs, network traffic, and online databases. This step ensures that all relevant information is available for analysis. The collected data often contains inconsistencies or missing values. Cleaning and formatting the data make it usable for anomaly detection. For instance, estimation techniques can address missing data points effectively.

The ingestion process involves storing the data in Elasticsearch, where it becomes accessible for further analysis. Preprocessing also includes converting raw data into formats suitable for machine learning algorithms. By ensuring high-quality input, you improve the system's performance and accuracy in detecting anomalies.

Machine Learning Models for Anomaly Detection

Machine learning models are the core of anomaly detection and pattern recognition. These models analyze data to identify unusual patterns. Algorithms like Isolation Forest, Local Outlier Factor, and One-Class SVM are particularly effective. Isolation Forest isolates anomalies by dividing data points using decision trees. Local Outlier Factor evaluates the density of data points to detect anomalies based on their neighborhood. One-Class SVM creates boundaries around normal data points, marking those outside as anomalies.

Selecting the right algorithm depends on your system's requirements. For example, unsupervised algorithms work well when you lack labeled data. These models enable you to uncover hidden anomalies and gain real-time insights into your system's behavior.

Algorithm	Description
Isolation Forest	Divides data points using decision trees to isolate anomalies.
Local Outlier Factor	Detects anomalies by analyzing the density of data points in their neighborhood.
One-Class SVM	Identifies anomalies by creating boundaries around normal data points.

Real-Time Processing and Alerting

Real-time processing ensures that anomalies are detected as they occur. Advanced analytics engines continuously analyze data streams, enabling immediate identification of unusual patterns. This capability is crucial for mitigating risks such as cyberattacks or system failures. For example, detecting anomalies in network traffic in real time allows you to respond quickly to potential security threats.

Alerting systems play a vital role in this process. They generate context-aware alerts that help you understand the root cause of anomalies. These alerts prioritize critical issues, enabling faster decision-making. By integrating real-time processing and intelligent alerting, you enhance your system's performance and ensure robust monitoring.

Observability and Monitoring Tools

Observability and monitoring tools are essential for maintaining the performance of your real-time anomaly detection system. These tools help you track system behavior, identify issues, and ensure smooth operations. By using observability tools, you gain a comprehensive view of your system's health and detect anomalies before they escalate into major problems.

Monitoring tools continuously collect and analyze data from various sources, such as logs, metrics, and traces. This data provides valuable insights into your system's performance. For example, you can monitor CPU usage, memory consumption, and network traffic to identify unusual patterns. These insights allow you to take immediate action to prevent downtime or security breaches.

Observability tools go beyond basic monitoring by offering advanced analysis capabilities. They enable you to understand the root cause of anomalies through detailed data visualization and correlation. For instance, you can use dashboards to view real-time metrics and identify trends that indicate potential issues. This level of observability ensures that your system remains reliable and efficient.

Alerting systems play a critical role in monitoring. They notify you when performance metrics deviate from expected values. These alerts prioritize critical issues, helping you focus on resolving the most urgent problems. By integrating observability and monitoring tools, you enhance your system's performance and maintain its stability.

Incorporating these tools into your anomaly detection system ensures proactive management. You can address issues before they impact your operations. This approach improves overall system performance and reduces the risk of unexpected failures.

Step-by-Step Guide to Building the System

Setting Up Alibaba Cloud Elasticsearch

To begin, you need to set up an Alibaba Cloud Elasticsearch cluster. Follow these steps to ensure a smooth setup process:

1）Create an Elasticsearch cluster, preferably version 8.5, for optimal performance.

2）Access the Kibana console of your cluster and add sample data for analysis.

3）Set up an unsupervised machine learning task to analyze behaviors, such as web server access patterns.

4）Create a supervised learning task to predict outcomes, like flight delays, using historical data.

5）Evaluate the model's metrics to ensure reliability and accuracy.

This setup forms the backbone of your anomaly detection system. It enables you to leverage aiops capabilities for real-time insights and efficient automation of monitoring tasks.

Configuring Data Ingestion Pipelines

Data ingestion pipelines are essential for feeding your system with high-quality data. To configure these pipelines:

1）Log in to the GlassFlow WebApp

2）Create a new pipeline and configure a data source.

3）Define the transformer to perform necessary data transformations.

4）Configure a data sink to store the processed data.

5）Confirm the pipeline and copy the credentials for integration.

Ensure the data stream matches the expected format and arrives at the predefined rate. This step guarantees that your system processes accurate and consistent data, enhancing the effectiveness of aiops-driven automation.

Training and Deploying ML Models

Training and deploying machine learning models is the core of anomaly detection. Use unsupervised learning models, such as Isolation Forest, to identify unusual patterns in your data. For predictive tasks, supervised learning models like regression or classification algorithms work best.

Once trained, deploy the models within your Elasticsearch cluster. Use the Kibana console to monitor their performance and refine them as needed. This process ensures your system remains adaptive and capable of handling real-time anomaly detection with minimal manual intervention. Automation of these tasks further enhances the efficiency of your aiops strategy.

Implementing Real-Time Anomaly Detection

To implement real-time anomaly detection, you need to integrate machine learning models into your system and configure them for continuous analysis. Start by deploying the trained models into your Alibaba Cloud Elasticsearch cluster. Use the Kibana interface to manage and monitor these models effectively.

Set up real-time data streams to feed the system with live data. Tools like Logstash or Beats can help you collect and forward data from various sources. Ensure the data pipeline is robust and capable of handling high volumes without delays. This step ensures the system processes data in real time, enabling immediate anomaly detection.

Next, configure detection rules and thresholds. These rules define what constitutes an anomaly in your system. For example, you might set thresholds for CPU usage or network traffic. The machine learning models will analyze incoming data against these rules to identify unusual patterns.

Finally, integrate alerting mechanisms. Use Elasticsearch's built-in alerting features to notify you of incidents as they occur. Alerts can be sent via email, SMS, or integrated with third-party tools like Slack. This setup ensures you respond to incidents promptly, minimizing potential disruptions.

Monitoring and Optimizing the System

Continuous monitoring and performance optimization are essential for maintaining an effective anomaly detection system. Use observability tools in Alibaba Cloud Elasticsearch to track system metrics like latency, throughput, and error rates. These metrics provide insights into the system's health and help you identify areas for improvement.

Implement predictive and proactive monitoring to anticipate potential issues. For instance, analyze historical data to predict future incidents. This approach allows you to address problems before they impact your operations.

Optimize the system by refining machine learning models and detection rules. Regularly evaluate model performance using metrics like precision and recall. Adjust thresholds and retrain models as needed to improve accuracy. Additionally, ensure your data pipelines remain efficient by monitoring their performance and addressing bottlenecks.

Benefits and Use Cases

Advantages of Real-Time Anomaly Detection with Alibaba Cloud

Real-time anomaly detection with Alibaba Cloud offers numerous advantages that enhance your IT operations. These benefits ensure your systems remain secure, efficient, and resilient. The following table highlights the key advantages:

Benefit Description
Real-time Detection: Identifies various attacks and threats as they occur, ensuring timely responses to security risks.
High-risk Operation Identification: Utilizes intelligent algorithms to pinpoint high-risk operations effectively.
Full Database Audit: Conducts real-time audits of all database activities to maintain security.
Abnormal Access Source Identification: Quickly detects new or unusual access sources to databases.
Custom Performance Dashboards: Supports tailored dashboards for performance monitoring and comparative analysis.
Automatic SQL Query Identification: Detects abnormal SQL queries and manages traffic to maintain business continuity.
Global Load Distribution Review: Automatically reviews SQL queries based on load distribution to enhance database stability.

These features empower you to address security and threat detection challenges proactively. By leveraging Alibaba Cloud's capabilities, you can maintain robust observability and ensure uninterrupted IT performance.

Real-World Use Cases Across Industries

Alibaba Cloud's real-time anomaly detection finds applications across various industries. It helps you tackle industry-specific challenges effectively. The table below illustrates some real-world use cases:

Industry	Application Description
Finance	Detects fraudulent transactions and unusual trading activities.
Healthcare	Monitors patient vitals for early signs of medical conditions.
Manufacturing	Identifies equipment malfunctions to prevent downtime.
Cybersecurity	Detects network intrusions and suspicious activities.
Retail	Analyzes sales data to spot irregular purchasing patterns.

These examples demonstrate how Alibaba Cloud supports diverse industries in improving observability and addressing IT challenges. Whether you aim to enhance cybersecurity or optimize manufacturing processes, real-time anomaly detection provides actionable insights.

Enhancing aiops with Alibaba Cloud Elasticsearch

Alibaba Cloud Elasticsearch significantly enhances aiops capabilities. Its AI Assistant leverages RAG technology and large language models to improve anomaly monitoring, alert handling, and data analysis. You can visualize complex data through statistical charts, making it easier to interpret even without prior knowledge of Elasticsearch query syntax.

The AI Assistant plays a vital role in real-time monitoring. It identifies abnormal conditions, analyzes error logs, and proposes defense strategies against potential threats. These features ensure your IT systems remain secure and efficient. By integrating advanced observability tools, Alibaba Cloud Elasticsearch empowers you to automate monitoring and optimize system performance. This approach strengthens your aiops strategy, enabling you to address challenges with precision and agility.

Building a real-time anomaly detection system with Alibaba Cloud Elasticsearch ML Modules equips you with powerful tools to monitor, analyze, and secure your IT operations. Observability plays a critical role in this process by offering:

Enhanced system stability, which optimizes performance and boosts availability.

Faster troubleshooting, reducing downtime through quick issue identification.

Scalability, helping you adapt to changes and support growth.

By leveraging these capabilities, you can ensure resilient and efficient systems. Start exploring Alibaba Cloud Elasticsearch ML Modules today to transform your IT operations with cutting-edge aiops and observability tools.

FAQ

What is real-time anomaly detection, and why is it important?

Real-time anomaly detection identifies unusual patterns or behaviors in data as they occur. It helps you address potential issues immediately, preventing disruptions and ensuring system stability. This capability is crucial for maintaining efficient IT operations and enhancing overall system reliability.

How does Alibaba Cloud Elasticsearch support anomaly detection?

Alibaba Cloud Elasticsearch uses advanced machine learning models to analyze time series data. It detects outliers and unusual patterns in real time. The platform integrates seamlessly with observability tools, enabling you to monitor, analyze, and respond to anomalies effectively.

Can I automate incident management with Alibaba Cloud Elasticsearch?

Yes, you can automate incident management and automation using Alibaba Cloud Elasticsearch. Its AI-powered tools, such as the AI Assistant, streamline anomaly detection and alerting. These features help you resolve incidents faster and improve system performance.

What industries benefit most from anomaly detection systems?

Industries like finance, healthcare, manufacturing, and cybersecurity benefit significantly. For example, anomaly detection helps you identify fraudulent transactions, monitor patient vitals, prevent equipment failures, and detect network intrusions. These applications enhance operational efficiency and security.

How do I ensure my anomaly detection system remains effective?

Regularly update your machine learning models and detection rules. Monitor system performance using observability tools. Analyze historical data to predict future incidents. These practices help you maintain an adaptive and reliable anomaly detection system

If you want to learn more, please click itand have a 30-day free trial.

What is RAG and how Alibaba Cloud Elasticsearch enhances AI search with retrieval-augmented generation

A_Lucas — Tue, 25 Feb 2025 05:27:00 +0000

Retrieval-augmented generation (RAG) combines the strengths of information retrieval and generative AI to create more accurate and context-aware responses. Unlike traditional generative models, which may rely on static or outdated data, RAG integrates real-time information diverse sources like articles, databases, and books. This approach ensures that the generated content remains relevant and reliable.

You can see the impact of RAG in applications like AI search, where it enhances accuracy and relevance. By optimizing information retrieval and enabling real-time data integration, RAG addresses challenges in industries such as healthcare, finance, and customer service. Its ability to provide nuanced, contextually rich results makes it a game-changer for modern search technologies.

Alibaba Cloud Elasticsearch leverages RAG to redefine AI search capabilities, offering enterprises a powerful tool to meet evolving demands.

Understanding Retrieval-Augmented Generation (RAG)

What is RAG?

Retrieval-augmented generation (RAG) is a cutting-edge approach that combines information retrieval with generative AI to produce accurate and contextually relevant responses. Unlike traditional generative AI models, which rely solely on pre-trained data, RAG retrieves up-to-date information from external sources. This retrieval mechanism ensures that the generated content is grounded in factual and current data, enhancing its reliability and relevance. By integrating retrieved information into the generation process, RAG delivers responses that are both precise and context-aware, making it a powerful tool for modern AI applications.

How RAG Works

The Retrieval Process

The retrieval process in RAG involves several key steps to ensure the system gathers the most relevant information for your query. These steps include:

1）Receiving your prompt or query.

2）Searching for relevant information from external sources.

3）Retrieving the most pertinent data to provide additional context.

4）Augmenting your prompt with this enriched context.

5）Submitting the enhanced prompt to a large language model (LLM).

6）Delivering an improved and contextually accurate response to you.

This structured process ensures that the retrieved information aligns with your query, enabling the system to provide grounded and reliable responses.

The Generation Process

Once the retrieval process is complete, the generation phase begins. The system enhances the original prompt with the retrieved data, creating an enriched input for the LLM. This enriched prompt allows the generative AI to produce responses that are informed by the latest information. Techniques like post-retrieval processing with frozen LLMs and fine-tuning LLMs for RAG further optimize the generation process. These methods ensure that the generated content is natural, contextually relevant, and grounded in factual information, minimizing errors and improving user satisfaction.

Applications of RAG

AI-powered search engines

RAG has revolutionized AI-powered search engines by enabling them to deliver highly accurate and context-aware results. By combining retrieval and generation mechanisms, these search engines can process complex queries and provide responses that go beyond simple keyword matching. This capability makes them invaluable for industries requiring precise and reliable information retrieval, such as healthcare and finance.

Customer support and chatbots

In customer support, RAG enhances chatbot functionalities by ensuring responses are based on accurate and relevant data. This approach reduces the risk of incorrect answers and minimizes AI hallucination. Chatbots powered by RAG can handle diverse customer queries effectively, leading to improved interaction quality and higher customer satisfaction. The integration of retrieval-augmented generation also boosts operational efficiency, making it a preferred choice for businesses.

Content creation and summarization

RAG excels in content creation and summarization tasks by leveraging both structured and unstructured data. It can summarize lengthy documents, generate detailed reports, and provide responses in various formats, such as summaries or in-depth explanations. This flexibility makes RAG an essential tool for businesses needing comprehensive and accurate content generation.

Benefits of Retrieval-Augmented Generation in AI Search

Access to Current and Relevant Information

RAG ensures you always have access to fresh information by dynamically retrieving data from diverse sources. Unlike traditional methods that rely on static databases, RAG continuously updates its knowledge base. This capability is crucial in fields like healthcare and finance, where timely and accurate information is essential.

1）RAG's architecture reduces the risk of generating outdated or misleading content.

2）It retrieves the most recent and relevant material, ensuring responses remain factual and reliable.

By leveraging this dynamic retrieval process, you can trust that the information provided aligns with the latest developments, enhancing the quality of your AI search results.

Improved Accuracy and Contextual Understanding

RAG enhances the contextual quality of AI-generated responses by integrating external knowledge into the generation process. This approach ensures that even small pieces of information maintain their relevance and clarity.

1）Contextual retrieval prevents the loss of critical details, which often leads to incomplete or inaccurate responses.

2）By enriching the generative process with external data, RAG delivers results that are both precise and contextually grounded.

This improved contextual understanding allows you to receive high-quality responses tailored to your specific queries, making your search experience more effective and satisfying.

Cost-Effectiveness Compared to Training Large Models

RAG offers a cost-efficient alternative to training large AI models. Instead of investing heavily in computational resources for fine-tuning, you can focus on maintaining a robust retrieval infrastructure.

Approach	Advantages	Challenges
RAG	Avoids resource-intensive fine-tuning; Cost mainly for retrieval infrastructure; Scales with evolving data	Initial setup investment; Querying external databases may incur costs
Fine-Tuning	One-time training investment for well-defined tasks	Requires significant computational resources; High-end GPUs needed; Cumulative costs for new tasks

By adopting RAG, you can achieve high-quality results without the financial burden of training and maintaining large-scale models. This makes it an ideal choice for businesses seeking scalable and cost-effective AI search solutions.

Scalability for Diverse Applications

Retrieval-augmented generation (RAG) demonstrates exceptional scalability, making it suitable for a wide range of industries and use cases. Its ability to integrate real-time data retrieval with generative AI allows you to adapt it to diverse applications, ensuring efficiency and relevance in your operations.

Many organizations have already leveraged RAG to address unique challenges. For example:

1）Delivery support chatbot: DoorDash uses RAG to power chatbots that assist independent contractors, improving response accuracy and resolving issues faster.

2）Customer tech support: LinkedIn integrates RAG with a knowledge graph to reduce customer service resolution times by 28.6%.

3）Internal policies chatbot: Bell employs RAG to provide employees with instant access to updated company policies.

4）SQL query assistance: Pinterest enhances user experience by guiding table selection through RAG-powered SQL query support.

5）Industry classification: Ramp standardizes customer classification using RAG with NAICS codes.

6）Financial research: Analysts rely on RAG to retrieve up-to-date market data and research reports efficiently.

RAG’s scalability also extends to specialized fields like compliance management, technical documentation, and scientific research. Compliance officers can retrieve regulations and guidelines during audits, while developers use RAG to locate technical documentation and code snippets. Researchers benefit from quick access to recent studies and journals tailored to their queries.

By adopting RAG, you can scale your AI capabilities to meet the demands of various industries. Its adaptability ensures that your business remains competitive, whether you aim to enhance customer support, streamline internal processes, or improve decision-making through data-driven insights. This versatility makes RAG an invaluable tool for modern enterprises.

Overview of Alibaba Cloud Elasticsearch

What is Alibaba Cloud Elasticsearch?

Alibaba Cloud Elasticsearch is a fully managed service designed to enhance the capabilities of the open-source Elasticsearch platform. It offers advanced features that improve performance, cost efficiency, and scalability, making it an ideal choice for modern AI applications. By separating storage from computing, it optimizes kernel performance and accelerates data writing, ensuring stability even during high-concurrency operations.

Key highlights of Alibaba Cloud Elasticsearch include:

1）Fully managed service with out-of-the-box functionality.

2）100% compatibility with open-source Elasticsearch features.

3）Integration with Elastic for advanced capabilities like security and machine learning.

4）Cost-effective pay-as-you-go pricing model.

5）Enhanced real-time log analysis and multi-dimensional data querying.

This robust platform empowers you to handle complex AI search tasks with ease, ensuring high performance and reliability.

Key Features Supporting RAG

Scalability and High Performance

Alibaba Cloud Elasticsearch ensures scalability and high performance through innovative optimizations. By separating storage from computing, it manages resources efficiently, allowing you to scale operations seamlessly. The platform also accelerates high-concurrency data writing, enabling smooth handling of large-scale data.

Optimization Technique	Description
Hardware Acceleration	Utilizes SIMD instructions for performance boosts through projects like Panama.
Quantization Techniques	Minimizes resource footprint while maintaining quality.

These features make Alibaba Cloud Elasticsearch a powerful tool for supporting retrieval-augmented generation in AI applications.

Real-time Data Retrieval

Real-time data retrieval is a cornerstone of Alibaba Cloud Elasticsearch, enabling you to access the latest information for your RAG-based solutions. This capability ensures that your AI models provide accurate and up-to-date responses, overcoming the limitations of static databases.

Key benefits include:

1）Dynamic retrieval of the most recent and relevant material.

2）Factually correct answers based on current data.

3）Enhanced applications in fields like healthcare, finance, and legal consulting.

With real-time retrieval, you can trust that your AI search results remain relevant and reliable.

Seamless Integration with AI Models

Alibaba Cloud Elasticsearch simplifies the integration of AI models, allowing you to load custom models directly into the cluster for end-to-end processing. This feature bridges search, ranking, and AI services, enabling advanced functionalities like semantic search and relevance ranking.

Additional integration capabilities include:

1）Support for diversified models and hybrid retrieval technologies.

2）Management of custom transformer models for context-specific searches.

3）Transition from traditional search to AI-powered semantic search.

These integration features enhance the adaptability of your RAG-based solutions, ensuring they meet diverse business needs.

Why Alibaba Cloud Elasticsearch is Ideal for RAG

Alibaba Cloud Elasticsearch stands out as an ideal platform for implementing RAG-based solutions. Its high-performance semantic search capabilities allow you to retrieve and generate accurate, context-aware responses without relying on exact keyword matches. The platform also provides full-text answers to complex questions and enables personalized recommendations, making it suitable for a wide range of applications.

By leveraging Alibaba Cloud Elasticsearch, you can unlock the full potential of retrieval-augmented generation, enhancing the accuracy, scalability, and efficiency of your AI search solutions.

How Alibaba Cloud Elasticsearch Enhances RAG-based AI Search

Optimizing the Retrieval Process

Alibaba Cloud Elasticsearch optimizes the retrieval process in RAG by employing advanced techniques that improve performance and relevance. These optimizations ensure that your AI-powered applications deliver accurate and contextually rich results. The following table highlights key enhancements:

Optimization Type	Description
Hardware Acceleration	Reduces query response time from 100ms to about 20ms, enabling faster vector retrieval.
Memory Optimization	Cuts memory usage by 95% through vector quantization, improving indexing speed and efficiency.
Semantic Expansion	Extends vocabulary with related concepts, enhancing semantic understanding.
Hybrid Search Strategies	Combines text and vector search for improved relevance and user experience.
Ranking Mechanism	Uses BM25 for initial ranking and refined models for secondary ranking to ensure top results.
Model Integration	Allows seamless loading of custom models for end-to-end processing within the cluster.

These features ensure that your retrieval-augmented generation workflows operate efficiently, providing precise and relevant data for the generation phase.

Enabling Real-time Data Updates

Real-time data updates are critical for maintaining the accuracy and relevance of RAG-enabled search applications. Alibaba Cloud Elasticsearch empowers you to build AI-powered solutions that process data dynamically. Key benefits include:

1）Automatic updates that enhance responsiveness and ensure data integrity.

2）Robust access control and security monitoring to protect sensitive information.

3）Seamless integration with large language models for improved semantic understanding.

The platform also supports hybrid retrieval strategies, combining multiple methods to refine search results. This capability ensures that your AI applications remain up-to-date and deliver factually correct responses, even in fast-changing industries like finance and healthcare.

Supporting Large-scale AI Applications

Alibaba Cloud Elasticsearch provides the scalability needed to support large-scale AI applications. You can develop AI-driven search solutions that integrate seamlessly with large language models. These applications benefit from features like automatic updates, robust access control, and security monitoring. Whether you aim to enhance enterprise search capabilities or build intelligent customer service tools, the platform adapts to your needs.

By leveraging these capabilities, you can scale your retrieval-augmented generation workflows to handle complex queries and vast datasets. This scalability ensures that your AI applications remain efficient and reliable, even as your business grows.

Enhancing Search Accuracy and Relevance

Search accuracy and relevance are critical for delivering meaningful results in AI-powered applications. Alibaba Cloud Elasticsearch employs advanced techniques to ensure your retrieval-augmented generation (RAG) workflows consistently produce precise and contextually relevant outputs.

One of the standout features is its ranking mechanism, which combines traditional and modern approaches to refine search results. Initially, the system uses BM25 to calculate document weights based on term frequency and location. It then applies integrated learning models for secondary ranking, ensuring the most relevant results appear at the top. This dual-layered approach significantly enhances the quality of search outcomes.

Alibaba Cloud Elasticsearch also leverages a hybrid retrieval strategy. By integrating text-based, sparse, and dense vector indexes, it improves retrieval precision and efficiency. This combination allows you to handle diverse query types, from simple keyword searches to complex semantic queries, with ease.

Another key capability is intent understanding. The platform analyzes user queries to identify their underlying intent, optimizing result sorting and ensuring the content aligns with user expectations. This feature is particularly valuable in applications like customer support, where accurate responses are essential.

The platform’s performance improvements further highlight its effectiveness. In knowledge base Q&A scenarios, Alibaba Cloud Elasticsearch achieved a remarkable accuracy increase from 48% to over 95%. This leap underscores its ability to deliver reliable and relevant results across various use cases.

Technique	Description
Ranking Mechanism	Utilizes BM25 to determine document weights based on frequency and location, followed by a secondary ranking using integrated learning models.
Hybrid Retrieval Strategy	Combines text, sparse, and dense vector indexes for improved retrieval precision and efficiency.
Intent Understanding	Analyzes user queries to optimize result sorting, ensuring content relevance.
Performance Improvement	Achieved a significant increase in accuracy from 48% to over 95% in knowledge base Q&A scenarios through advanced search techniques.

By adopting these advanced techniques, you can ensure your RAG-based systems deliver accurate, relevant, and context-aware results. This capability empowers your business to meet user expectations and maintain a competitive edge in AI-driven search applications.

Retrieval-augmented generation (RAG) has transformed AI search by integrating external knowledge for precise, contextually relevant responses. Its ability to retrieve real-time data and provide diverse response formats makes it indispensable for industries requiring accuracy and adaptability. By combining retrieval and generative models, RAG enhances user satisfaction and operational efficiency.

Alibaba Cloud Elasticsearch amplifies these benefits with its advanced features. A 5X improvement in vector performance reduces query response times, while memory optimization minimizes resource usage without compromising quality. The hybrid search strategy and seamless integration with large language models ensure accurate, scalable, and cost-effective solutions. Businesses can leverage these capabilities to improve search outcomes and streamline operations.

Explore Alibaba Cloud Elasticsearch to unlock the full potential of RAG and elevate your AI search applications.

FAQ

What is the difference between retrieval-augmented generation and semantic search?

Retrieval-augmented generation combines information retrieval with generative AI to create context-aware responses. Semantic search focuses on understanding user intent and meaning in queries. While both enhance AI search algorithms, retrieval-augmented generation integrates real-time data, whereas semantic search relies on natural language processing to improve relevance.

How do large language models contribute to retrieval-augmented generation?

Large language models process enriched prompts created by the retrieval mechanism. They generate responses informed by real-time data, ensuring accuracy and contextual relevance. Their ability to understand and generate human-like text makes them essential for retrieval-augmented generation workflows.

Why is real-time data important in AI search algorithms?

Real-time data ensures responses remain accurate and up-to-date. This is crucial for industries like healthcare and finance, where outdated information can lead to errors. Retrieval-augmented generation uses real-time data to enhance natural language processing and improve the relevance of search results.

How does Alibaba Cloud Elasticsearch support semantic search?

Alibaba Cloud Elasticsearch integrates advanced natural language processing techniques and hybrid retrieval strategies. It combines text-based and vector-based searches to improve semantic understanding. This ensures accurate and contextually relevant results, making it ideal for applications requiring precise query handling.

What industries benefit most from retrieval-augmented generation?

Industries like healthcare, finance, and customer service benefit significantly. Retrieval-augmented generation provides accurate, context-aware responses by leveraging large language models and real-time data. This improves decision-making, operational efficiency, and user satisfaction across diverse applications.

If you want to learn more, please click it and have a 30-day free trial.

Mastering Vector Search: How Alibaba Cloud’s Inference API Enhances Elasticsearch

A_Lucas — Fri, 21 Feb 2025 06:55:31 +0000

Vector search has revolutionized how modern applications handle data retrieval. By leveraging advanced indexing techniques, it significantly improves both speed and accuracy. Industries like e-commerce, healthcare, and education now rely on vector search to deliver personalized recommendations, and create intelligent learning platforms.

Alibaba Cloud Elasticsearch takes this innovation further by integrating with the Inference API. This integration enables you to utilize dense and sparse embeddings, enhancing the relevance of search results. With features like semantic reranking and support for complex queries, AlibabaCloud AI Search empowers you to build smarter, AI-driven applications.

Understanding Vector Search

What is Vector Search?

Vector search is a powerful method for retrieving information by analyzing the relationships between data points in a multi-dimensional space. Unlike traditional keyword-based searches, vector search relies on mathematical representations called vectors. These vectors capture the semantic meaning of data, enabling more accurate and context-aware results.

The process involves several key steps:

Vectorization: Models like Word2Vec or BERT convert items into numerical vectors.

Indexing: The system organizes these vectors into a structure optimized for efficient searching.

Query Vectorization: Search queries are transformed into vectors for comparison.

Similarity Search: The system compares query vectors to indexed vectors using distance metrics.

Ranking: Results are ordered based on their similarity to the query vector.

Retrieval: The top-ranked items are returned as search results.

This approach powers modern applications like semantic search, recommendation systems, and image retrieval, making it a cornerstone of AI search technologies.

Sparse vs. Dense Vector Search

Sparse and dense vector search differ in how they represent and process data. The table below highlights their key distinctions:

Feature	Sparse Vector Search	Dense Vector Search
Dimensionality	High, with many zero values	Lower, with few or no zero values
Information Richness	Less semantically rich	Semantically rich, capturing content meaning
Similarity Measurement	Based on keyword matches and frequency	Uses algorithms like cosine similarity

Dense vector search, supported by platforms like AlibabaCloud AI Search, excels in capturing the deeper semantic meaning of data. This makes it ideal for applications requiring advanced text embedding services and contextual understanding.

Applications of Vector Search in Modern Use Cases

Vector search has transformed industries by enabling smarter and more efficient data retrieval. Here are some notable applications:

E-Commerce:

Personalized product recommendations based on user preferences.

Visual search capabilities that let users find products using images.

Enhanced search experiences through semantic understanding of natural language queries.

Healthcare:

Medical image search for diagnostics by comparing new images with historical cases.

Retrieval of patient information from electronic health records using semantic analysis.

Clinical decision support systems that recommend tests and treatments.

By integrating vector search into platforms like Elasticsearch, you can unlock these capabilities and more. AlibabaCloud AI Search further enhances this by providing robust tools for embedding and analysis, making it a leader in AI-driven search solutions.

Integrating Alibaba Cloud’s Inference API with Elasticsearch

Setting Up the Inference API

To begin integrating the Inference API with Elasticsearch, you need to follow a structured setup process. This ensures seamless communication between the two platforms. Here are the steps:

1）Create an endpoint in Elasticsearch by specifying the service as alibabacloud-ai-search. Provide essential service settings, including workspace, host, service ID, and API keys.

2）Use a PUT request to configure the text embedding endpoint with the required parameters.

3）Verify the endpoint creation by checking the response from Elasticsearch.

4）Test the endpoint by issuing a POST request to perform inference and generate embeddings.

This setup allows you to leverage the full potential of AlibabaCloud AI Search for advanced vector search capabilities.

Creating Inference Endpoints

Creating inference endpoints is a critical step in enabling Elasticsearch to utilize Alibaba’s AI services. Follow these steps to create and test your endpoint:

1）Define the service as alibabacloud-ai-search and provide the necessary settings, such as workspace, host, service ID, and API keys.

2）Use the following command to create a text embedding endpoint:

PUT _inference/text_embedding/ali_ai_embeddings
{
    "service": "alibabacloud-ai-search",
    "service_settings": {
        "api_key": "<api_key>",
        "service_id": "ops-text-embedding-001",
        "host": "xxxxx.platform-cn-shanghai.opensearch.aliyuncs.com",
        "workspace": "default"
    }
}

3）Confirm the endpoint creation by reviewing the response from Elasticsearch.

4）Test the endpoint by calling the perform inference API with:

POST _inference/text_embedding/ali_ai_embeddings
{
    "input": "What is Elastic?"
}

These steps ensure that your Elasticsearch instance is ready to handle advanced AI-driven tasks.

Configuring Alibaba Cloud Elasticsearch for Vector Search

After creating an inference endpoint, configure Alibaba Cloud Elasticsearch to optimize it for vector search. Start by ensuring that your Elasticsearch instance supports vector-based indexing. Use the aliyun-knn plugin to enable efficient similarity searches. Configure the dimensions and similarity measures to align with your use case.

Next, integrate the embeddings generated by the Inference API into your Elasticsearch vector database. This allows you to perform semantic searches, reranking, and other advanced operations. Test the configuration by issuing queries and validating the results. AlibabaCloud AI Search simplifies this process, enabling you to focus on building intelligent applications.

By following these steps, you can unlock the full potential of Alibaba Cloud Elasticsearch for creating an indexing service instance tailored to your needs.

Testing and Validating the Integration

Testing and validating the integration between Alibaba Cloud Elasticsearch and the inference API ensures that your setup functions as expected. This step is crucial for identifying potential issues and optimizing performance.

1）Run Sample Queries:
Start by executing sample queries against your configured inference endpoint. Use a variety of inputs to test the system's ability to generate embeddings and return relevant results. For example, you can input a query like:

POST _inference/text_embedding/ali_ai_embeddings
{
    "input": "Find similar products to this item."
}

Observe the response to ensure the embeddings align with your expectations.

2）Validate Search Results:
Perform searches using the vector database in Elasticsearch. Compare the returned results with the expected outcomes. If the results lack relevance, revisit your embedding model or similarity measures to refine the configuration.

3）Monitor Performance Metrics:
Use Elasticsearch's built-in monitoring tools to track key performance indicators like query latency and throughput. These metrics help you identify bottlenecks and ensure the system meets your application's requirements.

4）Iterate and Optimize:
Based on your findings, fine-tune the embeddings, similarity measures, or indexing configurations. This iterative process ensures your vector search implementation remains robust and efficient.

By thoroughly testing and validating the integration, you can confidently deploy a reliable and high-performing vector search solution powered by Alibaba Cloud Elasticsearch.

Practical Use Cases of Enhanced Vector Search

Image Search and Retrieval

Enhanced vector search has transformed how you can perform image search and retrieval. By representing visual content as vectors, systems can identify similar patterns with remarkable accuracy. This capability is particularly impactful in e-commerce. For instance:

1）Users can upload images to find visually similar products, improving their shopping experience.

2）Online furniture retailers can match uploaded images of couches with similar items in their inventory.

Platforms like AlibabaCloud AI Search leverage this technology to deliver precise results. Additionally, vector search powers advanced systems like Google Images, enabling them to analyze visual patterns effectively. These applications highlight how vector search enhances user engagement and satisfaction in image-based searches.

Geospatial Analysis and Location-Based Applications

Geospatial analysis has become a cornerstone for applications that rely on location-based data. By leveraging vector search, you can unlock advanced geospatial capabilities, enabling precise and efficient handling of geospatial queries. Alibaba Cloud Elasticsearch, with its support for geo_point data types, empowers you to build intelligent systems that extract actionable location-based insights.

When working with geospatial data, you often need to process and analyze geo_point coordinates. These coordinates represent specific locations on the Earth's surface, such as latitude and longitude. For example, you can use geo_point fields to store the location of a delivery address, a retail store, or a user’s current position. With Alibaba Cloud Elasticsearch, you can index and query this data seamlessly, enabling real-time geospatial analysis.

Vector search enhances geospatial queries by enabling similarity-based searches. Instead of relying solely on traditional distance calculations, you can use embeddings to capture the contextual relationships between locations. This approach is particularly useful for applications like ride-hailing services, where matching drivers to riders requires analyzing both proximity and contextual factors like traffic patterns or preferred routes.

Location-based insights derived from geospatial analysis can transform industries. In logistics, you can optimize delivery routes by analyzing geo_point data for warehouses and customer addresses. In retail, you can identify high-performing store locations by correlating sales data with customer demographics. By integrating Alibaba Cloud Elasticsearch, you gain the tools to process location-based data efficiently and extract meaningful insights.

By combining vector search with geospatial capabilities, you can create innovative solutions that leverage the power of location-based data. Whether you’re building a navigation app or a recommendation system, Alibaba Cloud Elasticsearch provides the foundation for success.

Advanced Features and Optimization

Performance Tuning for Vector Search

Optimizing vector search in Elasticsearch ensures faster and more accurate results. You can achieve this by implementing several key strategies:

1）Reduce Vector Dimensions: Simplify data structures using techniques like PCA or UMAP. This reduces computational complexity and speeds up searches.

2）Index Efficiently: Use Approximate Nearest Neighbor (ANN) algorithms such as HNSW or FAISS to enhance indexing performance.

3）Batch Queries: Process multiple queries in a single request to minimize overhead and improve throughput.

4）Use Caching: Cache frequently accessed queries to reduce computational load and response times.

5）Monitor Performance: Regularly analyze metrics like query latency and throughput to identify bottlenecks and optimize configurations.

These techniques ensure that your vector search implementation remains efficient, even as your dataset grows. Alibaba Cloud Elasticsearch provides robust tools to support these optimizations, enabling you to deliver high-performance search solutions.

Leveraging the aliyun-knn Plugin

The aliyun-knn pluginenhances vector search capabilities in Alibaba Cloud Elasticsearch by leveraging the Proxima vector library developed by Alibaba DAMO Academy. This plugin offers several advanced features:

1）It supports distributed searches with multiple replica shards, ensuring scalability and fault tolerance.

2）It enables real-time incremental synchronization and near-real-time searches, making it ideal for dynamic datasets.

3）It supports algorithms like HNSW and Linear Search, which are effective for both small and large datasets.

The plugin powers applications like image search, video fingerprinting, and recommendation systems. For example, Alibaba uses it in platforms like Pailitao and Taobao to deliver precise and efficient search results. By integrating this plugin, you can unlock advanced functionalities for machine learning and AI-driven applications.

Best Practices for Scalability and Efficiency

Scalability and efficiency are critical for handling large-scale vector search implementations. Follow these best practices to ensure optimal performance:

1）Choose a database that aligns with your application's requirementsand supports seamless integration.

2）Optimize indexing by using techniques like inverted indexes or KD-trees to improve search operations.

3）Implement distributed architectures to handle large datasets effectively. Horizontal scaling and sharding allow you to manage growing data volumes.

4）Use quantization techniquesto reduce vector size and storage requirements.

5）Cache frequently accessed queries to minimize system load and improve response times.

6）Regularly monitor performance metrics and optimize indexes to maintain efficiency.

Alibaba Cloud Elasticsearch simplifies these processes with its built-in tools and support for distributed architectures. By following these strategies, you can build scalable and efficient vector search solutions tailored to your needs.

Integrating Alibaba Cloud’s Inference API with Elasticsearch unlocks measurable benefits for vector search. You can achieve faster query response times, reduced memory utilization, and enhanced semantic understanding. For instance:

Metric	Before Integration	After Integration	Improvement
Query Response Time	100ms	20ms	80% reduction
Memory Utilization	Original demand	25% of original	75% reduction
Query Speed Improvement	N/A	2-5x faster	N/A

This integration also provides a fully-managed service, balancing scale and performance while minimizing operational costs. By leveraging Alibaba Cloud Elasticsearch, you can focus on building innovative applications without worrying about infrastructure management.

Start implementing this integration today to transform your search capabilities and stay ahead in the AI-driven era.

FAQ

What is the purpose of integrating the Inference API with Elasticsearch?

The integration allows you to enhance search capabilities by leveraging AI-driven embeddings and semantic reranking. It improves the relevance and accuracy of search results, enabling advanced use cases like recommendation systems and semantic text search.

How does the aliyun-knn plugin improve vector search performance?

The aliyun-knn plugin optimizes similarity searches by using efficient algorithms like HNSW. It supports distributed architectures and real-time synchronization, ensuring scalability and faster query responses for large datasets.

Can you use Alibaba Cloud Elasticsearch for hybrid search?

Yes, you can combine dense and sparse embeddings to implement hybrid search. This approach enables you to handle both keyword-based and semantic queries, delivering more comprehensive search results.

What are the prerequisites for setting up the Inference API?

You need an Alibaba Cloud account, an Elasticsearch instance, and API credentials. Ensure your Elasticsearch instance supports vector-based indexing and has the aliyun-knn plugin installed.

How do you monitor the performance of vector search?

Use Elasticsearch’s built-in monitoring tools to track metrics like query latency and throughput. Regularly analyze these metrics to identify bottlenecks and optimize configurations for better performanc

Singapore Meetup Invitation：Decoding the Future of AI Search

A_Lucas — Wed, 19 Feb 2025 08:30:11 +0000

Join Us for an Exclusive AI Search Meetup Hosted by Alibaba Cloud!

We are excited to extend a warm invitation to you for an exclusive AI Search Meetup hosted by Alibaba Cloud. Dive deep into the latest advancements in AI-powered search technologies at this unique event.

Our gathering will feature expert speakers from both Elastic and Alibaba Cloud, who will share their insights and experiences on harnessing AI to enhance search capabilities and drive innovation in the industry.

For detailed information about the meetup, please see the image.

We look forward to welcoming you to an afternoon of knowledge sharing and networking!

If you want to learn more, Please click here.

How to manage clusters in Alibaba Cloud Elasticsearch for beginners

A_Lucas — Mon, 17 Feb 2025 05:26:36 +0000

Managing clusters in Alibaba Cloud Elasticsearch involves overseeing the resources and configurations that power your search and analytics engine. You ensure that you can effectively manage clusters to operate efficiently, scale seamlessly, and remain secure. Effective cluster management offers several benefits: faster query speeds through optimized configurations, improved processing power with high-spec servers, and elastic scaling for additional disk space or node upgrades. Security also improves as clusters stay isolated in VPCs, access is controlled via whitelists, and role-based access control strengthens identity verification. This tutorial simplifies the process, making it accessible even if you're just starting out.

Setting Up Your Alibaba Cloud Elasticsearch Cluster

Prerequisites for Cluster Setup

Alibaba Cloud account and permissions

Before creating an Elasticsearch cluster, you need an Alibaba Cloud account. You can register for one through the official registration page. Ensure that your account has completed real-name verification. This step is essential for accessing Alibaba Cloud services.

You also need the necessary permissions to manage resources. If you are part of a team, confirm that your account has the required roles assigned.

Understanding billing options

Alibaba Cloud Elasticsearch offers two billing methods:

Billing Method	Description
Subscription	Requires an upfront fee and is more cost-effective for long-term use.
Pay-as-you-go	Charged hourly, suitable for short-term use or testing, and can be released at any time.

Choose the billing method that aligns with your project needs. For example, if you are testing or experimenting with the ELK stack, the pay-as-you-go option provides flexibility.

Steps to Create an Alibaba Cloud Elasticsearch Cluster

Choosing the right Elasticsearch version

When creating an Elasticsearch instance, select a version compatible with your application. Alibaba Cloud Elasticsearch supports multiple versions, ensuring flexibility for different use cases. Always choose the latest stable version for optimal performance and security.

Selecting instance types and storage

Alibaba Cloud Elasticsearch separates storage from computing, reducing costs and improving performance. Select an instance type based on your workload. For example, a high-spec instance is ideal for data-intensive tasks.

The following table shows how cluster configurations affect response times:

Cluster nodes	Average RT for 10 concurrent retrievals	Average RT for 50 concurrent retrievals	Average RT for 100 concurrent retrievals	Average RT for 200 concurrent retrievals
1	77ms	459ms	438ms	1001ms
3	38ms	103ms	162ms	298ms
10	21ms	36ms	48ms	81ms

Initial Cluster Configurations

Configuring cluster name and region

When building an Elasticsearch cluster, assign a unique name to identify it easily. Choose a region close to your users to minimize latency. Specifying a zone is unnecessary during setup, simplifying the process.

Setting up IP whitelists for access control

IP whitelists enhance security by restricting access to specific IP addresses. Add your host's IP address to the whitelist to enable public network access. This step prevents unauthorized access and ensures your data remains secure.

Tip: Regularly update your IP whitelist to reflect changes in your network configuration.

Managing Clusters in Alibaba Cloud Elasticsearch

Adding and Removing Nodes

When and why to scale your cluster

Scaling your cluster ensures it can handle increasing workloads or optimize resource usage. You should add nodes when your cluster experiences high traffic, increased data ingestion, or slow query responses. Removing nodes may be necessary to reduce costs or reallocate resources. Maintaining at least two data nodes is essential for reliability. For multi-zone clusters, balance the number of nodes across zones to enhance stability.

Steps to add or remove nodes

Follow these steps to manage nodes effectively:

Adding Nodes:

Use the Alibaba Cloud console to add nodes to your cluster.

Ensure the new nodes have sufficient resources, such as memory and disk space.

Elasticsearch automatically redistributes shards across the new nodes

Removing Nodes:

Verify that the cluster has enough resources to handle the workload after node removal.

Use the GET _cat/indices?v command to check resource usage.

Adjust the number of replica shards to avoid allocation errors.

Consideration	Details
Resource Management	Ensure sufficient resources after node removal to prevent shard allocation errors.
Shard Allocation	Verify that replica shards are fewer than data nodes. Adjust replicas if needed.
Cluster Stability	Maintain at least two data nodes. Balance nodes across zones for multi-zone clusters.

Scaling Your Cluster

Vertical scaling (upgrading instance types)

Vertical scaling involves upgrading the hardware of existing nodes. This method is ideal for consistent high-resource demand. For example, upgrading to a high-spec instance improves performance for data-intensive tasks. Vertical scaling is efficient but may require downtime during the upgrade process.

Horizontal scaling (adding more nodes)

Horizontal scaling adds more nodes to distribute the workload. Elasticsearch's distributed architecture simplifies this process. This method is suitable for applications with varying traffic patterns or high availability needs. Horizontal scaling minimizes downtime and enhances workload distribution.

Scenario	Preferred Scaling Type	Explanation
Traffic patterns	Horizontal Scaling	Adapts to varying traffic with the ability to add or remove resources.
Resource efficiency	Vertical Scaling	Efficient for consistent high-resource demand by boosting existing capabilities.
Application architecture	Horizontal Scaling	Suitable for applications designed to run on multiple servers.
Downtime tolerance	Horizontal Scaling	Facilitates less downtime, ideal for high availability needs.
Workload distribution	Horizontal Scaling	Excels in distributing workloads across multiple nodes.

Configuring Security Settings

Updating IP whitelists

IP whitelists restrict access to specific IP addresses, enhancing security. Regularly update your whitelist to reflect changes in your network. Use the Alibaba Cloud console to add or remove IP addresses. This step ensures only authorized users can access your cluster.

Enabling alert features for monitoring

Enable alert features to monitor your cluster's health and performance. Alerts notify you of critical events, such as high CPU usage or abnormal cluster statuses. Configure alerts for key metrics like disk usage and cluster health. This proactive approach helps you address issues before they escalate, ensuring your cluster operates efficiently.

Monitoring and Maintaining Your Cluster

Monitoring Cluster Health

Using the Alibaba Cloud console

Monitoring your cluster's health ensures optimal performance and stability. The Alibaba Cloud console provides tools to simplify this process:

Cluster Check: Automates health checks to identify and resolve issues proactively. This feature minimizes downtime and enhances stability.

CloudMonitor: Tracks metrics and detects service availability. It enables you to monitor resource usage and health status while setting alarms for critical metrics.

These tools help you stay informed about your cluster's condition and act promptly when issues arise.

Key metrics to track (e.g., CPU, memory, disk usage)

Tracking key metrics is essential for maintaining your Elasticsearch cluster. Use the following table to understand the most critical metrics:

Metric	Description
Cluster Status	Indicates overall health (Red, Yellow, Green).
Nodes	Total number of nodes, including successful and failed nodes.
JVM Heap Usage	Percentage of JVM heap memory used.
CPU Usage	Percentage of CPU used by Elasticsearch.
Disk Usage	Percentage of disk space used.
Active Shards	Number of active shards in the cluster.
Query Cache Hit Ratio	Ratio of cache hits to total requests.

Monitoring these metrics helps you identify bottlenecks and optimize resource usage.

Managing Indices

Creating, deleting, and optimizing indices

Efficient index management improves query performance and reduces resource consumption. Follow these best practices:

1)Create one type per index and separate indexes for data with different fields. This approach avoids large index queries.

2)Merge read-only indexes into larger segments to reduce fragmentation and memory usage.

3)Disable historical data indexes that are not queried. This saves JVM memory.

4)Use batch requests for better performance. Commit 5 MB to 15 MB of data at a time.

These practices ensure your indices remain optimized and manageable.

Performing Backups

Setting up automated snapshots

Automated snapshots protect your data and simplify recovery. Follow these steps to set up snapshots:

1)Register a snapshot repository on Alibaba Cloud OSS.

2)Configure the OSS bucket with Standard storage class and Public Read ACL.

3)Use Snapshot Lifecycle Management (SLM) to automate snapshot handling and retention.

This setup ensures your data remains secure and recoverable.

Restoring data from backups

Restoring data from backups is straightforward. You can restore snapshots to the original cluster or a different one using a shared OSS repository. Follow these steps:

1)Register a snapshot repository on Alibaba Cloud OSS.

2)Create automatic or manual snapshots of your data.

3)Use SLM for automated snapshot handling.

4)Restore data to the desired cluster.

This process ensures reliable data recovery and management.

Troubleshooting Common Issues in Alibaba Cloud Elasticsearch

Resolving Connection Issues

Diagnosing access problems

Connection issues can disrupt your ability to interact with your Elasticsearch cluster. Identifying the root cause is the first step to resolving these problems. The table below outlines common causes and their solutions:

Possible cause	Solution
The Elasticsearch cluster cannot be accessed over the Internet.	Ensure the IP address of your device is whitelisted and check network connectivity using ping or telnet commands.
The Elasticsearch cluster cannot be accessed over an internal network.	Verify that the client is in the same VPC and test connectivity with ping commands.
The Elasticsearch cluster is unhealthy.	Check the cluster's health status using the GET _cat/health?v command and monitor resource usage.

Fixing IP whitelist misconfigurations

Misconfigured IP whitelists can block access to your cluster. Follow these steps to fix the issue:

1)Enable public network access and add your host's IP address to the public IP address whitelist.

2)Add the private IP address of your host to the cluster's private IP address whitelist for internal access.

3)Configure a whitelist for your host's IP address for Kibana access, ensuring both public and private IPs are accounted for.

Tip: Regularly update your whitelist to reflect changes in your network configuration.

Addressing Performance Problems

Identifying bottlenecks

Performance bottlenecks often arise from disk read and write limitations. SSDs provide faster speeds compared to HDDs, making them a better choice for Elasticsearch clusters. The number of nodes and the configuration of indexes and replicas also significantly impact performance. Monitoring these factors helps you pinpoint bottlenecks.

Solutions for high resource usage

High resource usage can slow down your cluster. Use these strategies to address the issue:

Expand the cluster by adding more nodes or upgrading existing ones.

Distribute bulk requests into smaller batches to reduce CPU strain.

Cancel long-running searches using the tasks management API.

Avoid resource-intensive searches, such as fuzzy or wildcard queries.

Note: Optimizing your cluster's configuration ensures efficient resource utilization.

Fixing Indexing Errors

Common causes of indexing failures

Indexing errors can occur due to several reasons:

Node capacity issues during spikes in queries or write requests.

Memory overload caused by excessive index cache usage.

Low-spec cluster configurations.

Disk usage exceeding 85%, preventing new shard allocation.

Steps to resolve indexing issues

Follow these steps to fix indexing problems:

1)Run the POST /Index name/_cache/clear?fielddata=true command to clear the cache for indexes.

2)Use the GET /_cat/indices?v command to check shard distribution.

3)Reduce write concurrency and delete invalid indexes to free up resources.

4)Upgrade the cluster configuration if issues persist..

Managing clusters in Alibaba Cloud Elasticsearch becomes easier when you follow a structured approach. Start by creating an account, configuring your cluster, and ensuring proper access control. Use elastic scaling to handle growing workloads and optimize performance with high-spec servers. Regular monitoring is essential for maintaining cluster health and preventing downtime. Proactive maintenance saves resources and ensures smooth operations. To deepen your knowledge, explore Alibaba's documentation on creating clusters, managing access, and using API operations. These resources help you master Elasticsearch cluster management and unlock its full potential.

FAQ

What is the first step in installing Elasticsearch on Alibaba Cloud?

You need to create an Alibaba Cloud account and complete real-name verification. Afterward, access the Alibaba Cloud console to start creating a general-purpose business edition instance. This instance serves as the foundation for installing Elasticsearch and managing your cluster.

How do you begin working with ELK on Alibaba Cloud?

Start by installing the ELK stack. This involves installing Elasticsearch, Logstash, and Kibana. Each component plays a role in data ingestion, storage, and visualization. Ensure you install the required Elasticsearch client to interact with your cluster effectively.

What are the benefits of installing Kibana with Elasticsearch?

Kibana provides a user-friendly interface for visualizing data stored in Elasticsearch. It simplifies data analysis by offering dashboards, charts, and graphs. Installing Kibana enhances your ability to monitor and manage your Elasticsearch cluster.

How do you troubleshoot issues when accessing the Alibaba instance?

Check your IP whitelist settings to ensure your device's IP address is authorized. Verify network connectivity using ping or telnet commands. If problems persist, review your cluster's health status and resource usage through the Alibaba Cloud console.

What is the role of Logstash in the ELK stack?

Logstash processes and transforms data before sending it to Elasticsearch. Installing Logstash allows you to collect data from various sources, filter it, and store it efficiently. This ensures your Elasticsearch cluster receives clean and structured data.

Alibaba Cloud AI Search Solution Explained: Intelligent Search Driven by Large Language Models, Helping Enterprises in Digital

A_Lucas — Sat, 08 Feb 2025 02:09:29 +0000

In the past year, the capabilities of large language models have been growing as the technology for large language models has skyrocketed. These have driven the rapid evolution of our search technology to the next generation, the AI search technology. The rapid development of large language models has not only reshaped the foundation of search technology, but also provided strong support for the digital transformation of various industries.

1） AI search technology trend

AI search technology has the following significant features:

Reconstruction: One is the reconstruction of the technical aspects of AI search. the AI search technology is currently undergoing a comprehensive reconstruction based on the large language model, through which the whole chain of search capabilities have been reorganized, including the redefinition of the capabilities of text parsing, slicing and vectorization. Another is the reconstruction of the way of information access and the shape of the product. While traditional search relies on keyword matching, it now uses more natural language Q&A interactions, which brings new business scenarios, such as virtual digital people, Q&A for enterprise knowledge bases and intelligent customer service for e-commerce platforms.

AI Infrastructure: AI search technology has become an important part of AI native applications, including search vector retrieval, semantic search and retrieval generation technologies, which constitute the infrastructure for many AI applications. This not only improves the efficiency of data processing, but also enhances the user interaction experience with the system, helping enterprises realize more efficient information management and services.

Effectiveness enhancement: Currently, the focus on effectiveness has reached an unprecedented level in both academia and industry. The addition of the large language model makes the search effect a qualitative leap compared with the traditional search. Users are able to obtain relevant information more quickly, improving the efficiency and accuracy of decision-making, especially in complex queries and deep information retrieval scenarios.

However, the introduction of large language models also brings some troubles, especially the problem of phantom rate. In some scenarios that require very high answer accuracy, it becomes difficult to ensure the accuracy of the big model answer, which imposes constraints on enterprises and developers in the landing of AI search services.

2）The efforts and evolution of Alibaba Cloud AI search

With the evolution of the AI era, Alibaba Cloud has made corresponding efforts on the cloud. Alibaba Cloud's search products have historically been divided into two engines: one is the open source engine Elasticsearch, which cooperates with Elastic, and the other is Havenask, a self-developed search engine based on years of experience.

2.1 The evolution of the open source ecosystem:

2017: Alibaba Cloud has a strategic partnership with Elastic and a joint product release. Elasticsearch on Alibaba Cloud is what we are currently using.

2019: We continue to target this fully managed ES product on the cloud, continuously improve the operation and maintenance management capabilities, enhance the intelligent elastic expansion and contraction and monitoring and alarm services, to ensure that customers can obtain stable and efficient services in the process of use, and reduce the cost of development. In addition, Alibaba Cloud has developed some new features based on the ES kernel. For example: support for index construction Indexing service to achieve write acceleration. Through self-developed storage engine Openstore, to help customers in the case of massive data, reduce our storage costs.

2022: Serverless service release, to help small and medium-sized customers and large customers in the case of large amounts of data to reduce costs by 50%, optimize resource allocation.

2023: Alibaba Cloud's products fully enter the field of AI search, starting with version 8.X. The vector search capability has been continuously enhanced. The ability characteristics have also been continuously enhanced.

2.2 Self-research program evolution:

2008: Alibaba Cloud began to self-research an internal open source engine Havenask, is also the history of our support for Taobao, Tmall, including internal double eleven a lot of this more high concurrency, more extreme scenarios of a search engine.

2014: Alibaba Cloud went to explore its commercialization in the cloud one after another. Scenario as the core to help courseware construction, to help customers build scenarios of intelligent search, to provide some industry templates and personalized programs. As well as in e-commerce, content education, games and other industries to do some personalized programs and effect enhancement.

2023: After entering the era of large language models, Havenask is also the first one-stop intelligent Q&A RAG product in China, as well as multimodal RAG products. Until today, Alibaba Cloud has been continuously optimizing the capabilities of RAG and AI search based on the accumulation of internal depth.

2.3 The core focus of search products:

From the past to the present, and even into the future, the core concerns of search products still revolve around three main areas of optimization:

Cost:This may also be a very personal concern of many customers. Because in the era of large language models, we have a lot of this resource is mainly based on the GPU to complete the GPU itself is relatively expensive, so how to help customers reduce costs, is the subsequent evolution of the entire product is an important direction.

Performance:This is mainly two points, one is the massive AI data writing and processing speed. One is the response speed of online queries. These two speeds should realize the performance of good experience of such a requirement, but also Alibaba Cloud products to help you solve a problem.

Effect:Alibaba Cloud and many customers have done online or offline exchanges. Customers have some scenarios where they want the interactive ability of AI, but also want to ensure that the results are 100 percent accurate, and they can't introduce any illusion of a large model, and they can't let the large model do any fabrication. We are introducing the search link, and under the guarantee of accuracy, we are able to apply to the AI's ability to carry out a complete interaction. As for the specific effect optimization is as follows:

3）Alibaba Cloud AI Search Product Introduction

Alibaba Cloud AI Search Open Platform provides five scenario-based products, including LLM Intelligent Q&A Edition, Log Retrieval Serverless Edition, Industry Algorithm Edition, Vector Retrieval Edition, and Graph Retrieval Edition. The bottom layer is based on a very large number of open source search engines, engines including the enterprise version of Elastcsearch, Alibaba's own engine Havenask and Milvus, etc., to facilitate support and docking. Through these products, users are able to realize an end-to-end search solution to quickly obtain the information they need.

Currently, Alibaba Cloud AI Search provides customers with all the AI search capabilities as shown in the figure below:

3.1 Alibaba Cloud AI Search Open Platform Product Overview

AI Search Open Platform, as a flagship product of Alibaba Cloud search team, is a precipitated fruit of more than 20 years of experience, which includes almost all services that can be used in all aspects of the current Alibaba Cloud AI search field, aiming to provide users with powerful search capabilities.

For customers who are familiar with development, using the platform API calls can achieve higher flexibility and speed. And for customers who want to deliver services quickly, Alibaba Cloud provides scenario-based products based on feedback. These products are categorized and developed for different business scenarios, and there is a corresponding product version for each scenario. Users only need to write data into the system to directly access the relevant results, and there is almost no need to write code.

3.2 Introduction to Alibaba Cloud ES version 8.15 features and application scenarios

The latest version 8.15 of Alibaba Cloud ES is a vector-enhanced version based on the latest kernel. This version improves performance by more than five times compared to version 8.9, supports data quantization, and can significantly reduce memory storage costs by 75%. In addition, this version natively supports vector search without the need for plug-ins, and supports the multi-way fusion sorting algorithm, which improves the fusion effect of vector search and traditional search results. This version also supports seamless integration with the AI search open platform.

In terms of search scenarios (RAG), Alibaba Cloud's ES can be widely used in the following scenarios:

Intelligent Customer Service: Through natural language processing, intelligent customer service can quickly respond to customer queries and provide accurate information. For example, a customer can ask about refund policy or express delivery status, and the system can instantly give relevant answers to improve customer satisfaction.

Big Data Retriveal: Enterprises can use AI search technology to build an internal knowledge base to help employees quickly find the information they need. This approach not only improves work efficiency, but also promotes knowledge sharing.

Vector Search: in e-commerce platforms, users can get personalized product recommendations through AI search. The system is able to provide accurate product suggestions based on the user's historical behavior and preferences, increasing the conversion rate.

Smart Q&A: In finance and other industries, AI search technology can handle complex queries, users can ask questions such as “how to apply for desktop computer”, the system will be based on historical data in the form of a table to output the relevant results. This ability allows users to compare and analyze data more intuitively, supporting more accurate investment decisions.

4）Alibaba Cloud AI Search Program

4.1 A full-link solution to build a RAG system based on Alibaba Cloud Elastisearch

This scheme shows how to build a full-link RAG (Retrieval-Augmented Generation) system by utilizing the modeling services provided by Alibaba Cloud Elasticsearch and AI search open platform.

First, customer data sources (e.g., PDF, Word documents) are imported into the system, and the information is recognized and extracted by document parsing services, followed by slicing and vectorization of the documents, and ultimately these vector data are stored in Alibaba Cloud ES for index construction.

Offline data writing mainly involves customers importing documents (e.g., word, PPT, PDF) into the AI search open platform and recognizing and extracting them by calling the document parsing service. For long documents, the system will be cut, such as the use of subheadings or subheadings and other structures for semantic or document structure cut. Then, the sliced text is transformed into dense vectors or sparse vectors and stored in Alibaba Cloud ES to construct indexes, completing the construction of original text indexes and dense vector and sparse data indexes.

When querying online, the user inputs a question, the internal service understands and expands the customer's question, then the query understanding service is introduced to determine the intent of the question and generate multiple possible question variants. Then, the problem is transformed into vector data, indexes are constructed, and multiplexed recall is performed to return Top N knowledge fragments. Subsequently, after rearranging the model, it is integrated into the larger model in a Prompt engineering manner for aggregation processing to form the final answer to the customer.

In addition, with version 8.15 of Alibaba Cloud ES, AI semantic search models can be created using the Influence API. The demonstration includes steps such as selecting a generic commercial version, configuring visualization control, setting up access whitelisting, selecting a model service, obtaining configuration information, and creating and debugging the model in ES. This process demonstrates how to seamlessly integrate Alibaba Cloud AI model services, reduce the threshold of model usage, and realize functions such as semantic search.

4.2 Effectiveness Evaluation and Optimization

To ensure the effectiveness of the AI search solution, Alibaba Cloud also provides a full-link effect evaluation service. The platform can help customers evaluate the relevance, illusion rate and credibility of answer results in a three-dimensional way. This evaluation mechanism not only supports the effect test of a single service, but also supports the synchronized evaluation of multiple services, which improves the decision-making efficiency of developers.

Through these evaluations, enterprises are able to identify problems and optimize in time to ensure the stability and accuracy of AI search services, thus improving user satisfaction.

Closing

Alibaba Cloud AI search solution version 8.15 adds the AI service center column and model management function on the original basis, supports one-click creation of AI open platform space and API, and maintains support for all the functions of version 8.13. 8.15 kernel also emphasizes its vector capability and AI search capability, and also supports basic application scenarios such as logging, which is suitable for customers with the need to reduce costs.

In terms of billing, it flexibly supports pay-per-volume or annual and monthly packages, and the AI platform is billed on a per-call basis, offering the first 100 free calls.

In addition, Alibaba Cloud has launched major promotional activities to help enterprises develop.

Alibaba Cloud Elacticsearch version 8.15 Full specification 50% off on new purchases of annual payment pass-through!

Alibaba Cloud Elacticsearch non-8.15 version, 2-core/4-core full specification new purchase annual payment pass through 50% off!

From Enterprise RAG to AI Assistant—Alibaba Cloud Elasticsearch Technology Practice in AI Search

A_Lucas — Wed, 22 Jan 2025 08:28:33 +0000

01 Challenges in landing AI search

Over the past year, rapid iteration of base large language model technology has driven the evolution of AI search in the following ways:

Refactoring of the search technology chain

The comprehensive reconstruction based on the large language model is reshaping the technology chain of AI search. Every link from data collection, document parsing, and vector retrieval to query analysis, intent recognition, ranking models, and knowledge graphs is undergoing profound changes. New interaction methods such as conversational search, answer summarization, intelligent customer service, enterprise digital employees and virtual people are gradually becoming mainstream, which not only improves the user experience, but also provides the possibility for more application scenarios.

AI search as infrastructure

AI search has become one of the basic technologies for various AI applications. As a popular AI native application, it not only drives the development of knowledge-based AI applications, but also gradually becomes a built-in capability of major basic models. For example, techniques such as vector retrieval, retrieval-enhanced generation (RAG), and semantic search have been widely used in several domains. This integration trend enhances the adaptability and flexibility of AI search in different scenarios.

Bottlenecks Facing Effectiveness Improvement

Although AI search has made significant progress in terms of effectiveness, the illusion problem is still a major constraint to its wide application, especially in business scenarios that require high knowledge accuracy. In addition, high cost and low privacy security controllability are also important challenges in the implementation process.

To cope with these issues, Alibaba cloud Elasticsearch has introduced an innovative AI search solution that uses RAG technology to enhance the capabilities of all aspects of retrieval enhancement generation, and deeply integrates the enterprise version of the AI Assistant, applying RAG technology to the field of AIOps.

02 5X Improvement in Elasticsearch Vector Performance

The Elasticsearch vector engine continues to be optimized, with particular focus on performance and cost improvements. Initially, due to a common perception bias - that ES vector engines are powerful but may have performance shortcomings, especially for applications in the Java ecosystem - this view is gradually being overturned by its technical evolution. From the initial 8.0 release to the current 8.15 release, Elasticsearch has iterated and made significant progress in performance optimization, including but not limited to the effective integration of hardware acceleration technologies.

Elasticsearch utilizes hardware acceleration technology to achieve significant performance leaps in the field of vector retrieval, especially when dealing with complex similarity computation tasks. This technological innovation is not limited to the theoretical level; it has been proven that the deep integration of hardware gas pedals has increased the efficiency of some computationally intensive operations by several times or more. For example, benchmark data from September 2022 to the present can be visualized that the query response time has been drastically reduced from the initial 100ms to about 20ms now, highlighting the huge performance improvement brought by the iterative upgrading of Elasticsearch's vector retrieval.

Elasticsearch's memory optimization is also noteworthy. Through vector quantization technology, the memory required is only a quarter of the original demand, which greatly improves resource utilization. In the latest version, BBQ (Better Binary Quantization) brings a quantization leap to Elasticsearch, reducing float32 dimensions to bits, which reduces memory by about 95% while maintaining high ranking quality. BBQ outperforms traditional methods such as Product Quantization (PQ) in terms of indexing speed (20-30x reduction in quantization time) and query speed (2-5x improvement in query speed), with no additional loss of accuracy.

03 Comprehensive Explanation of Elasticsearch for the Enterprise AI Capabilities

Semantic expansion and sparse vector representation:

Elasticsearch utilizes techniques such as sparse coding, which not only indexes the original vocabulary, but also effectively extends it to concepts or words related to it, with model-calculated weights attached to each extension, enhancing the depth and breadth of semantic understanding. This is due to the sparse vector technology, which stores information efficiently with a low memory footprint, compared to dense vectors that require full-memory indexing, which significantly improves resource efficiency.

Query efficiency and resource optimization:

The query process benefits from the inverted index structure, which avoids the overhead of vector similarity matching and accelerates the retrieval speed. In addition, Elasticsearch's sparse vectors reduce memory requirements, further optimizing resource utilization.

Hybrid Search Strategy:

Modern search requirements have motivated Elasticsearch to support multimodal queries, combining text, vector search, and rrf hybrid sorting methods to enhance the relevance and coverage of results. This hybrid search strategy recalls more diverse data and enhances the user experience.

Ranking and Relevance Adjustment:

In order to accurately select the most relevant results from the large amount of data recalled, ES employs a ranking mechanism such as BM25 to initially determine the weights considering factors such as document frequency and location. Subsequently, a secondary ranking of the initially filtered documents is performed by integrated learning or a more refined model (e.g., Rerank stage) to ensure that the top results are highly relevant.

Model integration and native support:

Elasticsearch demonstrates powerful model integration capabilities, allowing users to load custom models directly into the cluster and run them, enabling end-to-end processing from input to output (e.g., word embedding generation) without external preprocessing steps. This not only simplifies the workflow, but also facilitates the seamless integration of machine learning models with the search engine, reinforcing the system's level of intelligence and adaptability.

04 Alibaba Cloud Elasticsearch accuracy has been increased to 95 percent

Relying on the powerful Elasticsearch foundation and based on the Alibaba Cloud AI search open platform, the Alibaba cloud Elasticsearch AI search product integrates diversified models and hybrid retrieval technologies to realize the leap from traditional search to AI semantic search. The solution forms a complete and efficient application framework for RAG scenarios through fine data preprocessing, intelligent vectorization, multi-dimensional search recall, and large model-assisted generation.

Document parsing and slicing: Use self-developed models to identify unstructured data, extract key information, and ensure content integrity and semantic coherence.

Efficient vectorization: Adopting parameter-optimized vector models to reduce costs while ensuring effectiveness and achieve efficient execution of the vectorization process.

RRF Hybrid Retrieval Strategy: Combining text, sparse and dense vector indexes to realize multi-way recall and significantly improve retrieval precision and efficiency.

Intent Understanding and Rearrangement Optimization: Understand user intent through query analysis model, and work with the rearrangement model to fine-sort the results to ensure the relevance of the content.

Comprehensive Evaluation and Flexible Configuration: AI search open platform provides one-stop service, including multiple model components, compatible with open source ecology, helping enterprises quickly build customized search systems.

Through the comprehensive application of Alibaba cloud Elasticsearch AI search, the customer witnessed remarkable results in the knowledge base Q&A scenario, with the accuracy rate increasing from 48% initially to over 95% eventually. In addition, the combination of three-way hybrid search and re-ranking model further enhances the accuracy of search and guarantees the excellence of search experience.

05 AI Assistant Integrates Qwen LLM to Enable AIOps

Elasticsearch Enterprise Edition's AI Assistant incorporates RAG technology and Alibaba cloud's large language model services to provide an AIOps assistant for enterprises. This innovative tool shows excellent application potential in multiple areas such as universal search, observability analysis and security, and not only helps developers make significant progress in anomaly monitoring, alert handling, problem identification and diagnosis, data analysis and modeling, and query performance optimization, but also greatly improves work efficiency through a more intuitive and easy-to-use interactive interface.

Especially in terms of observability, the AI Assistant can efficiently request, analyze, and visualize your data with the help of an automated function call mechanism, transforming it into information with practical operational value. In addition, a knowledge base based on Elastic Learned Sparse EncodeR (ELSER) support further enriches contextual information and suggestions from private datasets; while RAG technology combined with generalized large models ensures more accurate data understanding and expressiveness.

After integrating the Qwen Model on the Alibaba Cloud AI Search Open Platform, Elasticsearch's AI Assistant pays special attention to activation function simulation calls to ensure seamless compatibility between different systems. This allows users to flexibly switch between multiple connectors according to specific needs, thus realizing efficient information retrieval and processing processes. Especially in microservice operation and maintenance scenarios, AI Assistant plays a crucial role - it not only monitors abnormal conditions and potential failure points in real time, but also analyzes detailed error logs and quickly locates the root cause of the problem in conjunction with existing operation and maintenance manuals. At the same time, AI Assistant can also effectively integrate all kinds of alarm information, comprehensively analyze the chain of security attacks, and then propose practical defense strategies, significantly improving the speed and quality of problem solving.

By calling the API interface and automatically generating ESQL query statements, AI Assistant is able to perform complex data analysis tasks and generate intuitive and easy-to-understand statistical charts, even for users who do not know much about Elasticsearch query syntax. Whether it's exploring relationships between fields or interpreting data trends and other data insights, AI Assistant meets the diverse needs of users with high efficiency and easy operation.

Free Trial：AlibabaCloud Elasticsearch official website

Learn More：AlibabaCloud Technology Solutions

Vector search using Alibaba Cloud inference API and semantic text

A_Lucas — Mon, 20 Jan 2025 08:09:29 +0000

In the previous post "Elasticsearch Open Inference API Adds Support for Alibaba Cloud AI Search"，It describes in detail how to use the Elastic inference API to display Ali's dense vector model, sparse vector model, re-ranking and completion. In that article, it uses a lot of English examples. However, Ali's model is more suitable for displaying in Chinese. We know that Elastic's out-of-the-box sparse vector model, ELSER, is only suitable for English. It doesn't support Chinese at the moment. Ali's sparse vector model fills this gap. Sparse vectors can be used out of the box. For many developers who are not well versed in AI, this is definitely a boon, and it uses very little resources.

For more reading, see Elasticsearch's "AliCloud AI Search Inference Service"

Sparse Vector

According to the documentation, we use the following command to create an inference API endpoint for sparse vectors:

PUT _inference/sparse_embedding/alibabacloud_ai_search_sparse
{
    "service": "alibabacloud-ai-search",
    "service_settings": {
        "api_key": "<api_key>",
        "service_id": "ops-text-sparse-embedding-001",
        "host": "default-j01.platform-cn-shanghai.opensearch.aliyuncs.com",
        "workspace": "default"
    }
}

On top of that, we need to get the api_key from AliCloud to do so. Run the above command:

Next, We can test it by doing the following (the following information is obtained from the address)

POST _inference/alibabacloud_ai_search_sparse
{
    “input": ”Alibaba (China) Limited was founded on March 26, 2007, legal representative Jiang Fang”
}

The above command shows the result as:

We can see from the above displayed result that it is different from our previous Elastic ELSER output. Here it should be unicode.

The command above is the same as the command below:

POST _inference/sparse_embedding/alibabacloud_ai_search_sparse
{
  "input": "Alibaba (China) Limited was founded on March 26, 2007, legal representative Jiang Fang"
}

Because each endpoint is automatically detected and recognized as what type of model it is when it is created by Elasticsearch, the sparse_embedding in the path above can be omitted.

Dense Vectors

Similarly, we use the following command to create an inference API endpoint for dense vectors:

PUT _inference/text_embedding/alibabacloud_ai_search_embeddings
{
    "service": "alibabacloud-ai-search",
    "service_settings": {
        "api_key": "<api_key>",
        "service_id": "ops-text-embedding-001",
        "host": "default-j01.platform-cn-shanghai.opensearch.aliyuncs.com",
        "workspace": "default"
    }
}

Running the above command, we can see that:

We can use the following command to generate dense vectors:

POST _inference/alibabacloud_ai_search_embeddings
{
  "input": "Alibaba (China) Limited was founded on March 26, 2007, legal representative Jiang Fang"
}

A dense vector is an array of floating point numbers. We can still actually scalar quantize it when we generate it, which saves memory consumption and improves the speed of the search.

Completion

We can even apply a big model to the search results to get a result of "completion", e.g., we use the following command to generate an inference API endpoint for completion:

PUT _inference/completion/alibabacloud_ai_search_completion
{
    "service": "alibabacloud-ai-search",
    "service_settings": {
        "host" : "default-j01.platform-cn-shanghai.opensearch.aliyuncs.com",
        "api_key": "{{API_KEY}}",
        "service_id": "ops-qwen-turbo",
        "workspace" : "default"
    }
}

We can demonstrate completion with the following example：

POST _inference/completion/alibabacloud_ai_search_completion
{
  “input": ”When was Alibaba (China) Limited founded?”
}

This result is generated in conjunction with the big model. The big model provides a result based on the results obtained during the training of the big model.

Rerank

Similarly, we can generate an inference API endpoint for rerank by following the commands below:

PUT _inference/rerank/alibabacloud_ai_search_rerank
{
    "service": "alibabacloud-ai-search",
    "service_settings": {
        "api_key": "<api_key>",
        "service_id": "ops-bge-reranker-larger",
        "host": "default-j01.platform-cn-shanghai.opensearch.aliyuncs.com",
        "workspace": "default"
    }
}

In the "input" above, we have listed two documents. We can reorder these two documents using the "rerank" endpoint. Let's say the text we want to search for is "When was Alibaba (China) Limited founded?". Then "rerank" returns the result:

From the above, we can see that the second document is more relevant than the first, that is, it is more relevant.

RAG Applications

In many cases, our organization has a lot of business data or private data being generated every now and then, and the knowledge of the big model is limited to when it is generated, so a lot of times, there is a lot of knowledge that the big model doesn't know. If we don't limit the answers of the big model, then it may give wrong answers and thus create illusions. In practice, we usually keep business or private data in a vector database like Elasticsearch. When searching, we first search Elasticsearch and send the results of the search to the big model as part of the prompt, which solves the hallucination problem.

First, let's create a sparse vector index:

PUT alibaba_sparse
{
  "mappings": {
    "properties": {
      "inference_field": {
        "type": "semantic_text",
        "inference_id": "alibabacloud_ai_search_sparse"
      }
    }
  }
}

Then, we write the following two documents:

PUT alibaba_sparse/_bulk
{“index”: {“_id”: “1”}}
{“inference_field”: “Alibaba (China) Limited was founded on March 26, 2007, legal representative Jiang Fang”}}
{“index”: {“_id”: “2”}}
{“inference_field”: “Founded in 2009 and headquartered in Hangzhou, Alibaba Cloud Computing Co. Ltd (also known as Alibaba Cloud Computing Co. Ltd) is the digital technology and intelligence backbone of the Alibaba Group, providing a full range of cloud services to customers around the globe, including its own servers, elastic computing, storage, network security, database and big data. It provides a full range of cloud services to global customers, including its own servers, elastic computing, storage, network security, database and big data services."}

We perform a search with the following command:

GET alibaba_sparse/_search
{
  "query": {
    "semantic": {
      "field": "inference_field",
      "query": "when was Aliyun founded？"
    }
  }
}

The search result above is:

Alibaba Cloud is in the first place

"Founded in 2009 and headquartered in Hangzhou, Alibaba Cloud Computing Co. Ltd (also known as Alibaba Cloud Computing Co. Ltd) is the digital technology and intelligence backbone of the Alibaba Group, providing a full range of cloud services to customers around the globe, including its own servers, elastic computing, storage, network security, database and big data. It provides a full range of cloud services to global customers, including its own servers, elastic computing, storage, network security, database and big data services."

Alibaba is in second place

"Alibaba (China) Limited was founded on March 26, 2007, legal representative Jiang Fang"

The document containing Alibaba cloud is in the first position. This is what we call vector search.

In practice, in many cases, we sometimes want to get a unique answer, or even this answer is reasoned out, not the original document. This time we need to use the big model, that is, use the completion of the reasoning endpoint.

Let's start by doing the following search:

GET alibaba_sparse/_search
{
  "query": {
    "semantic": {
      "field": "inference_field",
      "query": "Who is the legal person of Alibaba？"
    }
  }
}

This time only one document is listed. We use the "completion" endpoint below:

POST _inference/completion/alibabacloud_ai_search_completion
{
  "input": "<|system|>
    You are a knowledgeable person.</s>
    <|user|>
    CONTEXT:
    Alibaba (China) Limited was founded on March 26, 2007, legal representative Jiang Fang

    QUESTION: 
    Who is the legal person of Alibaba？</s>
    <|assistant|>"
}

In the above, we document the results of the search above "Alibaba (China) Limited was founded on March 26, 2007, legal representative Jiang Fang" as a part of the document，and submit the search question along with it. We use the completion endpoint API to view the results:

The result will be "Alibaba's legal representative is Jiang Fang"。Obviously this is quite different from the previous approach of using "completion" without "context":

POST _inference/completion/alibabacloud_ai_search_completion
{
  "input": "Who is the legal person of Alibaba?"
}

It can be understood in this way that with the results from the Elasticsearch vector database (real-time business data or private data) searched and made available to the big model, the big model can get search results that are more closely aligned to the answer based on these contexts. This is used in practice to avoid illusions!

How to build a prompt

Following the method of constructing a prompt, let's create an index of a dense vector:

PUT alibaba_dense
{
  "mappings": {
    "properties": {
      "inference_field": {
        "type": "semantic_text",
        "inference_id": "alibabacloud_ai_search_embeddings"
      }
    }
  }
}

We follow the same method for writing documents:

PUT alibaba_dense/_bulk
{“index”: {“_id”: “1”}}
{“inference_field”: “Alibaba (China) Limited was founded on March 26, 2007, legal representative Jiang Fang”}
{“index”: {“_id”: “2”}}
{“inference_field”: “Aliyun (aka: Aliyun Computing Co., Ltd.) was founded in 2009, headquartered in Hangzhou, is the digital technology and intelligence backbone of Alibaba Group, providing a full range of cloud services to customers around the world.”}}

Let's search this dense vector index alibaba_dense:

GET alibaba_dense/_search
{
  "query": {
    "semantic": {
      "field": "inference_field",
      "query": "Who is the legal person of Alibaba?"
    }
  }
}
GET alibaba_dense/_search
{
  "query": {
    "semantic": {
      "field": "inference_field",
      "query": "What kind of company is Aliyun?"
    }
  }
}

The results of the search above are:

Obviously, documents containing "Aliyun" are ranked first although we searched for "Alibaba".

Automatic chunking

We know that the semantic_text field can help us automatically chunk a large piece of text without us having to manually chunk it. This is one of the great benefits of semantic_text. Let's create an index called alibaba_dense_large_text:

PUT alibaba_dense_large_text
{
  "mappings": {
    "properties": {
      "inference_field": {
        "type": "semantic_text",
        "inference_id": "alibabacloud_ai_search_embeddings"
      }
    }
  }
}

We also create a large document for this index:

PUT alibaba_dense_large_text/_doc/1
{
    “inference_field": ”
    About Alibaba Group
Alibaba Group (Alibaba Group) is a globally recognized technology company headquartered in Hangzhou, Zhejiang Province, China, founded in 1999 by Jack Ma and his team. As a technology-driven company, Alibaba has a wide range of businesses in e-commerce, cloud computing, financial technology, digital media, logistics and other fields, and has become a key player in the global Internet industry.
Background and Early Development
The birth of Alibaba began in 1999, when the Internet was just emerging in China. With the vision of empowering small and medium-sized enterprises (SMEs) through the Internet, Jack Ma and his team launched Alibaba.com, a global wholesale marketplace. Initially, the platform provided SMEs with the opportunity to showcase their products and find international buyers, helping Chinese manufacturers reach out to the global market. After several years of development, Alibaba quickly emerged as the world's leading B2B e-commerce platform.
In 2003, Alibaba launched Taobao, a C2C platform for individual consumers, which quickly attracted a large number of users. With its “free store” model, Taobao competed with other e-commerce platforms in a short period of time and achieved success. To enhance the user experience, Alibaba launched Alipay, a third-party payment platform, in 2004, solving the trust problem of online payment, an innovation that laid the foundation for the rapid development of e-commerce in China.
Business Ecology and Strategic Layout
Alibaba has gradually built a diversified ecosystem covering e-commerce, finance, logistics and cloud computing through continuous business expansion.
E-commerce
As Alibaba's core business, e-commerce includes Taobao, Tmall and AliExpress. Among them, Taobao focuses on meeting the needs of individual consumers, while Tmall caters to brands and enterprises, providing high-quality goods and services. Global Express is dedicated to connecting overseas buyers with Chinese sellers, and is gaining influence in the international market.
Cloud Computing and Big Data
Alibaba Cloud, the core technology division of Alibaba Group, was founded in 2009 to provide cloud computing services, artificial intelligence solutions and big data technical support. Today, Alibaba Cloud has become the world's leading cloud service provider, supporting the digital transformation of enterprises.
Financial Technology
Ant Group, the parent company of Alipay, is an important part of Alibaba's fintech layout. Through Alipay, Alibaba provides users with online payment, wealth management, loans and other services, promoting the development of a cashless society in China.
Logistics and Supply Chain
Founded in 2013, Cainiao Network (Cainiao) is committed to optimizing logistics efficiency and building an intelligent logistics network with global coverage. Cainiao's technology and platform capabilities enhance the speed and reliability of e-commerce logistics services.
Digital Media and Entertainment
Alibaba has also been actively laying out in the digital media sector, including Youku, AliFilm and Shrimp Music, to promote the construction of a content ecosystem. In addition, the Group has further enriched its brand by investing in cultural and creative industries and expanding its sports business.
Internationalization and Innovation
In recent years, Alibaba has been accelerating its pace of internationalization, aiming to become a bridge between global consumers and businesses. The Group has expanded its business in markets such as Southeast Asia, Europe and North America through mergers and acquisitions and partnerships. For example, through the acquisition of Lazada, Alibaba has enhanced its e-commerce presence in Southeast Asia.
In addition, Alibaba has been actively exploring new technologies, including artificial intelligence, blockchain and the Internet of Things. The application of these technologies not only optimizes the Group's existing business, but also lays the foundation for future innovations.
Corporate Culture and Social Responsibility
Alibaba's corporate culture is based on the core mission of “making the world a business without difficulty”, and encourages employees to innovate and collaborate. Jack Ma emphasizes corporate social responsibility and advocates the use of resources for public welfare. For example, the Alibaba Foundation supports environmental protection, education development and poverty alleviation, creating long-term value for society.
Challenges and Future Prospects
Despite its remarkable achievements, Alibaba faces challenges such as intensified international competition, increased industry regulation and pressure for technological innovation. In order to maintain its leading position, Alibaba needs to continue to innovate globally while actively adapting to the regulations and cultures of different markets.
Looking ahead, Alibaba will continue to promote the integration of technology and commerce to help SMEs and consumers realize more value. At the same time, through technology development and globalization, Alibaba is expected to play a more important role in the development of the global digital economy.
Conclusion
As one of the most influential Internet companies in China and the world, Alibaba's growth history has demonstrated the profound transformation of technology on business models. Through continuous innovation and expansion, Alibaba is injecting new vitality into the global economy and creating unprecedented opportunities for countless businesses and consumers. In the future, Alibaba will continue to be driven by technology and commerce, and strive to build a more convenient and efficient digital world.
    ”
}

The above article is about 4500 words long.

We can view the written document in the following way:

GET alibaba_dense_large_text/_search

From above, we can see that the semantic_text field automatically chunks a very long textual document and vectorizes them separately.

We hope this article will provide you with valuable reference and inspiration in your exploration of vector search using AliCloud and Elasticsearch. In the future, as the technology continues to evolve, we expect to see more intelligent applications based on semantic understanding to land, and push the search technology to a higher level.

Harness the power of Alibaba Cloud Elasticsearch for your vector search needs. Its robust infrastructure and AI capabilities make it a top choice for businesses seeking to leverage AI for improved search outcomes.

Ready to start your journey with Elasticsearch on Alibaba Cloud? Explore our tailored Cloud solutions and services to take the first step towards transforming your data into a visual masterpiece. Click here, Embark on Your 30-Day Free Trial.

Mastering Elasticsearch with Alibaba Cloud: How to Create an Instance

A_Lucas — Mon, 29 Jul 2024 01:59:44 +0000

Introduction

Alibaba Cloud Elasticsearch offers a powerful and flexible solution for managing and analyzing vast amounts of data in real-time. This tutorial will guide you through the process of creating an Elasticsearch instance, from understanding the prerequisites to executing API requests.

Prerequisites

Before delving into the technical details of creating an Elasticsearch instance, ensure that:

You are well-versed with the billing and pricing standards of Alibaba Cloud Elasticsearch. For more information, see Pricing of Alibaba Cloud Elasticsearch.
You have completed real-name verification.
You understand that you do not need to specify a zone when creating a cluster; by default, it is deployed in the same zone as the specified VPC.

Request Headers

The createInstance operation does not require operation-specific headers but uses common request headers. For more information, see Common Request Headers.

Request Syntax

The base syntax for creating an instance is as follows:

POST /openapi/instances HTTP/1.1

Request Parameters

Below are some essential parameters you'll need for the request:

Parameter	Type	Required	Description
clientToken	String	No	Unique token to ensure idempotence of the request.
nodeAmount	Integer	Yes	The number of data nodes. Valid values: 2 to 50.
instanceCategory	String	No	Type of instance, e.g., advanced.
esAdminPassword	String	Yes	Password for accessing the cluster, must be 8 to 32 characters long and include at least three types of characters.
esVersion	String	Yes	Version of the Elasticsearch instance.
networkConfig	NetworkConfig	Yes	The network configurations.
paymentType	String	No	Billing method, either `postpaid` or `prepaid`.
description	String	No	The name or description of the instance.
nodeSpec	NodeSpec	No	Configurations of data nodes. Example: `{"spec":"elasticsearch.sn2ne.xlarge", "disk":50, "diskType":"cloud_ssd"}`.
warmNodeConfig	WarmNodeConfig	No	Configurations of warm nodes.
kibanaConfig	KibanaNodeConfig	No	Configurations of Kibana nodes' settings.

For a detailed list of parameters, refer to the Full API Documentation.

Example Request Body

Creating a General-purpose Business Edition Instance with Both Hot and Cold Nodes

{
  "description": "xpack-hot-cold",
  "nodeAmount": 2,
  "esVersion": "7.10_with_X-Pack",
  "instanceCategory": "x-pack",
  "nodeSpec": {
    "spec": "elasticsearch.sn2ne.xlarge",
    "disk": 20,
    "diskType": "cloud_ssd"
  },
  "warmNodeConfiguration": {
    "spec": "elasticsearch.sn1ne.xlarge",
    "amount": 3,
    "diskType": "cloud_efficiency",
    "disk": 500
  },
  "networkConfig": {
    "type": "vpc",
    "vpcId": "vpc-8vbfu74bmw4m7m84a****",
    "vswitchId": "vsw-8vbnk6cmurbpump2f****",
    "vsArea": "cn-zhangjiakou-c"
  },
  "paymentType": "postpaid",
  "esAdminPassword": "yourPassword",
  "kibanaConfiguration": {
    "spec": "elasticsearch.sn1ne.large"
  }
}

Creating an Indexing Service Instance with Only Hot Nodes

{
  "description": "is-hot",
  "nodeAmount": 2,
  "esVersion": "7.10_with_X-Pack",
  "instanceCategory": "IS",
  "nodeSpec": {
    "spec": "elasticsearch.sn1ne.xlarge",
    "disk": 20,
    "diskType": "cloud_ssd"
  },
  "networkConfig": {
    "type": "vpc",
    "vpcId": "vpc-8vbfu74bmw4m7m84a****",
    "vswitchId": "vsw-8vbnk6cmurbpump2f****",
    "vsArea": "cn-zhangjiakou-c"
  },
  "paymentType": "postpaid",
  "esAdminPassword": "yourPassword",
  "kibanaConfiguration": {
    "spec": "elasticsearch.sn1ne.large"
  }
}

Response Parameters

Upon successful execution, the response will include:

RequestId: A unique ID for the request.
Result.instanceId: The ID of the created Elasticsearch instance.

Sample Success Responses

XML Format

HTTP/1.1 200 OK
Content-Type:application/xml
<createInstanceResponse>
    <RequestId>838D9D11-8EEF-46D8-BF0D-BC8FC2B0C2F3</RequestId>

    <Result>
        <instanceId>es-is-0u2ecp69tt****</instanceId>

    </Result>

</createInstanceResponse>

JSON Format

HTTP/1.1 200 OK
Content-Type:application/json
{
  "RequestId": "838D9D11-8EEF-46D8-BF0D-BC8FC2B0C2F3",
  "Result": {
    "instanceId": "es-is-0u2ecp69tt****"
  }
}

Conclusion

Creating an Elasticsearch instance on Alibaba Cloud is a streamlined process, ensuring that you can efficiently manage and scale your data analytics capabilities. By following the guidelines and examples provided in this article, you should be well-equipped to get started.
Ready to start your journey with Elasticsearch on Alibaba Cloud? Explore our tailored Cloud solutions and services to take the first step towards transforming your data into a visual masterpiece.
Click here to embark on Your 30-Day Free Trial

Integrate AMAP into Alibaba Cloud Kibana: A Complete Guide

A_Lucas — Wed, 24 Jul 2024 02:12:44 +0000

Background Information

By default, Elastic Maps uses data from OpenStreetMap. However, Alibaba Cloud Elasticsearch Kibana does not display this data. Fortunately, you're able to integrate AMAP into Kibana, allowing you to leverage AMAP data directly on the Maps page of the Kibana console.

Prerequisites

1）You need to create an Alibaba Cloud Elasticsearch cluster. For this example, we'll use an Alibaba Cloud Elasticsearch V8.9 cluster.
2）Ensure Cross-Origin Resource Sharing (CORS) is configured for your tile server to grant tile requests from the Kibana origin access to your tile server origin.

Step 1: Configure an AMAP Tile URL

Log into the Kibana console of your Elasticsearch cluster via the Access over Internet entry. See Log on to the Kibana console for more information.
Note: Ensure your browser can access the Internet when logging into the Kibana console.

Steps:

1）After logging into the Kibana console, click on the menu icon in the upper-left corner.

2）Navigate to Analytics > Maps.

3）(Optional) Click Create map in the upper-right corner of the Maps page.

4）Click Add layer in the upper-right corner of the Maps page.

5）In the Add layer panel, select Tile Map Service.

6）Enter a tile URL of AMAP in the Url field. Use HTTPS for the URL. Here are some common AMAP tile URL formats:

{1,4} indicates that you can select a number from 1 to 4.
{x}, {y}, and {z} separately indicate the related layers.
lang=zh_cn indicates that the language of annotations is Chinese. lang=en indicates that the language of annotations is English.

7）In the lower part of the Add layer panel, click Add and close.

Step 2: Upload a Geographic Information Dataset

1）Prepare a geographic information dataset. You can use a provincial boundary dataset which can be obtained from DataV Atlas.

2）Click Add layer in the upper-right corner of the Maps page.

3）Click Upload file in the Add layer panel.

4）Select or drag and drop your dataset file. Supported file formats include .json, .geojson, and .shp.

5）Click Import file and then Add as document layer.

6）Enter a name in the Name field and click Keep changes.

7）Click Save in the upper-right corner.

With these steps, AMAP data will be integrated into your Alibaba Cloud Kibana Map!

Conclusion

Integrating AMAP into Alibaba Cloud Kibana can significantly enhance your data visualizations, providing rich geographic insights directly within your Elasticsearch environment.
Alibaba Cloud Elasticsearch offers a robust platform that simplifies these complex integrations. Ready to start your journey with Elasticsearch on Alibaba Cloud? Explore our tailored Cloud solutions and services to take the first step towards transforming your data into a visual masterpiece.
Please Click here, Embark on Your 30-Day Free Trial

Elevate Your Elasticsearch Experience with Java High Level REST Client (7.x)

A_Lucas — Wed, 17 Jul 2024 01:45:55 +0000

Introduction

Java High Level REST Client (7.x) is a powerful tool for interacting with Elasticsearch clusters, making server communication more accessible and efficient. In this guide, we will walk you through the steps to use the High Level REST Client to call Elasticsearch Java APIs on an Alibaba Cloud Elasticsearch cluster.

Preparations

Step 1: Create an Elasticsearch Cluster

Ensure your cluster version is the same as or newer than the Java High Level REST Client version you plan to use. For step-by-step instructions, see Create an Alibaba Cloud Elasticsearch cluster.

Step 2: Enable Auto Indexing

Enable the Auto Indexing feature in the YAML configuration file. For details, see Configure the YML file.

Step 3: Configure IP Address Whitelist

Ensure proper communication by configuring an IP address whitelist. If you're accessing the cluster over the Internet, allow requests from the required IP addresses by following the guidelines in Configure a public or private IP address whitelist.

Step 4: Install JDK

Install Java Development Kit (JDK) version 1.8 or later. For more information, see Install a JDK.

Step 5: Create a Java Maven Project

Add the necessary dependencies to your pom.xml file. Change the version number in the dependencies from 7.x to the specific version of the High Level REST Client you are using.

<dependency>
    <groupId>org.elasticsearch.client</groupId>
    <artifactId>elasticsearch-rest-high-level-client</artifactId>
    <version>7.x</version>
</dependency>
<dependency>
    <groupId>org.apache.logging.log4j</groupId>
    <artifactId>log4j-core</artifactId>
    <version>2.20.0</version>
</dependency>
<dependency>
    <groupId>org.apache.logging.log4j</groupId>
    <artifactId>log4j-api</artifactId>
    <version>2.20.0</version>
</dependency>

Example: Managing an Index

Below is an example of creating and deleting an index using the High Level REST Client. Replace placeholders {} with your specific parameters.

import org.apache.http.HttpHost;
import org.apache.http.auth.AuthScope;
import org.apache.http.auth.UsernamePasswordCredentials;
import org.apache.http.client.CredentialsProvider;
import org.apache.http.impl.client.BasicCredentialsProvider;
import org.apache.http.impl.nio.client.HttpAsyncClientBuilder;
import org.elasticsearch.action.delete.DeleteRequest;
import org.elasticsearch.action.delete.DeleteResponse;
import org.elasticsearch.action.index.IndexRequest;
import org.elasticsearch.action.index.IndexResponse;
import org.elasticsearch.client.*;

import java.io.IOException;
import java.util.HashMap;
import java.util.Map;

public class RestClientExample {
    private static final RequestOptions COMMON_OPTIONS;

    static {
        RequestOptions.Builder builder = RequestOptions.DEFAULT.toBuilder();
        builder.setHttpAsyncResponseConsumerFactory(
                new HttpAsyncResponseConsumerFactory
                        .HeapBufferedResponseConsumerFactory(30 * 1024 * 1024));
        COMMON_OPTIONS = builder.build();
    }

    public static void main(String[] args) {
        final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
        credentialsProvider.setCredentials(AuthScope.ANY, 
            new UsernamePasswordCredentials("{Username}", "{Password}"));

        RestClientBuilder builder = RestClient.builder(new HttpHost("{Endpoint of the Elasticsearch cluster}", 9200, "http"))
                .setHttpClientConfigCallback(new RestClientBuilder.HttpClientConfigCallback() {
                    @Override
                    public HttpAsyncClientBuilder customizeHttpClient(HttpAsyncClientBuilder httpClientBuilder) {
                        return httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider);
                    }
                });

        RestHighLevelClient highClient = new RestHighLevelClient(builder);

        try {
            Map<String, Object> jsonMap = new HashMap<>();
            jsonMap.put("{field_01}", "{value_01}");
            jsonMap.put("{field_02}", "{value_02}");

            IndexRequest indexRequest = new IndexRequest("{index_name}", "_doc", "{doc_id}").source(jsonMap);
            IndexResponse indexResponse = highClient.index(indexRequest, COMMON_OPTIONS);
            long version = indexResponse.getVersion();
            System.out.println("Index document successfully! " + version);

            DeleteRequest deleteRequest = new DeleteRequest("{index_name}", "_doc", "{doc_id}");
            DeleteResponse deleteResponse = highClient.delete(deleteRequest, COMMON_OPTIONS);
            System.out.println("Delete document successfully! \n" + deleteResponse.toString());

            highClient.close();
        } catch (IOException ioException) {
            ioException.printStackTrace();
        }
    }
}

High-Concurrency Configuration

For high-concurrency scenarios, increase the number of client connections:

httpClientBuilder.setMaxConnTotal(500);
httpClientBuilder.setMaxConnPerRoute(300);

Sample code snippet:

String host = "127.0.0.1";
int port = 9200;
String username = "elastic";
String password = "passwd";
final int max_conn_total = 500;
final int max_conn_per_route = 300;

RestHighLevelClient restHighLevelClient = new RestHighLevelClient(
  RestClient.builder(new HttpHost(host, port, "http")).setHttpClientConfigCallback(new RestClientBuilder.HttpClientConfigCallback() {
    public HttpAsyncClientBuilder customizeHttpClient(HttpAsyncClientBuilder httpClientBuilder) {
      httpClientBuilder.setMaxConnTotal(max_conn_total);
      httpClientBuilder.setMaxConnPerRoute(max_conn_per_route);
      return httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider);
    }
  })
);

For more details on features and configurations, see the official Java High Level REST Client documentation.

Conclusion

Using the Java High Level REST Client ensures efficient interaction with your Alibaba Cloud Elasticsearch cluster. Follow this guide to make the most out of your Elasticsearch setup.
Ready to start your journey with Elasticsearch on Alibaba Cloud? Explore our tailored Cloud solutions and services to transform your data into a visual masterpiece.

Click here to embark on Your 30-Day Free Trial