Building Multilingual Image Search with Marqo

Introduction

Multilingual Image Search

For global businesses, multilingual image search can prove to be a critical component for enhancing accessibility of their search platforms and expand their content reach. As e-commerce and digital platforms grow internationally, multilingual capabilities become essential in tapping into new markets and demographics.

Why Marqo?

Marqo differentiates itself by integrating state-of-the-art multilingual CLIP models with dense retrieval techniques, facilitating accurate and context-aware image searches across 100 languages.

In this article, we will explore steps to set up and build multilingual image search. We'll be covering end-to-end flow from System Requirements to implementation to Use cases.

Prerequisites

Before implementing a multilingual image search with Marqo, ensure your system meets the following requirements:

System Requirements

1. Python Environment: Requires Python 3.7 or later.
2. Hardware Requirements:
   • CPU: Modern multi-core processor (e.g., Intel Core i5/i7/i9, AMD Ryzen 5/7/9) with high clock speed for efficient processing.
   • GPU: Recommended for performance acceleration, especially beneficial for larger datasets. CUDA-compatible GPUs are ideal.
3. Memory: Minimum of 8GB RAM; more is beneficial for large datasets.

Setup and installation

Lets start by downloading and installing Marqo:

1. Install Docker. Follow Docker Docs.
2. Use Docker to run Marqo server instance:

docker pull marqoai/marqo:latest
docker rm -f marqo
docker run --name marqo -it -p 8882:8882 marqoai/marqo:latest

Setting up Marqo Client (in Python)

pip install marqo

Note:

1. If you're using Marqo 2.0.0, you'll need to use marqo1 package.
2. Find more detailed instructions on setting up Marqo here.

Code Walkthrough

In this section, we will go through implementation steps and code snippets for Multilingual image search with Marqo.

Initializing Search Index

Firstly, we initialize the Marqo search index with the desired settings.

Code Snippet

import marqo

def initialize_search_index(client_url, index_name, settings):
    # Initialize Marqo client with the provided URL
    mq = marqo.Client(url=client_url)
    try:
        # Attempt to create a new search index with specified settings
        mq.create_index(index_name, settings_dict=settings)
    except marqo.errors.MarqoWebError as e:
        # Check if index already exists to avoid duplication
        if e.code == 'index_already_exists':
            print(f"Index {index_name} already exists. Skipping creation.")
        else:
            # Raise other exceptions
            raise
    return mq

Explanation

• Client Initialization: Establishes connection to Marqo server.
• Index Creation: Tries to create a new index; handles case where index already exists.

Populating the Index with Image Data

We populate the index with images and their descriptions in multiple languages (English & Spanish).

Code Snippet

def populate_index_with_images(mq, index_name, image_data):
    # Add image data to the specified index
    return mq.index(index_name).add_documents(
        image_data,
        tensor_fields=['description', 'description_es'],  # Fields for dense retrieval
        device="cpu",  # Specify the device for processing (CPU in this case)
        client_batch_size=1  # Process data in individual batches
    )

Explanation

• Document Addition: Adds images and descriptions to the index.
• Tensor Fields: Fields used for embedding and retrieval.
• Batch Processing: Efficient data ingestion in batches.

Performing Searches in Different Languages

This step involves searching the index using queries in multiple languages.

Code Snippet

def perform_search(mq, index_name, query_text, language_code):
    # Choose the search field based on language code
    search_field = "description_es" if language_code == "es" else "description"

    # Perform the search with the given query
    search_results = mq.index(index_name).search(
        q=query_text,
        attributes_to_retrieve=[search_field, "url"],  # Fields to retrieve in results
        limit=1  # Limit the number of results
    )
    return search_results

Explanation

• Field Selection: Dynamically selects the search field based on language (description_es, description).
• Executing Query: Searches the index with the specified query and retrieves relevant fields.

Displaying Search Results

Finally, we extract and display image URLs from the search results.

Code Snippet

def display_image_urls(search_results):
    # Iterate through each hit in the search results
    for hit in search_results.get('hits', []):
        # Print the URL of the image
        print(hit.get('url'))

Explanation

• Result Iteration: Loops through each hit in the results.
• URL Display: Outputs the URL of each image.

Putting It All Together (Example Implementation)

This section demonstrates the main function, which ties the previously discussed functionalities into a complete workflow, implementing a client for this functionality.

Code Snippet

def main():
    client_url = "http://localhost:8882"
    index_name = "multilingual-image-search"
    settings = {
        "treatUrlsAndPointersAsImages": True,
        "model": "multilingual-clip/XLM-Roberta-Large-Vit-L-14",
        "normalizeEmbeddings": True,
    }

    # Initialize Marqo client and index
    mq = initialize_search_index(client_url, index_name, settings)

    # Image data format: List of dictionaries, each with 'url', 'description', and 'description_es'
    image_data = [
        # Example:
        # {"url": "http://example.com/image1.jpg", 
        #  "description": "Description in English", 
        #  "description_es": "Descripción en español"}
        # ... [Add your image data with descriptions here] ...
    ]

    # Populate the index with images
    populate_index_with_images(mq, index_name, image_data)

    # Perform searches in different languages
    english_results = perform_search(mq, index_name, "A streetwise ape with a flair for the dramatic", "en")
    spanish_results = perform_search(mq, index_name, "Un simio callejero con un sentido dramático", "es")

    # Print and display search results
    print("Search results for English query:", english_results)
    print("Search results for Spanish query:", spanish_results)

    print("\nImage URL from English query:")
    display_image_urls(english_results)

    print("\nImage URL from Spanish query:")
    display_image_urls(spanish_results)

if __name__ == '__main__':
    main()

Explanation

• Initialization: Sets up the client URL and index name, and defines the index settings.
• Index Population: Loads the image data into the Marqo index.
• Search Execution: Performs searches using English and Spanish queries.
• Result Display: Outputs the search results and displays the image URLs.

Note: Find the full implementation on Github.

Best practices and Optimization techniques

Optimize Index Settings

• Select the Right Model: Choose a model that best fits your dataset. For instance, multilingual-clip/XLM-Roberta-Large-Vit-L-14 is ideal for diverse linguistic content.
• Normalization: Use the normalizeEmbeddings setting for more accurate distance measurements in vector space, improving search relevance.

Efficient Data Management

• Batch Processing: When populating the index, adjust client_batch_size based on your system's capabilities to balance between speed and resource usage.
• Regular Index Updates: Keep your index up-to-date with new images and descriptions to maintain search relevance and accuracy.

Handle Multilingual Content Effectively

• Field Mapping: Use language-specific fields (like description_es for Spanish) to store and retrieve language-specific content.
• Language Detection: Implement automatic language detection in your application to choose the appropriate search field dynamically.

Performance Tuning

• Hardware Utilization: If using a GPU, ensure Marqo is configured correctly to leverage it, significantly speeding up indexing and search operations.
• Query Optimization: Refine your queries based on user feedback and search patterns. Short, concise queries often yield better results.

Utilize Marqo's Advanced Features

• Synonyms and Stopwords: Explore advanced features like synonyms and stopwords to refine the search experience.
• Custom Ranking: Use custom ranking parameters to tailor search results based on specific criteria like popularity or recency.

Use Cases for Multilingual Image Search

Multilingual image search powered by Marqo offers specialized applications across many industries. Let's explore some specific use cases:

E-Commerce

• Global Product Discovery: Online shoppers from different regions can search for products like red evening dress or 晚礼服 in their native languages, significantly enhancing user experience.
• Localized Marketing: Retailers can present products with images that resonate culturally with each linguistic market, like showcasing traditional attire in searches from specific regions.
• Customer Reviews and Q&A: Multilingual search can help in navigating through customer reviews and questions in various languages, aiding in informed purchase decisions.

Stock Photo Libraries

• Broad Accessibility: Photographers can tag their work in multiple languages, enabling users worldwide to discover relevant images, like searching for 日出 or sunrise to find similar thematic content.
• Cultural Archives: Stock libraries can serve as digital archives, preserving cultural imagery that's searchable in multiple languages, thus aiding cultural preservation and education.

Media and Entertainment

• Targeted Content Distribution: Distributors can use multilingual image search to categorize films or shows based on culturally relevant themes, optimizing content for different linguistic audiences.
• Promotional Campaigns: Movie studios can create promotional campaigns with imagery that resonates differently with each language-speaking audience, enhancing global reach and relatability.

Conclusion and External References

Recap & Key Points

In this guide, we explored the implementation of multilingual image search using Marqo, starting from the System Requirements and installation, to a detailed code walkthrough.

We emphasized on best practices, error handling, and optimization strategies to enhance search performance. The exploration of diverse use cases across sectors like e-commerce, stock photos, and Media & Entertainment highlighted the versatility and impact of Marqo.

Further Exploration

To explore Marqo further, here are some valuable resources:

• Marqo Documentation: Dive deeper into Marqo's capabilities and features by visiting Marqo's official documentation.
• Multilingual CLIP Models: Understand the underlying models used in Marqo for multilingual search by reading about Multilingual CLIP models.
• Community and Support: Engage with the Marqo community and seek support for specific issues on GitHub.
• Case Studies and Articles: Gain insights from practical implementations and case studies, such as the Medium article on implementing text-to-image search using Marqo.