DEV Community

gbenga fagbola
gbenga fagbola

Posted on

Building a Distributed Search Engine in Go

Introduction

Searching through large datasets efficiently is a common problem in distributed systems. This article demonstrates how to build a distributed search engine using Go’s concurrency features, allowing multiple workers to process and search data simultaneously.

Understanding the Concept

A distributed search engine divides the dataset into chunks, assigns each chunk to a worker, and runs multiple workers in parallel to perform the search. This improves performance by reducing the time needed to find a match, especially for large datasets.

In our example, we will:

  • Simulate a distributed search across a user database.
  • Use multiple worker goroutines to search concurrently.
  • Use channels for communication between workers and the main process.

Code Implementation

Below is the complete Go implementation of a distributed search engine:

package main

import (
    "log"
    "os"
    "strings"
    "time"
)

// User struct represents a user with an email and name
type User struct {
    Email string
    Name  string
}

// DataBase is a slice of User representing stored user data
var DataBase = []User{
    {Email: "adebayo.olu@example.com", Name: "Adebayo Olu"},
    {Email: "chioma.okafor@example.com", Name: "Chioma Okafor"},
    {Email: "ibrahim.abubakar@example.com", Name: "Ibrahim Abubakar"},
    {Email: "ngozi.uchenna@example.com", Name: "Ngozi Uchenna"},
    {Email: "chinedu.ekene@example.com", Name: "Chinedu Ekene"},
    {Email: "toyin.adebayo@example.com", Name: "Toyin Adebayo"},
    {Email: "uche.nwosu@example.com", Name: "Uche Nwosu"},
    {Email: "bola.johnson@example.com", Name: "Bola Johnson"},
    {Email: "femi.ogunleye@example.com", Name: "Femi Ogunleye"},
    {Email: "damilola.ogun@example.com", Name: "Damilola Ogun"},
    {Email: "amaka.nnamdi@example.com", Name: "Amaka Nnamdi"},
    {Email: "segun.olawale@example.com", Name: "Segun Olawale"},
    {Email: "bisi.adewale@example.com", Name: "Bisi Adewale"},
    {Email: "kunle.akintola@example.com", Name: "Kunle Akintola"},
    {Email: "funke.adebisi@example.com", Name: "Funke Adebisi"},
    {Email: "nkechi.okonkwo@example.com", Name: "Nkechi Okonkwo"},
}

// Worker struct represents a worker that processes a subset of users
type Worker struct {
    users []User  // Slice of users assigned to the worker
    ch    chan *User  // Channel for sending found users
    name  string  // Worker name identifier
}

// NewWorker initializes a new Worker instance
func NewWorker(users []User, ch chan *User, name string) *Worker {
    return &Worker{users: users, ch: ch, name: name}
}

// Find searches for a user email within the worker's assigned users
func (w *Worker) Find(email string) {
    for i := range w.users {
        user := &w.users[i]
        if strings.Contains(user.Email, email) { // Check if email contains search term
            w.ch <- user // Send found user to the channel
        }
    }
}

// DistributeWorkload creates workers dynamically based on database size
func DistributeWorkload(data []User, ch chan *User) {
    numWorkers := 3 // Define number of workers
    batchSize := len(data) / numWorkers

    for i := 0; i < numWorkers; i++ {
        start := i * batchSize
        end := start + batchSize
        if i == numWorkers-1 {
            end = len(data) // Assign remaining users to last worker
        }
        go NewWorker(data[start:end], ch, "Worker"+string(i+1)).Find(os.Args[1])
    }
}

func main() {
    if len(os.Args) < 2 {
        log.Fatal("Usage: go run main.go <email>")
    }

    email := os.Args[1] // Get the email argument from command line
    ch := make(chan *User) // Create a channel for user search results

    log.Printf("Looking for %s", email)

    // Distribute workload dynamically
    DistributeWorkload(DataBase, ch)

    // Listen for user results or timeout if no result is found
    for {
        select {
        case user := <-ch:
            log.Printf("The email is %s owned by %s", user.Email, user.Name)
        case <-time.After(100 * time.Millisecond):
            log.Printf("The email %s was not found", email)
            return // Exit loop after timeout
        }
    }
}
Enter fullscreen mode Exit fullscreen mode

Explanation of the Code

  1. User Data and Database:

    • We define a User struct containing Email and Name fields.
    • A slice of User acts as our database.
  2. Worker Struct and Methods:

    • Each Worker processes a subset of the database.
    • The Find method checks if an email contains the search term and sends results to a channel.
  3. Concurrency Using Goroutines:

    • The dataset is split among three workers.
    • Each worker runs concurrently to search for the provided email.
  4. Communication with Channels:

    • A channel is used to send found users from workers to the main function.
    • The select statement listens for results or a timeout.
  5. Timeout Mechanism:

    • If no results are received within 100ms, the search terminates with a “not found” message.

Running the Code

To run the program, execute:

$ go run main.go miller
Enter fullscreen mode Exit fullscreen mode

This will search for emails containing “miller” and return matching results.

Benefits of This Approach

  • Parallel Processing: Multiple workers search concurrently, reducing response time.
  • Scalability: More workers can be added to handle larger datasets.
  • Efficiency: The timeout prevents unnecessary waiting.

Conclusion

This article demonstrated how to build a simple distributed search engine using Go. By leveraging goroutines and channels, we efficiently parallelized the search process, making it scalable for larger datasets. Future improvements could include:

  • Using a distributed architecture like Apache Kafka or Elasticsearch.
  • Implementing an indexing mechanism for faster searches.
  • Running the system across multiple machines for even better performance.

Let me know your thoughts and how you’d optimize this approach further!

Billboard image

Deploy and scale your apps on AWS and GCP with a world class developer experience

Coherence makes it easy to set up and maintain cloud infrastructure. Harness the extensibility, compliance and cost efficiency of the cloud.

Learn more

Top comments (0)

Sentry image

See why 4M developers consider Sentry, “not bad.”

Fixing code doesn’t have to be the worst part of your day. Learn how Sentry can help.

Learn more