DEV Community: Prem

Size Doesn't Matter: Why Your Elasticsearch Fields Need to Stop Caring About Length

Prem — Wed, 06 Nov 2024 03:52:06 +0000

Understanding Elasticsearch Field Length Normalization: When and Why to Disable Norms

"Why Are My Elasticsearch Scores Wrong?" -"Why are my documents with identical content scoring differently?"
"How come shorter fields always seem to rank higher?"
"Why doesn't my exact-match search return equal scores?"

If you've ever asked these questions while working with Elasticsearch, you're not alone. You've probably noticed that documents with the same matching terms sometimes receive unexpectedly different scores. What might appear as random scoring variations actually stems from one of Elasticsearch's most impactful – and often misunderstood – relevance features: field length normalization, also known as norms.

This built-in feature, while powerful for natural language search, can sometimes be your search results' worst enemy. It's like having an overeager assistant who assumes shorter is always better – helpful when summarizing novels, but problematic when dealing with product codes or categories. Let's unravel this mystery and discover when this "helpful" feature might be secretly sabotaging your search results, and more importantly, how to fix it.

When working with Elasticsearch, one of the most subtle yet impactful aspects of relevance scoring is field length normalization. While this feature is beneficial for natural language search, it can sometimes work against you depending on your use case. Let's dive deep into what norms are, when they help, when they hurt, and how to control them.

What are Norms?

Norms are scoring factors in Elasticsearch that contribute to how relevance is calculated for a document. One of their primary functions is field length normalization - making shorter fields score higher than longer ones when they contain the same search term.

A Real-World Example

Imagine you're building a recipe search engine. You have two fields:

title: The recipe name
instructions: The step-by-step cooking instructions

Let's look at two recipes:

{
  "title": "Simple Tomato Pasta",
  "instructions": "Boil pasta. Add tomato sauce. Serve."
}

{
  "title": "Tomato and Basil Pasta with Fresh Garden Herbs and Parmesan",
  "instructions": "1. Boil water and cook pasta until al dente. 2. In a pan, sauté garlic... [50 more detailed steps]"
}

When someone searches for "tomato pasta", with default norms enabled, the first recipe might score higher because:

The term "tomato" in a shorter title field gets more weight
The shorter instructions field doesn't dilute the overall score

When Norms Help

Article Search: When searching article content, a keyword appearing in a 100-word article might be more relevant than the same keyword in a 10,000-word article.
Product Descriptions: A product specifically about "bluetooth headphones" might have a shorter, more focused description than one that merely mentions bluetooth headphones as a compatible accessory.

When Norms Hurt

SKU/Product Code Search: Consider a product catalog with a product_codes field:

{
  "product_codes": ["ABC123", "XYZ789", "DEF456"]
}

{
  "product_codes": ["ABC123"]
}

When searching for "ABC123", should the first product score lower just because it has more product codes? Probably not!

Category Lists: For an e-commerce site with product categories:

{
  "categories": ["Electronics", "Computers", "Laptops"]
}

{
  "categories": ["Electronics"]
}

A search for "Electronics" should treat both products equally, regardless of how many categories they belong to.

How to Disable Norms

When you decide norms aren't appropriate for your use case, you can disable them in your mapping:

{
  "mappings": {
    "properties": {
      "product_codes": {
        "type": "text",
        "norms": false
      }
    }
  }
}

Impact of Disabling Norms

Positive Effects:

More predictable scoring for structured data
Reduced index size (saves 1 byte per field per document)
Slightly improved indexing performance
Reduced memory usage

What You Lose:

Field-length normalization
Index-time field boost capabilities

Best Practices

Enable norms for:
- Full-text fields like article content, descriptions
- Fields where length indicates relevance
- Search scenarios requiring fine-tuned relevance scoring
Disable norms for:
- Identifier fields
- Category or tag lists
- Structured data where field length doesn't indicate relevance
- Fields used primarily for filtering rather than scoring

Implementation Strategy

When implementing norm changes in production:

Create a new index with updated mappings
Use aliases to point to the current index
Reindex data to the new index
Switch the alias to point to the new index
Remove the old index

Conclusion

Understanding and properly configuring norms is crucial for building effective search experiences. While Elasticsearch's defaults work well for natural language text, structured data often benefits from disabling norms. By carefully considering your use case and data characteristics, you can make informed decisions about norm configuration and improve both search relevance and performance.

Remember: The best configuration is one that matches your users' expectations of what makes a document relevant. Don't be afraid to experiment and test different approaches with real user queries and feedback.

How to Clean Up Deleted Git Branches: A Developer's Guide

Prem — Tue, 22 Oct 2024 19:48:41 +0000

In any active development environment, Git branches can accumulate quickly. While remote branches might be deleted after merging pull requests, their local counterparts often remain, cluttering your workspace. This guide will show you how to efficiently clean up these obsolete local branches.

The Problem

When working with Git, especially in a team environment, you'll often create multiple branches for different features, bug fixes, or experiments. After merging these branches into the main codebase and deleting them from the remote repository (like GitHub or GitLab), the local references often stick around on your machine. This can lead to:

Cluttered branch listings
Confusion about which branches are still relevant
Unnecessary use of disk space
Difficulty in managing active branches

The Solution

Here's a step-by-step guide to cleaning up your local Git branches that no longer exist on the remote repository.

Step 1: Update Your Local Repository

First, fetch the latest changes from your remote repository and clean up remote tracking branches:

git fetch --prune

The --prune flag is crucial here - it removes references to remote branches that no longer exist on the remote repository.

Step 2: Identify Obsolete Branches

To see which local branches no longer have a corresponding remote branch:

git branch -vv | grep ': gone]' | awk '{print $1}'

This command:

git branch -vv: Shows all local branches with their tracking information
grep ': gone]': Filters for branches whose remote tracking branch is gone
awk '{print $1}': Extracts just the branch names

Step 3: Delete the Branches

You have two options for deletion:

Safe Deletion

To delete only fully merged branches:

git branch -d $(git branch -vv | grep ': gone]' | awk '{print $1}')

This command will fail if there are unmerged changes, protecting you from accidentally losing work.

Force Deletion

To delete branches regardless of their merge status:

git branch -D $(git branch -vv | grep ': gone]' | awk '{print $1}')

⚠️ Warning: Use force deletion with caution, as it will delete the branches even if they contain unmerged changes.

Best Practices

Regular Maintenance: Run these commands periodically to keep your repository clean.
Check Before Deleting: Before force-deleting branches, ensure you won't need the changes later.
Backup Important Work: If unsure, create a backup branch or tag before cleaning up.
Communicate with Team: Ensure branches are truly obsolete before removing them locally.

Automating the Process

You can create a Git alias for this cleanup process. Add this to your ~/.gitconfig:

[alias]
    cleanup = "!git fetch --prune && git branch -vv | grep ': gone]' | awk '{print $1}' | xargs git branch -d"

Then simply run:

git cleanup

Conclusion

Keeping your Git repository clean and organized is essential for efficient development. By following these steps, you can easily maintain a clutter-free workspace and focus on what matters most - writing great code.

Remember: While it's important to clean up unused branches, always verify that you're not deleting important work. When in doubt, use the safe deletion method (-d) rather than force deletion (-D).

Happy coding! 🚀

Amazon Lightsail vs AWS EC2: Which one is better?

Prem — Sat, 24 Feb 2024 21:33:35 +0000

So you're diving into the exciting world of cloud computing, and naturally, the question arises: Which server should I choose? Amazon Web Services (AWS) throws two heavyweights into the ring: Amazon Lightsail and AWS EC2. Both offer virtual servers, but they cater to different needs and user profiles. Let's unpack their strengths and weaknesses to help you pick the champion for your cloud journey.

Lightsail: The Easy Breezy Contender

Imagine a server that sets up in minutes, with no complex configurations to wrestle with. That's Lightsail in a nutshell. It's like a pre-built PC, offering bundles that combine virtual machines (VMs), storage, and networking into neat packages. Perfect for beginners or those who value simplicity, Lightsail takes care of the behind-the-scenes stuff, letting you focus on what matters: building your application.

EC2: The Granular Gladiator

EC2 is the seasoned warrior, offering unmatched control and flexibility. It provides a vast arsenal of individual components, from various VM types to intricate networking configurations. This granular approach empowers experienced users to craft their infrastructure with surgical precision. However, like any powerful tool, EC2 comes with a steeper learning curve.

The Deciding Factors: When to Choose Which

Choosing the right server boils down to a few key considerations:

Technical Expertise: New to the cloud game? Lightsail is your friendly neighborhood guide. Its intuitive interface and pre-configured options make launching your server a breeze. EC2, on the other hand, demands more technical prowess.
Application Complexity: For simple websites or small projects, Lightsail's bundled resources are your trusty companions. But if you're building a complex application with specific requirements, EC2's granular control becomes your secret weapon.
Scalability: Need to adapt to growing demands? Lightsail offers limited scaling options, mainly through upgrading bundles. EC2 grants you the freedom to scale individual resources like CPU, memory, and storage independently.
Cost: Lightsail's pricing is transparent and predictable, with monthly charges based on your chosen bundle. EC2's pricing is more nuanced, varying based on the specific resources you use and billed hourly.
The Verdict: Choosing Your Champion

Lightsail emerges victorious for:

Users seeking a simple and user-friendly experience
Quick deployments with pre-configured bundles
Predictable pricing for small-scale projects

EC2 reigns supreme for:

Experienced users comfortable with granular control
Complex applications with specific resource requirements
Scalability to meet fluctuating demands

Ultimately, the best server is the one that aligns with your specific needs and technical expertise. If you're unsure, starting with Lightsail's user-friendly approach and graduating to EC2 as your skills and requirements grow can be a winning strategy. Remember, the cloud is your playground, so experiment, explore, and find the server that empowers you to build your dreams!

Planting Flowers with No Adjacent Restrictions

Prem — Sun, 08 Oct 2023 09:05:27 +0000

Gardening can be a delightful pastime, but it comes with its own set of rules and constraints. In the world of programming, we sometimes encounter similar challenges, such as the "Planting Flowers with No Adjacent Restrictions" problem. In this blog post, we'll explore this problem and discover an efficient solution using the greedy algorithm approach.

The Problem

Imagine you have a long flowerbed with some plots where flowers are already planted (denoted by 1) and some empty plots (denoted by 0). However, there's a catch: flowers cannot be planted in adjacent plots. You're given a binary array flowerbed representing the flowerbed's current status, where 0 indicates an empty plot and 1 indicates a planted flower. You also have an integer n, representing the number of new flowers you want to plant.

The task is to determine if it's possible to plant n new flowers in the flowerbed without violating the no-adjacent-flowers rule.

The Approach

To solve this problem efficiently, we'll use a greedy algorithm approach. The idea is to iterate through the flowerbed and check each empty plot. If a plot is empty and the adjacent plots (if they exist) are also empty, we can plant a flower in that plot. We'll keep track of the number of flowers planted and return true if we successfully plant n flowers.

Here's a step-by-step approach:

Initialize a variable count to 0 to keep track of the number of flowers planted.
Iterate through the flowerbed array from left to right.
For each empty plot (0), check if the previous and next plots (if they exist) are also empty. If so, plant a flower (set the plot to 1) and increment the count by 1.
Continue this process until we reach the end of the flowerbed or plant n flowers.
After the loop, check if the count is equal to n. If it is, return true, indicating that we successfully planted n flowers without violating the adjacent flower rule. Otherwise, return false.

The Code

Here's the Go code implementing the described approach:

func canPlaceFlowers(flowerbed []int, n int) bool {
    count := 0
    i := 0
    for i < len(flowerbed) {
        if flowerbed[i] == 0 {
            // Check if previous and next plots are empty (or out of bounds)
            prev := i == 0 || flowerbed[i-1] == 0
            next := i == len(flowerbed)-1 || flowerbed[i+1] == 0
            if prev && next {
                flowerbed[i] = 1
                count++
            }
        }
        i++
    }
    return count >= n
}

Example Usage

Let's see how to use the canPlaceFlowers function with some examples:

fmt.Println(canPlaceFlowers([]int{1, 0, 0, 0, 1}, 1)) // Output: true
fmt.Println(canPlaceFlowers([]int{1, 0, 0, 0, 1}, 2)) // Output: false

Conclusion

The "Planting Flowers with No Adjacent Restrictions" problem challenges us to find an efficient way to plant flowers in a flowerbed while adhering to the rule that no two flowers can be adjacent. By applying the greedy algorithm approach, we can iteratively check each plot and decide whether to plant a flower, ultimately determining if it's possible to plant the desired number of flowers.

This problem-solving technique can be applied to various real-world scenarios where we need to make decisions based on local conditions and constraints.

Reversing Vowels in a String Using Two Pointers in Golang

Prem — Sun, 08 Oct 2023 09:02:24 +0000

When working with strings in Go (Golang), you may come across scenarios where you need to manipulate the characters within the string. One common task is reversing the positions of vowels while keeping consonants and other characters in their original places. In this blog post, we will explore how to achieve this using a simple and efficient approach with the help of two pointers.

The Problem

Given an input string, we want to reverse the positions of vowels (i.e., 'a', 'e', 'i', 'o', 'u', 'A', 'E', 'I', 'O', 'U') while preserving the positions of consonants and other characters. For example, if the input string is "hello," the output should be "holle."

The Approach

To solve this problem, we will use two pointers, one starting from the beginning of the string (left) and the other starting from the end of the string (right). We will iterate through the string and swap vowels found at these two pointers. The process continues until the left pointer is less than the right pointer.

Here's the step-by-step approach:

Initialize a string variable vowels containing all lowercase and uppercase vowel characters: "aeiouAEIOU."
Convert the input string into a mutable rune slice called strRunes. In Go, strings are immutable, so we need to work with runes to make modifications.
Initialize two pointers, left and right, to 0 and len(s)-1, respectively, where s is the input string.
Use a while loop to iterate through the string while the left pointer is less than the right pointer.
Within the loop:
- Increment the left pointer until a vowel character is found or until left is no longer less than right.
- Decrement the right pointer until a vowel character is found or until left is no longer less than right.
If the left pointer is still less than the right pointer, swap the characters at these positions. Then, increment the left pointer and decrement the right pointer.
After the loop, return the string representation of the modified strRunes.

The Code

Here's the Go code implementing the described approach:

func reverseVowels(s string) string {
    vowels := "aeiouAEIOU"
    strRunes := []rune(s)
    left, right := 0, len(s)-1

    for left < right {
        for left < right && !strings.ContainsRune(vowels, strRunes[left]) {
            left++
        }
        for left < right && !strings.ContainsRune(vowels, strRunes[right]) {
            right--
        }
        if left < right {
            strRunes[left], strRunes[right] = strRunes[right], strRunes[left]
            left++
            right--
        }
    }

    return string(strRunes)
}

Example Usage

Let's see how to use the reverseVowels function with some examples:

fmt.Println(reverseVowels("hello")) // Output: "holle"
fmt.Println(reverseVowels("leetcode")) // Output: "leotcede"
fmt.Println(reverseVowels("GoLang")) // Output: "GoLeNg"

Conclusion

Reversing vowels in a string is a common string manipulation task that can be efficiently solved using two pointers in Go. By following the approach described in this blog post, you can easily reverse the positions of vowels while maintaining the order of consonants and other characters in the string. This technique can be handy in various text processing scenarios where you need to modify or analyze text data.

How to install make on windows

Prem — Sat, 26 Aug 2023 11:32:15 +0000

Installing make on Windows involves using tools like MinGW or Cygwin to provide a Unix-like environment that includes the make utility. Here's a step-by-step guide using MinGW:

Install Chocolatey (Optional but Recommended): Chocolatey is a package manager for Windows that makes it easy to install software from the command line.

Open a Command Prompt with administrator privileges and run the following command to install Chocolatey:

   Set-ExecutionPolicy Bypass -Scope Process -Force; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; iex (iwr https://chocolatey.org/install.ps1)

Install MinGW: MinGW (Minimalist GNU for Windows) is a development environment that provides a Unix-like command-line environment on Windows.

Run the following command in the Command Prompt to install MinGW using Chocolatey:

   choco install mingw -y

Add MinGW Bin Directory to PATH: After installing MinGW, you need to add its bin directory to your system's PATH environment variable.

Right-click on "This PC" or "My Computer" and select "Properties."
Click on "Advanced system settings."
In the "System Properties" window, click on the "Environment Variables" button.
Under the "System variables" section, find and select the "Path" variable, then click the "Edit" button.
Click the "New" button and add the path to the MinGW bin directory (usually C:\MinGW\bin).
Click "OK" to close all the windows.

Verify Installation: To verify that make has been successfully installed, open a new Command Prompt and run the following command:

   make --version

You should see the version information of the make utility if the installation was successful.

That's it! You've now successfully installed make on your Windows system using MinGW. You can now use the make command to build projects that use Makefiles for compilation and other tasks.

How to install mockery in windows?

Prem — Wed, 07 Dec 2022 16:22:14 +0000

Struggling with installing mockery in windows?
Say no more. Just a single command and there you go.

Step 1: open cmd. (Or Terminal, your choice)
Step 2: Run the following command.
go install github.com/vektra/mockery/v2@latest

And that's it.
This single command would suffice.

Thanks for stopping by btw.

Cheers!!!

Setting up golang-migrate for windows

Prem — Mon, 13 Sep 2021 20:56:33 +0000

First Install scoop

But Wait! What is scoop

Scoop is a command-line package manager for Windows which makes it easier to install and use common programs and tools.

Make sure PowerShell 5 (or later, include PowerShell Core) and .NET Framework 4.5 (or later) are installed. Then run:

Invoke-Expression (New-Object System.Net.WebClient).DownloadString('https://get.scoop.sh')

or 
iwr -useb get.scoop.sh | iex

In case you get an error, you might need to change the execution policy (i.e. enable Powershell) with:

Set-ExecutionPolicy RemoteSigned -scope CurrentUser

Now that you have installed scoop on your windows system, we can move ahead with our main task i.e. golang-migrate

Download pre-built binary (Windows, MacOS, or Linux)

curl -L https://github.com/golang-migrate/migrate/releases/download/$version/migrate.$platform-amd64.tar.gz | tar xvz

Now that we have downloaded scoop and the pre-built library for golang-migrate, we will go ahead and install it using this simple command

For Windows

scoop install migrate

And There You GO!!!
Your golang-migrate is setup for windows.