DEV Community: Pepe

It's time to try OMARCHY!

Pepe — Tue, 18 Nov 2025 15:48:05 +0000

For a long time, I’ve been curious about trying something new in the Linux world, specifically, a distribution built on top of Arch. While I’ve always appreciated the stability and familiarity of Ubuntu, the idea of a more minimal, rolling release system kept pulling me in. That’s how I ended up testing Omarchy Linux, a lightweight Arch-based distribution designed to offer a clean, modern, and highly customizable environment.

For my testing purposes, I am using one of my old laptops - Asus Zenbook UX305L.

CPU: Intel Core i5-5200U
GPU: Intel HD Graphics 5500.
Memory: 8 GB DDR3 RAM.
Storage: 128 GB SSD.
Connectivity: Wi-Fi AC, Bluetooth, USB 3.0 and LAN.

What Is Omarchy Linux?

Omarchy Linux is an Arch-based distro that focuses on simplicity, performance, and a smooth user experience. It comes preconfigured with Hyprland, a dynamic and eye-catching Wayland compositor known for its animations, tiling capabilities, and focus on modern hardware acceleration. The distro aims to deliver a polished setup out of the box while still keeping the Arch philosophy—lightweight, cutting-edge, and under your control.

Downloading the ISO & Creating a Bootable USB

Getting started with Omarchy Linux was simple. I downloaded the latest ISO directly from the official project page, which provides a clean and compact image ready for installation.

To create a bootable USB, I used BalenaEtcher, a reliable cross-platform tool that makes flashing ISO files effortless. After selecting the Omarchy ISO, choosing my USB drive, and hitting Flash, the process completed within minutes.

With the bootable USB ready, I rebooted my machine, selected the stick from the boot menu, and was prepared to start the installation.

A Simple and Intuitive Installation

One pleasant surprise was the installation process. Unlike traditional Arch, where you manually configure almost everything, Omarchy Linux provides an intuitive and guided installer. It’s straightforward enough for intermediate Linux users, yet flexible enough to customize partitions or choose specific components. Everything—from disk setup to user creation—felt smooth and fast.

Within minutes, the system was ready to boot into the Hyprland desktop environment.

First Boot & My Experience With Hyprland

The first login into Hyprland was impressive. The animations felt fluid, the tiling behaviour was responsive, and the overall interface looked clean without being cluttered. Hyprland gives you a sense of control and minimalism, but with just enough eye candy to make everything feel modern.

Coming from Ubuntu’s GNOME environment, the transition definitely required adjusting, especially with the floating-vs-tiling dynamics. But after a few minutes of exploring, the workflow quickly became enjoyable—and surprisingly efficient.

Keyboard Shortcuts

Hyprland relies heavily on keyboard shortcuts, which is one of its biggest strengths. After the first startup of Omarchy you will get a hint for the most useful shortcuts, like entering the control menu and application launcher. Here is the list of some of them, which might help you, when you are jumping from different desktop environments as I did.

Windows + K - keyboard bindings
Windows + Space - Application Launcher
Windows + Return - Open default Terminal
Windows + W - close the active window
Super + 1/2/3/4 - Jump to specific workspace
Shift + Super + 1/2/3/4 - Move window to workspace

more shortcuts you can find here.

Mastering these shortcuts makes the environment extremely fast to navigate and much more efficient than a traditional mouse-driven workflow. It is also a game-changer in productivity, at least I felt it.

Final Thoughts

Testing Omarchy Linux has been a refreshing experience. The combination of an easy installation, a modern Wayland environment, and the flexibility of an Arch base makes it a compelling option for daily use, especially for Devs. I genuinely enjoyed working with Hyprland and the overall lightweight feel of the system.

Omarchy comes with a well-thought-out selection of predefined software and dotfiles, giving the system a ready-to-use feel right after installation. The curated setup saves a lot of initial configuration time and provides a clean starting point for productivity. Installing or removing software is extremely easy. Pacman and the AUR offer a huge library of tools, all accessible through a fast and consistent workflow.

I’m planning to spend more time working in Omarchy Linux to see how it performs for real-world DevOps tasks. And who knows if things continue going this well, I might even consider a full switch from Ubuntu in the near future.

When it comes to my Windows development environment, though, I’ll continue using Ubuntu as my WSL distro. Ubuntu remains the most stable and well-supported option in the WSL ecosystem, especially for DevOps-focused tasks where reliability and predictable updates matter.

See ya and Happy coding!

My Dev Environment [CLI Tools]

Pepe — Tue, 18 Nov 2025 15:02:07 +0000

The command line is where real productivity happens. It's fast, scriptable, and distraction-free. Over time, I’ve tested numerous tools and utilities to enhance my terminal workflow, making it smoother, smarter, and more efficient. In this post, I’ll share the most useful CLI tools I rely on every day.

Whether you’re a developer, DevOps engineer, or just someone who loves optimizing their setup, these tools will help you work faster and enjoy a cleaner, more powerful terminal experience.

Neovim + NvChad

Neovim is my daily driver for editing code and configuration files. Combined with NvChad, it becomes a fast, modern IDE with smart defaults, LSP support, treesitter syntax highlighting, and a beautiful UI. I love how customizable it is. I can tweak everything from keybindings to themes, making it fit my exact workflow. Even though it's my secondary code editor, but No.1 in CLI.

If you are not sure about shortcuts, you can always open the nvchad cheatsheet with + c + h. I'm using space as a leader key.

bat

bat is a drop-in replacement for cat, but with style. It adds syntax highlighting, line numbers, and Git integration, making it much easier to read code or config files directly in the terminal. Once you try it, going back to plain cat feels impossible.

fzf

fzf is a fuzzy finder that makes searching through files, history, or command outputs effortless. I use it daily to quickly open files, find Git commits, or browse command history. It’s one of those tools that quietly saves seconds every minute.

zoxide

zoxide is a smarter cd. It remembers the directories I visit most often and lets me jump to them with just a few keystrokes. It’s perfect for moving around complex project structures without typing long paths over and over.

yazi

yazi is a terminal file manager that’s both fast and visually clean. It feels like a modern version of ranger, with smooth navigation, previews, and Vim-style keybindings. I use it to browse and organize files when I want more context than a simple ls can give.

stow

GNU stow makes managing dotfiles across multiple machines effortless. It symlinks configuration files from a single repository into the right places, keeping everything neat and version-controlled. It’s my favorite way to maintain a consistent setup everywhere.

tmux

tmux is the ultimate terminal multitasking tool. It lets me split the screen, detach sessions, and reconnect later without losing progress. Whether I’m monitoring logs, editing code, or running long builds, tmux keeps everything organized in one place.

lazygit

lazygit turns Git into a visual, keyboard-driven experience inside the terminal. It’s perfect for staging, committing, and reviewing changes quickly — especially during code reviews or when managing multiple branches. It’s Git made simple and fast.

glow

glow lets me preview Markdown files directly in the terminal with elegant formatting. I use it for reading documentation, notes, or README files without ever leaving the CLI. It’s small, fast, and looks great.

Honorable Mentions

A few more tools that deserve a shout-out:

fd – a faster, more user-friendly alternative to find.

eza – a modern ls replacement with icons, colors, and Git integration.

tldr – simplified man pages that show practical examples.

ripgrep – lightning-fast recursive search for code and text.

bottom – a resource monitor with a clean TUI and system insights.

procs – a colorful, structured alternative to ps for process management.

I hope this list of useful CLI tools I am using will help you enhance your command line productivity. See you in the next blog post.

Happy coding!

My Dev Environment [Zshell]

Pepe — Thu, 23 Oct 2025 12:26:02 +0000

If you spend a good chunk of your day in the terminal, you know how much difference a well-tuned shell can make. For years, I worked with the default Bash setup. Yeah, it got the job done, but it always felt a bit… dull. Then I discovered Zsh, a powerful, extensible shell that turns the command line into a fast, elegant, and personalized workspace.

This blog post is my personal guide to configuring the command line with all the tools I use most often. So, let's start.

My preferred distro for development is Ubuntu (either as WSL or a standalone version)

First things first - update:

sudo apt update && sudo apt upgrade -y

Then we can download the latest zshell from the repository:

sudo apt install zsh

and make it your default shell

chsh -s $(which zsh)

I’m using a combination of Oh My Zsh and Oh My Posh to take my Zsh setup to the next level. Oh My Zsh handles the core customization like managing plugins, aliases, and shell behavior, making the terminal faster and more capable. On top of that, Oh My Posh adds a beautiful, information-rich prompt with support for Git status, context awareness, and dynamic themes. Using both gives me the best of both worlds. However, in future I might end up with Oh My Posh only.

To display the icons correctly, we need to install Nerd Fonts. I am using MesloLG Nerd Font.

In WSL Terminal Settings, you have to configure it in your Ubuntu profile > Appearance > Font Face [MesloLGM Nerd Font].

Oh My Zsh installation

sh -c "$(curl -fsSL https://raw.githubusercontent.com/ohmyzsh/ohmyzsh/master/tools/install.sh)"

in your ~/.zshrc file create ZSH_CUSTOM environment variable. It will be used as a path to custom plugins.

export ZSH_CUSTOM="$HOME/.zsh_custom"

Installation of plugins for autosuggestion and syntax highlighting:

git clone https://github.com/zsh-users/zsh-autosuggestions.git $ZSH_CUSTOM/plugins/zsh-autosuggestions

git clone https://github.com/zsh-users/zsh-syntax-highlighting.git $ZSH_CUSTOM/plugins/zsh-syntax-highlighting

Again, let's update our ~/.zshrc config file

plugins=(git zsh-autosuggestions zsh-syntax-highlighting)

Now we can install Oh My Posh:

curl -s https://ohmyposh.dev/install.sh | bash -s

Changing our ~/.zshrc for the last time

export PATH=$HOME/.local/bin:$PATH
eval "$(oh-my-posh init zsh)"

exec zsh

Now we should have a prompt which looks like this:

I really love the default theme because it has everything I need for my job, including username, current folder, branch name, and current Git status, as well as providing visual information about the last successful/failed command. But you can choose from these themes here and you can even customize them.

If you want to customize the default theme, firstly export the config do the changes you like, and initialize your oh-myposh with the new config.

oh-my-posh config export --output ~/default.omp.json

updated default.omp.json use as new configuration file in .zshrc

eval "$(oh-my-posh init zsh --config ~/default.omp.json)"

Thanks for reading, and see you in my next blog post.
Happy coding!

My Note-taking System or Zettelkasten for Devs

Pepe — Fri, 17 Oct 2025 14:18:46 +0000

I’ve been using the Zettelkasten method with Obsidian to take notes, connect ideas, and build a personal knowledge system. In this post, I’ll share how I’ve set it up, why it works for me, and how it’s helped me think more clearly and creatively.

I tried several note-taking apps like Evernote, Notion, OneNote, Google Keep; but none felt quite right until I stumbled upon Obsidian. One of its biggest advantages for me is the high level of customization and the fact that notes are written in Markdown and stored locally. However, it wasn’t just about switching apps—my entire note-taking workflow had to change.

Customization (Themes, Fonts, Plugins)

First of all, I changed my Obsidian theme. In Settings > Appearance > Themes, you can manage themes. My favourite one is Cupertino, but FastPpuccin and Tokyo Night are pretty good as well.

For fonts, I’m using JetBrains Mono. It’s a really nice monospace font and reminds me of working in an IDE rather than a note-taking app. You can change it in Settings > Appearance > [Interface font | Text font | Monospace font]. I use the same font for each.

Here’s my list of installed community plugins (Settings > Community plugins):

calendar
git
kanban
iconize
mind map
excalidraw
commander
tasks
style settings
templater
advanced tables

The top ones by usage are:

Git – for syncing notes to a private GitHub repo
Iconize – for customizing folder structure
Calendar – for daily notes

I’m also using a custom CSS snippet for rainbow folder colors. I found a good snippet in the Obsidian forum here in this thread.

.tree-item-self.nav-folder-title.fn-is-active,
.tree-item-self.nav-file-title.fn-is-active,
.nav-folder-title.is-active,
.nav-file-title.is-active {
    background-color: var(--background-modifier-hover);
}

.nav-files-container>div {

    >.tree-item:nth-of-type(8n+1) * {
        --folder-color: var(--color-pink);
        --folder-color-rgb: var(--color-pink-rgb);
    }

    >.tree-item:nth-of-type(8n+2) * {
        --folder-color: var(--color-red);
        --folder-color-rgb: var(--color-red-rgb);
    }

    >.tree-item:nth-of-type(8n+3) * {
        --folder-color: var(--color-orange);
        --folder-color-rgb: var(--color-orange-rgb);
    }

    >.tree-item:nth-of-type(8n+4) * {
        --folder-color: var(--color-yellow);
        --folder-color-rgb: var(--color-yellow-rgb);
    }

    >.tree-item:nth-of-type(8n+5) * {
        --folder-color: var(--color-green);
        --folder-color-rgb: var(--color-green-rgb);
    }

    >.tree-item:nth-of-type(8n+6) * {
        --folder-color: var(--color-cyan);
        --folder-color-rgb: var(--color-cyan-rgb);
    }

    >.tree-item:nth-of-type(8n+7) * {
        --folder-color: var(--color-blue);
        --folder-color-rgb: var(--color-blue-rgb);
    }

    >.tree-item:nth-of-type(8n+8) * {
        --folder-color: var(--color-purple);
        --folder-color-rgb: var(--color-purple-rgb);
    }

    >.tree-item {

        * {
            --nav-item-color-active: var(--folder-color);
            --nav-item-background-active: var(--background-modifier-hover);
            --nav-item-background-hover: rgba(var(--folder-color-rgb), var(--folder-hover-opacity));
            --nav-indentation-guide-color: rgba(var(--folder-color-rgb), var(--folder-indentation-opacity));
        }

        .tree-item-children {
            border-inline-start: var(--nav-indentation-guide-width) solid var(--nav-indentation-guide-color);
        }

        .nav-folder-title {
            --nav-item-color-hover: var(--folder-color);
            --nav-collapse-icon-color: var(--folder-color);
            --nav-item-color: var(--folder-color);
        }

        .nav-file-title: "hover,"
        .tree-item-self.fn-is-active:hover {
            background-color: rgba(var(--folder-color-rgb), var(--folder-hover-opacity));
        }
    }
}

Last but not least, the Drag-and-Drop feature, you can completely change the experience of using Obsidian. This is how it might look in the end:

Note-taking Workflow

Vault, Folders, Metadata, Tags, Backlinks

I am using one vault for everything. You don't need to separate your personal notes from study, work, and travel logs. I recommend using the same vault for your whole life, and you might be surprised by the relations between them. There are some rules for the Zettelkasten method I'm not following at all. However, there is no "one fits them all" rule, so I have little bit improved it for myself.

My folder structure looks like this:

The naming convention in my case is {NUMBER} - {FOLDER NAME}, so I am in charge of the folder order. An icon in front of the name was added by the plugin iconize. You can choose from many of them, and each folder or even file can have a different icon. I am not using any subfolders, only this base structure.

Folder Flow Overview

Folder	Purpose	Typical Note Types	Next Step
00 - Drafts	Capture quick thoughts, meeting notes, or daily reflections.	Daily logs, idea dumps, scratch notes.	Move to Develop when you refine the idea.
01 - Develop	Build structured knowledge from drafts.	Concepts, designs, troubleshooting guides.	Link to Research or Knowledge Sharing.
02 - Research	Store external info and analysis.	Article summaries, experiments, docs references.	Connect to related Develop notes.
03 - Knowledge Sharing	Write polished, shareable outputs.	Internal guides, training notes, blog drafts.	Archive when published or outdated.
05 - Other	Keep miscellaneous notes.	Admin info, lists, random ideas.	Archive or clean up periodically.
96 - Images	Store visual assets.	Diagrams, screenshots, architecture sketches.	Embed into any note using `![[image.png]]`.
97 - Indexes	Central navigation and overview.	MOCs, topic indexes, dashboards.	Continuously updated as knowledge grows.
98 - Archives	Store finished or outdated notes.	Completed drafts, old guides, deprecated docs.	None — reference when needed.
99 - Templates	Define reusable note structures.	Templates for daily, books, courses, README, MOC.	Used by Templater to create new notes.

Example Flow

1️⃣ You start your day → create a new note in 00 - Drafts
2️⃣ You refine an idea → move to 01 - Develop
3️⃣ You read new docs or articles → save to 02 - Research
4️⃣ You write an internal guide → store in 03 - Knowledge Sharing
5️⃣ Once published → move to 98 - Archives
6️⃣ Update links in your 97 - Indexes
7️⃣ Use 99 - Templates to stay consistent

Metadata

Metadata lives at the top of your note, between --- lines.
It gives structure to your knowledge, so you can filter, sort, and connect notes automatically.

Example:

---
type: concept
status: draft
project: gitlab-pipelines
created: 2025-10-14
updated: 2025-10-15
tags: [gitlab, cicd, optimization]
related:
  - [[Docker BuildKit caching performance]]
  - [[Improving GitLab Pipeline Performance]]
---

Why it matters?

Makes notes queryable (type: concept, status: draft).
Tracks lifecycle (created, updated, status).
Enables custom dashboards (e.g., list all drafts or notes in a project).
Creates semantic connections (through related).

Backlinks

Backlinks are connections between notes. This is the heart of the Zettelkasten method.
When you link [[Another Note]], Obsidian automatically creates a backlink in that note.

Example in context:

## Related
- [[Docker BuildKit caching performance]]
- [[GitLab CI caching explained]]

Obsidian’s Backlinks panel then shows where this note is referenced — helping you discover hidden connections.

💡 Why backlinks matter

Mimic the way your brain links ideas.
Create a web of knowledge instead of isolated documents.
Help resurface forgotten notes when writing or researching.

How They Work Together

Feature	Function	Example
Metadata	Structured attributes for automation and queries	`type: concept`, `project: gcp-training`
Tags	Quick topic grouping and filtering	`#gcp #terraform #security`
Backlinks	Semantic connections between ideas	`[[Terraform state management]]`

Together they allow you to:

Build topic-based dashboards (via tags)
Track note lifecycle (via metadata)
Discover hidden relationships (via backlinks)
Navigate your “DevOps brain” effortlessly

My Templates

In a world where DevOps work involves managing endless streams of configurations, scripts, ideas, and documentation, Obsidian offers more than just a note-taking tool. It becomes a personal knowledge system.
By combining Obsidian’s flexibility with the Zettelkasten method, Developers and DevOps engineers can transform scattered information into a living network of insights.

Here you can find my Obsidian Vault template described in this article:
https://github.com/pkorsch/obsidian-vault-template

Feel free to use it and improve it according to your needs. The best way to write and keep your notes is "YOUR WAY!"

OS Login Issue on GCP: “This region is not supported by the OS Login Sign API at this time.”

Pepe — Mon, 05 May 2025 12:17:17 +0000

If you’re working with Google Cloud Platform (GCP) and using OS Login to manage SSH access to your VM instances, you might have come across the following error when trying to connect via gcloud:

ERROR: (gcloud.beta.compute.ssh) INVALID_ARGUMENT: This region is not supported by the OS Login Sign API at this time.

This error can be confusing, especially if OS Login worked just fine for a year and suddenly stops working. In this blog post, I’ll explain what this error means, why it happens, and how you can resolve or work around it.

What is OS Login?

OS Login allows you to manage SSH keys using IAM identities rather than manually placing public keys in ~/.ssh/authorized_keys. It simplifies access management, improves security, and provides centralized control.

Why does this error occur?

This error occurs because some GCP regions or zones currently do not support the OS Login Sign API —specifically, the part of the API that generates signed SSH login tickets for authentication.

You may encounter this error when using a command like:

gcloud beta compute ssh INSTANCE_NAME --zone=REGION-ZONE

If the specified region doesn’t support the OS Login Sign API, gcloud fails to generate a valid login ticket and throws this error.

How can you fix or work around it?

1. Use a supported region

If possible, deploy your VM instance in a region where OS Login Sign API is fully supported. Common supported regions include:

us-central1
europe-west1
asia-east1

You can check the limitations of the os-login here.

2. Disable OS Login for the instance or project

If you must use a region where the API isn’t supported, you can temporarily disable OS Login and revert to traditional SSH access:

Disable OS Login at the instance level:

gcloud compute instances add-metadata INSTANCE_NAME \
  --metadata enable-oslogin=FALSE \
  --zone=REGION-ZONE

Add an SSH key manually:

gcloud compute instances add-metadata INSTANCE_NAME \
  --metadata "ssh-keys=USER:$(cat ~/.ssh/id_rsa.pub)" \
  --zone=REGION-ZONE

3. Add key to your os-login profile and use regular SSH

If you are locked to a specific region, and for some reason you cannot or won't disable os-login. Here is my workaround:

Add an SSH key to your os-login profile:

gcloud compute os-login ssh-keys add --key-file=~/.ssh/id_rsa.pub

Get your SSH username aka service account id:

gcloud compute os-login describe-profile --format=json | jq -r '.posixAccounts[].username'

Use regular SSH with ssh key to your VM instance:

ssh -i ~/.ssh/id_rsa.pub YOUR_SA_ACCOUNT_ID@YOUR_VM_IP

4. Wait for support expansion

If OS Login is a key part of your security policy but the region you need doesn’t support it yet, consider monitoring GCP’s roadmap or contacting support to determine if and when the feature will become available there.

Conclusion

While the error This region is not supported by the OS Login Sign API might look like a bug, it’s actually a limitation in specific GCP regions. Honestly, I don't like it when there is no standard behaviour, and a basic feature is not working everywhere.

"The devil is in details."

Troubleshooting Self Managed Node Groups in Terraform EKS

Pepe — Tue, 04 Mar 2025 07:07:36 +0000

When deploying an EKS cluster with self managed node groups using the Terraform AWS EKS module, a common issue arises where the nodes simply aren’t created. In one case study, the root cause was identified as Terraform attempting to locate a custom AMI in an incorrect catalog. The solution was straightforward: explicitly specify the ami_type parameter.

Overview of the Issue

The Terraform configuration was designed to set up an EKS cluster complete with self managed node groups. However, despite proper settings for VPCs, subnets, and IAM roles, the node groups were never created. The investigation revealed that Terraform was mistakenly searching for a custom AMI in a catalog that didn’t contain the desired image.

Diagnosing the Problem

I was trying to build cluster nodes with AMI based on Amazon Linux 2023. However, AWS Console error message:

Launch template nodegroup-launch-template-123456 should not specify an instance profile. The noderole in your request will be used to construct an instance profile.

and the terraform plan output looks a little bit strange.

Acquiring state lock. This may take a few moments...
module.eks.data.aws_partition.current[0]: Reading...
module.eks.module.kms.data.aws_partition.current[0]: Reading...
module.eks.data.aws_caller_identity.current[0]: Reading...
module.eks.data.aws_iam_policy_document.assume_role_policy[0]: Reading...
data.aws_caller_identity.current: Reading...
module.eks.module.kms.data.aws_caller_identity.current[0]: Reading...
module.eks.module.kms.data.aws_partition.current[0]: Read complete after 0s [id=aws]
module.eks.data.aws_partition.current[0]: Read complete after 0s [id=aws]
module.eks.data.aws_iam_policy_document.assume_role_policy[0]: Read complete after 0s [id=xxxxxx]
module.eks.data.aws_caller_identity.current[0]: Read complete after 0s [id=xxxxxx]
module.eks.data.aws_iam_policy_document.custom[0]: Reading...
module.eks.data.aws_iam_session_context.current[0]: Reading...
data.aws_caller_identity.current: Read complete after 0s [id=xxxxxx]
module.eks.data.aws_iam_policy_document.custom[0]: Read complete after 0s [id=xxxxxx]
module.eks.module.kms.data.aws_caller_identity.current[0]: Read complete after 0s [id=xxxxxx]
module.eks.module.self_managed_node_group["custom_nodegroup "].data.aws_partition.current: Reading...
module.eks.module.self_managed_node_group["custom_nodegroup "].data.aws_partition.current: Read complete after 0s [id=aws]
module.eks.module.self_managed_node_group["custom_nodegroup "].data.aws_caller_identity.current: Reading...
module.eks.module.self_managed_node_group["custom_nodegroup "].data.aws_iam_policy_document.assume_role_policy[0]: Reading...
module.eks.module.self_managed_node_group["custom_nodegroup "].data.aws_caller_identity.current: Read complete after 0s [id=xxxxxx]
module.eks.module.self_managed_node_group["custom_nodegroup "].data.aws_iam_policy_document.assume_role_policy[0]: Read complete after 0s [id=xxxxxx]
module.eks.data.aws_iam_session_context.current[0]: Read complete after 0s [id=arn:aws:sts::xxxxxx:assumed-role/eks-user/eks-1234-123456]
module.eks.module.self_managed_node_group["custom_nodegroup "].data.aws_ssm_parameter.ami[0]: Reading...
module.eks.module.self_managed_node_group["custom_nodegroup "].data.aws_ssm_parameter.ami[0]: Read complete after 0s [id=/aws/service/eks/optimized-ami/1.32/amazon-linux-2/recommended/image_id]
Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:

It was trying to locate AMI in the wrong catalog.

[id=/aws/service/eks/optimized-ami/1.32/amazon-linux-2/recommended/image_id]

According to terraform EKS module documentation here, you should use values from official AWS documentation here.

amiType
If the node group was deployed using a launch template with a custom AMI, then this is CUSTOM. For node groups that weren't deployed using a launch template, this is the AMI type that was specified in the node group configuration.

Type: String

Valid Values:

AL2_x86_64 | AL2_x86_64_GPU | AL2_ARM_64 | CUSTOM | BOTTLEROCKET_ARM_64 | BOTTLEROCKET_x86_64 | BOTTLEROCKET_ARM_64_NVIDIA | BOTTLEROCKET_x86_64_NVIDIA | WINDOWS_CORE_2019_x86_64 | WINDOWS_FULL_2019_x86_64 | WINDOWS_CORE_2022_x86_64 | WINDOWS_FULL_2022_x86_64 | AL2023_x86_64_STANDARD | AL2023_ARM_64_STANDARD | AL2023_x86_64_NEURON | AL2023_x86_64_NVIDIA

From the documentation, we have to use CUSTOM value, but it is not true. There is a github issue: #3094. Terraform is not accepting the CUSTOM value. Instead of CUSTOM it has to be value, which the image is based on. In my example, it is AL2023_x86_64_STANDARD.

Fixed version:

variable "ami_type" {
  description = "type of AMI to be used"
  type        = string
  default     = "AL2023_x86_64_STANDARD"
}

  self_managed_node_groups = {
    custom_nodegroup = {
      name            = local.cluster_name
      use_name_prefix = false

      subnet_ids                    = var.subnet_ids
      additional_security_group_ids = [aws_security_group.eks_sg.id]

      instance_type        = var.instance_type
      ami_type             = var.ami_type
      ami_id               = var.ami_id
      key_name             = var.key_name
      asg_min_size         = 1
      asg_desired_capacity = 1
      asg_max_size         = 2

      launch_template = {
        create_launch_template = true
        name                   = "nodegroup-launch-template"
        use_name_prefix        = true
        description            = "Self managed node group launch template"
      }

      enable_monitoring = true

      create_iam_role_policy   = true
      iam_role_name            = local.cluster_name
      iam_role_use_name_prefix = false
      iam_role_description     = "Self managed node group role"

      create_security_group          = true
      security_group_name            = local.cluster_name
      security_group_use_name_prefix = false
      revoke_rules_on_delete         = true
      create_before_destroy          = true
      security_group_description     = "Self managed node group security group"

      security_group_tags = {
        Purpose = "Protector of the kubelet"
      }
    }
  }

  tags = local.tags
}

Conclusion

In this case, the failure to create self managed node groups was not due to security groups or IAM roles alone—it was the misdirection of Terraform attempting to retrieve a custom AMI from the wrong catalog. The simple act of specifying the ami_type parameter resolved the issue by guiding Terraform to the correct image repository.

By ensuring consistent configuration across all parameters and paying close attention to details like AMI selection, you can avoid similar pitfalls and ensure a successful EKS cluster deployment using Terraform.

MongoDB Performance Indexes

Pepe — Tue, 25 Feb 2025 13:45:12 +0000

NOTE: This is my older article moved from medium.com

Query Targeting: Scanned Objects / Returned has gone above 1000

Performance issues can be sometimes a pain in the a**, but I would like to show you how to improve MongoDB query performance on the practical example. I hope it will help you to understand query optimization by creating the proper index.

Prepare the testing environment

I am going to use MongoDB Atlas (simple and fast cluster deployment) and this dataset from kaggle.com. It doesn't matter if you will use Atlas or a self-managed MongoDB cluster. Mostly I will use the CLI tool mongoimport, mongo-shell, or MongoDB Compass. All necessary tools for this exercise, you can find on the MongoDB download page. Mongoimport binary is part of the database tools package. I am not going to describe how to install and use those tools in detail, because there is very good documentation for that.
I have created a dedicated M10 cluster, database user called user, and allow network communication from my IP or 0.0.0.0/0 - not so secure, but it is not a production environment, right? :-)

OK! Now when we have our cluster, let's import the huge dataset. We have two options on how to do it. GUI and CLI. I will describe both.

GUI - MongoDB Compass

This is pretty straightforward. Compass is a very intuitive tool. Firstly we have to create an empty database and collection and then we are able to add data. We will select our dataset file, mark CSV as the input file type and that's it!

CLI - mongoimport

If you are a command line lover, then this is the right option for you. Use mongoimport binary with those parameters:

uri = connection string in quotes
authenticationDatabase = database where your user was created, most common database is admin
db = database name for your import
collection = collection name for your import
drop = delete collection if it is already existing in the cluster, before import starts
type = type of your import file
headerline = the first line will be used as document schema/field names
file = import file

$ mongoimport --uri "mongodb+srv://user:<your-secure-password>@cluster1.xy66x.mongodb.net" --authenticationDatabase admin --db github --collection pull_requests --drop --type=csv --headerline --file=ghtorrent-2019–05–20.csv

Check the collection

We have our github.pull_requests collection in a test environment. Now we can look around with mongoshell for a bit.

$ mongosh "mongodb+srv://user:<your-secure-password>@cluster1.xy66x.mongodb.net"
> show dbs
admin 256.00 KiB
config 268.00 KiB
github 8.51 GiB
local 777.77 MiB
> use github
switched to db github
> show collections
pull_requests

> db.pull_requests.findOne()
{
  _id: ObjectId("632080afacad4db4ac0a54f2"),
  actor_login: 'EleisonC',
  actor_id: 37538393,
  comment_id: 197673683,
  comment: 'That has been rectified.',
  repo: 'openmrs-ocl-client',
  author_login: 'Karuhanga',
  author_id: 32985500,
  pr_id: 41412787,
  c_id: 1051024412,
  commit_date: ISODate("2019-05-20T06:24:09.000Z")
}
> db.pull_requests.find().count()
81994070
> db.pull_requests.getIndexes()
[ { v: 2, key: { _id: 1 }, name: '_id_' } ]
> db.pull_requests.stats()
{
  ns: 'github.pull_requests',
  size: Long("26118333454"),
  count: 81994070,
  avgObjSize: 318,
  storageSize: Long("15080230912"),
  freeStorageSize: 40480768,
  capped: false,
  wiredTiger: {
    metadata: { formatVersion: 1 },
    creationString: 'access_pattern_hint=none,allocation_size=4KB,app_metadata=(formatVersion=1),assert=(commit_timestamp=none,durable_timestamp=none,read_timestamp=none,write_timestamp=off),block_allocation=best,block_compressor=snappy,cache_resident=false,checksum=on,colgroups=,collator=,columns=,dictionary=0,encryption=(keyid=,name=),exclusive=false,extractor=,format=btree,huffman_key=,huffman_value=,ignore_in_memory_cache_size=false,immutable=false,import=(enabled=false,file_metadata=,repair=false),internal_item_max=0,internal_key_max=0,internal_key_truncate=true,internal_page_max=4KB,key_format=q,key_gap=10,leaf_item_max=0,leaf_key_max=0,leaf_page_max=32KB,leaf_value_max=64MB,log=(enabled=false),lsm=(auto_throttle=true,bloom=true,bloom_bit_count=16,bloom_config=,bloom_hash_count=8,bloom_oldest=false,chunk_count_limit=0,chunk_max=5GB,chunk_size=10MB,merge_custom=(prefix=,start_generation=0,suffix=),merge_max=15,merge_min=0),memory_page_image_max=0,memory_page_max=10m,os_cache_dirty_max=0,os_cache_max=0,prefix_compression=false,prefix_compression_min=4,readonly=false,source=,split_deepen_min_child=0,split_deepen_per_child=0,split_pct=90,tiered_object=false,tiered_storage=(auth_token=,bucket=,bucket_prefix=,cache_directory=,local_retention=300,name=,object_target_size=0),type=file,value_format=u,verbose=[],write_timestamp_usage=none',
    type: 'file',
    uri: 'statistics:table:collection-2-4287955632398155746',
    LSM: {
      'bloom filter false positives': 0,
      'bloom filter hits': 0,
      'bloom filter misses': 0,
      'bloom filter pages evicted from cache': 0,
      'bloom filter pages read into cache': 0,
      'bloom filters in the LSM tree': 0,
      'chunks in the LSM tree': 0,
      'highest merge generation in the LSM tree': 0,
      'queries that could have benefited from a Bloom filter that did not exist': 0,
      'sleep for LSM checkpoint throttle': 0,
      'sleep for LSM merge throttle': 0,
      'total size of bloom filters': 0
    },
    'block-manager': {
      'allocations requiring file extension': 504864,
      'blocks allocated': 563512,
      'blocks freed': 52250,
      'checkpoint size': Long("15039696896"),
      'file allocation unit size': 4096,
      'file bytes available for reuse': 40480768,
      'file magic number': 120897,
      'file major version number': 1,
      'file size in bytes': Long("15080230912"),
      'minor version number': 0
    },
    btree: {
      'btree checkpoint generation': 1550,
      'btree clean tree checkpoint expiration time': Long("9223372036854775807"),
      'btree compact pages reviewed': 0,
      'btree compact pages rewritten': 0,
      'btree compact pages skipped': 0,
      'btree skipped by compaction as process would not reduce size': 0,
      'column-store fixed-size leaf pages': 0,
      'column-store internal pages': 0,
      'column-store variable-size RLE encoded values': 0,
      'column-store variable-size deleted values': 0,
      'column-store variable-size leaf pages': 0,
      'fixed-record size': 0,
      'maximum internal page size': 4096,
      'maximum leaf page key size': 2867,
      'maximum leaf page size': 32768,
      'maximum leaf page value size': 67108864,
      'maximum tree depth': 4,
      'number of key/value pairs': 0,
      'overflow pages': 0,
      'row-store empty values': 0,
      'row-store internal pages': 0,
      'row-store leaf pages': 0
    },
    cache: {
      'bytes currently in the cache': 282058257,
      'bytes dirty in the cache cumulative': 1895528663,
      'bytes read into cache': Long("359307024373"),
      'bytes written from cache': Long("29670018281"),
      'checkpoint blocked page eviction': 5,
      'checkpoint of history store file blocked non-history store page eviction': 0,
      'data source pages selected for eviction unable to be evicted': 11185,
      'eviction gave up due to detecting an out of order on disk value behind the last update on the chain': 0,
      'eviction gave up due to detecting an out of order tombstone ahead of the selected on disk update': 0,
      'eviction gave up due to detecting an out of order tombstone ahead of the selected on disk update after validating the update chain': 0,
      'eviction gave up due to detecting out of order timestamps on the update chain after the selected on disk update':
 0,
      'eviction walk passes of a file': 91631,
      'eviction walk target pages histogram - 0-9': 9364,
      'eviction walk target pages histogram - 10-31': 12894,
      'eviction walk target pages histogram - 128 and higher': 0,
      'eviction walk target pages histogram - 32-63': 16845,
      'eviction walk target pages histogram - 64-128': 52528,
      'eviction walk target pages reduced due to history store cache pressure': 0,
      'eviction walks abandoned': 6316,
      'eviction walks gave up because they restarted their walk twice': 20244,
      'eviction walks gave up because they saw too many pages and found no candidates': 5531,
      'eviction walks gave up because they saw too many pages and found too few candidates': 1050,
      'eviction walks reached end of tree': 54382,
      'eviction walks restarted': 0,
      'eviction walks started from root of tree': 33145,
      'eviction walks started from saved location in tree': 58486,
      'hazard pointer blocked page eviction': 1085,
      'history store table insert calls': 0,
      'history store table insert calls that returned restart': 0,
      'history store table out-of-order resolved updates that lose their durable timestamp': 0,
      'history store table out-of-order updates that were fixed up by reinserting with the fixed timestamp': 0,
      'history store table reads': 0,
      'history store table reads missed': 0,
      'history store table reads requiring squashed modifies': 0,
      'history store table truncation by rollback to stable to remove an unstable update': 0,
      'history store table truncation by rollback to stable to remove an update': 0,
      'history store table truncation to remove an update': 0,
      'history store table truncation to remove range of updates due to key being removed from the data page during reconciliation': 0,
      'history store table truncation to remove range of updates due to out-of-order timestamp update on data page': 0,
      'history store table writes requiring squashed modifies': 0,
      'in-memory page passed criteria to be split': 977,
      'in-memory page splits': 489,
      'internal pages evicted': 36253,
      'internal pages split during eviction': 50,
      'leaf pages split during eviction': 6354,
      'modified pages evicted': 11379,
      'overflow pages read into cache': 0,
      'page split during eviction deepened the tree': 1,
      'page written requiring history store records': 0,
      'pages read into cache': 6486249,
      'pages read into cache after truncate': 1,
      'pages read into cache after truncate in prepare state': 0,
      'pages requested from the cache': 181005752,
      'pages seen by eviction walk': 47694902,
      'pages written from cache': 562692,
      'pages written requiring in-memory restoration': 6414,
      'the number of times full update inserted to history store': 0,
      'the number of times reverse modify inserted to history store': 0,
      'tracked dirty bytes in the cache': 0,
      'unmodified pages evicted': 6983465
    },
    cache_walk: {
      'Average difference between current eviction generation when the page was last considered': 0,
      'Average on-disk page image size seen': 0,
      'Average time in cache for pages that have been visited by the eviction server': 0,
      'Average time in cache for pages that have not been visited by the eviction server': 0,
      'Clean pages currently in cache': 0,
      'Current eviction generation': 0,
      'Dirty pages currently in cache': 0,
      'Entries in the root page': 0,
      'Internal pages currently in cache': 0,
      'Leaf pages currently in cache': 0,
      'Maximum difference between current eviction generation when the page was last considered': 0,
      'Maximum page size seen': 0,
      'Minimum on-disk page image size seen': 0,
      'Number of pages never visited by eviction server': 0,
      'On-disk page image sizes smaller than a single allocation unit': 0,
      'Pages created in memory and never written': 0,
      'Pages currently queued for eviction': 0,
      'Pages that could not be queued for eviction': 0,
      'Refs skipped during cache traversal': 0,
      'Size of the root page': 0,
      'Total number of pages currently in cache': 0
    },
    'checkpoint-cleanup': {
      'pages added for eviction': 0,
      'pages removed': 0,
      'pages skipped during tree walk': 2853123,
      'pages visited': 3876644
    },
    compression: {
      'compressed page maximum internal page size prior to compression': 4096,
      'compressed page maximum leaf page size prior to compression ': 127796,
      'compressed pages read': 6452136,
      'compressed pages written': 528035,
      'number of blocks with compress ratio greater than 64': 0,
      'number of blocks with compress ratio smaller than 16': 0,
      'number of blocks with compress ratio smaller than 2': 6447973,
      'number of blocks with compress ratio smaller than 32': 0,
      'number of blocks with compress ratio smaller than 4': 4163,
      'number of blocks with compress ratio smaller than 64': 0,
      'number of blocks with compress ratio smaller than 8': 0,
      'page written failed to compress': 0,
      'page written was too small to compress': 34657
    },
    cursor: {
      'Total number of entries skipped by cursor next calls': 0,
      'Total number of entries skipped by cursor prev calls': 0,
      'Total number of entries skipped to position the history store cursor': 0,
      'Total number of times a search near has exited due to prefix config': 0,
      'bulk loaded cursor insert calls': 0,
      'cache cursors reuse count': 246011,
      'close calls that result in cache': 246014,
      'create calls': 32,
      'cursor next calls that skip due to a globally visible history store tombstone': 0,
      'cursor next calls that skip greater than or equal to 100 entries': 0,
      'cursor next calls that skip less than 100 entries': 105,
      'cursor prev calls that skip due to a globally visible history store tombstone': 0,
      'cursor prev calls that skip greater than or equal to 100 entries': 0,
      'cursor prev calls that skip less than 100 entries': 1,
      'insert calls': 81994070,
      'insert key and value bytes': Long("26511435968"),
      modify: 0,
      'modify key and value bytes affected': 0,
      'modify value bytes modified': 0,
      'next calls': 105,
      'open cursor count': 4,
      'operation restarted': 0,
      'prev calls': 1,
      'remove calls': 0,
      'remove key bytes removed': 0,
      'reserve calls': 0,
      'reset calls': 1762637,
      'search calls': 1226093920,
      'search history store calls': 0,
      'search near calls': 1,
      'truncate calls': 0,
      'update calls': 0,
      'update key and value bytes': 0,
      'update value size change': 0
    },
    reconciliation: {
      'approximate byte size of timestamps in pages written': 1355280480,
      'approximate byte size of transaction IDs in pages written': 677640240,
      'dictionary matches': 0,
      'fast-path pages deleted': 0,
      'internal page key bytes discarded using suffix compression': 535709,
      'internal page multi-block writes': 811,
      'leaf page key bytes discarded using prefix compression': 0,
      'leaf page multi-block writes': 6732,
      'leaf-page overflow keys': 0,
      'maximum blocks required for a page': 31,
      'overflow values written': 0,
      'page checksum matches': 0,
      'page reconciliation calls': 12985,
      'page reconciliation calls for eviction': 9937,
      'pages deleted': 0,
      'pages written including an aggregated newest start durable timestamp ': 33861,
      'pages written including an aggregated newest stop durable timestamp ': 0,
      'pages written including an aggregated newest stop timestamp ': 0,
      'pages written including an aggregated newest stop transaction ID': 0,
      'pages written including an aggregated newest transaction ID ': 33861,
      'pages written including an aggregated oldest start timestamp ': 33861,
      'pages written including an aggregated prepare': 0,
      'pages written including at least one prepare': 0,
      'pages written including at least one start durable timestamp': 527834,
      'pages written including at least one start timestamp': 527834,
      'pages written including at least one start transaction ID': 527834,
      'pages written including at least one stop durable timestamp': 0,
      'pages written including at least one stop timestamp': 0,
      'pages written including at least one stop transaction ID': 0,
      'records written including a prepare': 0,
      'records written including a start durable timestamp': 84705030,
      'records written including a start timestamp': 84705030,
      'records written including a start transaction ID': 84705030,
      'records written including a stop durable timestamp': 0,
      'records written including a stop timestamp': 0,
      'records written including a stop transaction ID': 0
    },
    session: {
      'object compaction': 0,
      'tiered operations dequeued and processed': 0,
      'tiered operations scheduled': 0,
      'tiered storage local retention time (secs)': 0
    },
    transaction: {
      'race to read prepared update retry': 0,
      'rollback to stable history store records with stop timestamps older than newer records': 0,
      'rollback to stable inconsistent checkpoint': 0,
      'rollback to stable keys removed': 0,
      'rollback to stable keys restored': 0,
      'rollback to stable restored tombstones from history store': 0,
      'rollback to stable restored updates from history store': 0,
      'rollback to stable skipping delete rle': 0,
      'rollback to stable skipping stable rle': 0,
      'rollback to stable sweeping history store keys': 0,
      'rollback to stable updates removed from history store': 0,
      'transaction checkpoints due to obsolete pages': 0,
      'update conflicts': 0
    }
  },
  nindexes: 1,
  indexBuilds: [],
  totalIndexSize: Long("2582536192"),
  totalSize: Long("17662767104"),
  indexSizes: { _id_: Long("2582536192") },
  scaleFactor: 1,
  ok: 1,
  '$clusterTime': {
    clusterTime: Timestamp({ t: 1663158374, i: 22 }),
    signature: {
      hash: Binary(Buffer.from("6d46acf48f3154a0144772ca4187aeed7d480715", "hex"), 0),
      keyId: Long("7142807608573820932")
    }
  },
  operationTime: Timestamp({ t: 1663158374, i: 22 })
}
> db.getCollectionInfos()
[
  {
    name: 'pull_requests',
    type: 'collection',
    options: {},
    info: {
      readOnly: false,
      uuid: UUID("0081a40b-8a0c-47a0-81f8-10aaabbb0da5")
    },
    idIndex: { v: 2, key: { _id: 1 }, name: '_id_' }
  }
]

Great! Now we have an overview of our collection. It consists of 81994070 documents and has only one index _id. Ideal case for triggering a very slow query :-D

Slow Down!!!

Hmm, let's create a complex query to see what will happened. We are going to list all documents where actor_login is EleisonC, author_id is 32985500, and commit_date older than 2019–05–20 16:56:17. In the end we are going to sort the output by _id, where we are sure that has index already.

> db.pull_requests.find({"$and":[{"actor_login":"EleisonC"},{"author_id": 32985500},{"commit_date":{$lte:ISODate("2019-05-20T16:56:17.000Z")}}]}).sort({_id: 1})

Oooooh seems pretty slow, about 4 minutes to get results. In the meantime, we could check it with currentOp in a different session (needed role dbAdmin).

> db.currentOp( {"$ownOps": true } )
{
  inprog: [
    {
      type: 'op',
      host: 'atlas-u3sxt5-shard-00-01.xy66x.mongodb.net:27017',
      desc: 'conn13447',
      connectionId: 13447,
      client: '192.168.1.195:24750',
      appName: 'mongosh 1.5.4',
      clientMetadata: {
        driver: { name: 'nodejs|mongosh', version: '4.8.1' },
        os: {
          type: 'Linux',
          name: 'linux',
          architecture: 'x64',
          version: '5.10.16.3-microsoft-standard-WSL2'
        },
        platform: 'Node.js v16.16.0, LE (unified)',
        version: '4.8.1|1.5.4',
        application: { name: 'mongosh 1.5.4' }
      },
      active: true,
      currentOpTime: '2022-09-14T13:03:38.070+00:00',
      effectiveUsers: [ { user: 'user', db: 'admin' } ],
      threaded: true,
      opid: 6766360,
      lsid: {
        id: UUID("9cd19c2a-7e03-447f-b131-69c507270142"),
        uid: Binary(Buffer.from("78e672a9f1df895a5fbe23c35a0004466ccd167529f1405d95628de7416ff5fa", "hex"), 0)
      },
      secs_running: Long("0"),
      microsecs_running: Long("120"),
      op: 'command',
      ns: 'admin.$cmd.aggregate',
      command: {
        currentOp: 1,
        '$ownOps': true,
        lsid: { id: UUID("9cd19c2a-7e03-447f-b131-69c507270142") },
        '$clusterTime': {
          clusterTime: Timestamp({ t: 1663160603, i: 1 }),
          signature: {
            hash: Binary(Buffer.from("7f6f11951f921b79bdf4e8bd2e702cb238b17e00", "hex"), 0),
            keyId: Long("7142807608573820932")
          }
        },
        '$db': 'admin'
      },
      numYields: 0,
      locks: {},
      waitingForLock: false,
      lockStats: {},
      waitingForFlowControl: false,
      flowControlStats: {}
    },
    {
      type: 'op',
      host: 'atlas-u3sxt5-shard-00-01.xy66x.mongodb.net:27017',
      desc: 'conn12942',
      connectionId: 12942,
      client: '192.168.1.195:24389',
      appName: 'mongosh 1.5.4',
      clientMetadata: {
        driver: { name: 'nodejs|mongosh', version: '4.8.1' },
        os: {
          type: 'Linux',
          name: 'linux',
          architecture: 'x64',
          version: '5.10.16.3-microsoft-standard-WSL2'
        },
        platform: 'Node.js v16.16.0, LE (unified)',
        version: '4.8.1|1.5.4',
        application: { name: 'mongosh 1.5.4' }
      },
      active: true,
      currentOpTime: '2022-09-14T13:03:38.070+00:00',
      effectiveUsers: [ { user: 'user', db: 'admin' } ],
      threaded: true,
      opid: 6765691,
      lsid: {
        id: UUID("3d3c092e-3187-42f2-a0d6-ed30d61dc571"),
        uid: Binary(Buffer.from("78e672a9f1df895a5fbe23c35a0004466ccd167529f1405d95628de7416ff5fa", "hex"), 0)
      },
      secs_running: Long("19"),
      microsecs_running: Long("19815843"),
      op: 'query',
      ns: 'github.pull_requests',
      command: {
        find: 'pull_requests',
        filter: {
          '$and': [
            { actor_login: 'EleisonC' },
            { author_id: 32985500 },
            { commit_date: [Object] }
          ]
        },
        sort: { _id: 1 },
        lsid: { id: UUID("3d3c092e-3187-42f2-a0d6-ed30d61dc571") },
        '$clusterTime': {
          clusterTime: Timestamp({ t: 1663159643, i: 1 }),
          signature: {
            hash: Binary(Buffer.from("acde94b5658281a5b9eea21b0ffaae6f0653465f", "hex"), 0),
            keyId: Long("7142807608573820932")
          }
        },
        '$db': 'github'
      },
      planSummary: 'IXSCAN { _id: 1 }',
      numYields: 9194,
      locks: { FeatureCompatibilityVersion: 'r', Global: 'r' },
      waitingForLock: false,
      lockStats: {
        FeatureCompatibilityVersion: { acquireCount: { r: Long("9195") } },
        Global: { acquireCount: { r: Long("9195") } },
        Mutex: { acquireCount: { r: Long("1") } }
      },
      waitingForFlowControl: false,
      flowControlStats: {}
    }
  ],
  ok: 1,
  '$clusterTime': {
    clusterTime: Timestamp({ t: 1663160613, i: 1 }),
    signature: {
      hash: Binary(Buffer.from("bd234febc72dd8fa57796adbfd7fde21e6aa33ab", "hex"), 0),
      keyId: Long("7142807608573820932")
    }
  },
  operationTime: Timestamp({ t: 1663160613, i: 1 })
}

If you are not able to catch the query with db command, check logfiles. There will be all the necessary information you need.

mongodb.log:

{"t":{"$date":"2022-09-14T13:13:33.361+00:00"},"s":"I",  "c":"COMMAND",  "id":51803,   "ctx":"conn12942","msg":"Slow query","attr":{"type":"command","ns":"github.pull_requests","appName":"mongosh 1.5.4","command":{"find":"pull_requests","filter":{"$and":[{"actor_login":"EleisonC"},{"author_id":32985500},{"commit_date":{"$lte":{"$date":"2019-05-20T16:56:17.000Z"}}}]},"sort":{"_id":1},"lsid":{"id":{"$uuid":"3d3c092e-3187-42f2-a0d6-ed30d61dc571"}},"$clusterTime":{"clusterTime":{"$timestamp":{"t":1663160774,"i":23}},"signature":{"hash":{"$binary":{"base64":"kHRt5Xd0iUjWwslfWxjvuRcIxUA=","subType":"0"}},"keyId":7142807608573820932}},"$db":"github"},"planSummary":"IXSCAN { _id: 1 }","keysExamined":81994070,"docsExamined":81994070,"cursorExhausted":true,"numYields":82007,"nreturned":74,"queryHash":"33DBB573","planCacheKey":"98434520","reslen":21182,"locks":{"FeatureCompatibilityVersion":{"acquireCount":{"r":82008}},"Global":{"acquireCount":{"r":82008}},"Mutex":{"acquireCount":{"r":1}}},"readConcern":{"level":"local","provenance":"implicitDefault"},"storage":{"data":{"bytesRead":31566325989,"timeReadingMicros":54395866},"timeWaitingMicros":{"cache":8538}},"remote":"196.168.1.195:24389","protocol":"op_msg","durationMillis":181936}}

When we are checking the situation of query execution with db.currentOp() a couple of times, we can see that number of seconds is increasing, the plan is using index (IXSCAN) and numYields are increasing as well, which is not good. The same behavior is repeated every time we execute the query. That means the query is reading from disk (all the time) and it is inefficient. In logfile we can see another issue. We are scanning all documents for 74 results. That is a very bad Scanned/Returned ratio. But why? we are using the index, right? Looks like our existing index is not enough for processing such an amount of data and the query needs something more sophisticated.

How to choose the proper index?

Well, there are some rules. Ideally, the index might cover the whole query and be created by the ESR rule. Hmm okay, so how do we put four fields into one index? MongoDB supports a lot of index type and one of them is a compound index. To cover a query we have to use fields: actor_login, author_id, commit_date, _id.

> db.pull_requests.find({"$and":[{"actor_login":"EleisonC"},{"author_id": 32985500},{"commit_date":{$lte:ISODate("2019-05-20T16:56:17.000Z")}}]}).sort({_id: 1})

Fine, should I add all fields in order as they are used in the query? That's not a good solution for that. As I mentioned earlier we have to follow the ESR rule.

the ESR Rule - Equality, Sort, Range

Order of fields in compound index start with fields with exact match {"actor_login":"EleisonC"},{"author_id": 32985500} (equality), then follows fields used for sorting operation sort({_id: 1}) (sort) and last one is field with not exact match, but when something is less than $lt , greater than $gt, not equal $ne, etc. (range)

>db.pull_requests.createIndex( { "actor_login" : 1, "author_id" : 1, "_id" : 1, "commit_date" : 1} )

This index is our WINNER!!!

Here you can see the difference between the index actor_login_1_author_id_1id_1_commit_date_1 and the index actor_login_1_author_id_1_commit_date_1id_1.

> db.pull_requests.explain("allPlansExecution").find({"$and":[{"actor_login":"EleisonC"},{"author_id": 32985500},{"commit_date":{$lte:ISODate("2019-05-20T16:56:17.000Z")}}]}).sort({_id: 1})
{
  explainVersion: '1',
  queryPlanner: {
    namespace: 'github.pull_requests',
    indexFilterSet: false,
    parsedQuery: {
      '$and': [
        { actor_login: { '$eq': 'EleisonC' } },
        { author_id: { '$eq': 32985500 } },
        {
          commit_date: { '$lte': ISODate("2019-05-20T16:56:17.000Z") }
        }
      ]
    },
    maxIndexedOrSolutionsReached: false,
    maxIndexedAndSolutionsReached: false,
    maxScansToExplodeReached: false,
    winningPlan: {
      stage: 'FETCH',
      inputStage: {
        stage: 'IXSCAN',
        keyPattern: { actor_login: 1, author_id: 1, _id: 1, commit_date: 1 },
        indexName: 'actor_login_1_author_id_1__id_1_commit_date_1',
        isMultiKey: false,
        multiKeyPaths: { actor_login: [], author_id: [], _id: [], commit_date: [] },
        isUnique: false,
        isSparse: false,
        isPartial: false,
        indexVersion: 2,
        direction: 'forward',
        indexBounds: {
          actor_login: [ '["EleisonC", "EleisonC"]' ],
          author_id: [ '[32985500, 32985500]' ],
          _id: [ '[MinKey, MaxKey]' ],
          commit_date: [
            '[new Date(-9223372036854775808), new Date(1558371377000)]'
          ]
        }
      }
    },
    rejectedPlans: [
      {
        stage: 'FETCH',
        inputStage: {
          stage: 'SORT',
          sortPattern: { _id: 1 },
          memLimit: 104857600,
          type: 'default',
          inputStage: {
            stage: 'IXSCAN',
            keyPattern: { actor_login: 1, author_id: 1, commit_date: 1, _id: 1 },
            indexName: 'actor_login_1_author_id_1_commit_date_1__id_1',
            isMultiKey: false,
            multiKeyPaths: {
              actor_login: [],
              author_id: [],
              commit_date: [],
              _id: []
            },
            isUnique: false,
            isSparse: false,
            isPartial: false,
            indexVersion: 2,
            direction: 'forward',
            indexBounds: {
              actor_login: [ '["EleisonC", "EleisonC"]' ],
              author_id: [ '[32985500, 32985500]' ],
              commit_date: [
                '[new Date(-9223372036854775808), new Date(1558371377000)]'
              ],
              _id: [ '[MinKey, MaxKey]' ]
            }
          }
        }
      }
    ]
  },
  executionStats: {
    executionSuccess: true,
    nReturned: 74,
    executionTimeMillis: 1,
    totalKeysExamined: 75,
    totalDocsExamined: 74,
    executionStages: {
      stage: 'FETCH',
      nReturned: 74,
      executionTimeMillisEstimate: 0,
      works: 76,
      advanced: 74,
      needTime: 0,
      needYield: 0,
      saveState: 0,
      restoreState: 0,
      isEOF: 1,
      docsExamined: 74,
      alreadyHasObj: 0,
      inputStage: {
        stage: 'IXSCAN',
        nReturned: 74,
        executionTimeMillisEstimate: 0,
        works: 75,
        advanced: 74,
        needTime: 0,
        needYield: 0,
        saveState: 0,
        restoreState: 0,
        isEOF: 1,
        keyPattern: { actor_login: 1, author_id: 1, _id: 1, commit_date: 1 },
        indexName: 'actor_login_1_author_id_1__id_1_commit_date_1',
        isMultiKey: false,
        multiKeyPaths: { actor_login: [], author_id: [], _id: [], commit_date: [] },
        isUnique: false,
        isSparse: false,
        isPartial: false,
        indexVersion: 2,
        direction: 'forward',
        indexBounds: {
          actor_login: [ '["EleisonC", "EleisonC"]' ],
          author_id: [ '[32985500, 32985500]' ],
          _id: [ '[MinKey, MaxKey]' ],
          commit_date: [
            '[new Date(-9223372036854775808), new Date(1558371377000)]'
          ]
        },
        keysExamined: 75,
        seeks: 1,
        dupsTested: 0,
        dupsDropped: 0
      }
    },
    allPlansExecution: [
      {
        nReturned: 74,
        executionTimeMillisEstimate: 0,
        totalKeysExamined: 75,
        totalDocsExamined: 74,
        executionStages: {
          stage: 'FETCH',
          nReturned: 74,
          executionTimeMillisEstimate: 0,
          works: 75,
          advanced: 74,
          needTime: 0,
          needYield: 0,
          saveState: 0,
          restoreState: 0,
          isEOF: 1,
          docsExamined: 74,
          alreadyHasObj: 0,
          inputStage: {
            stage: 'IXSCAN',
            nReturned: 74,
            executionTimeMillisEstimate: 0,
            works: 75,
            advanced: 74,
            needTime: 0,
            needYield: 0,
            saveState: 0,
            restoreState: 0,
            isEOF: 1,
            keyPattern: { actor_login: 1, author_id: 1, _id: 1, commit_date: 1 },
            indexName: 'actor_login_1_author_id_1__id_1_commit_date_1',
            isMultiKey: false,
            multiKeyPaths: {
              actor_login: [],
              author_id: [],
              _id: [],
              commit_date: []
            },
            isUnique: false,
            isSparse: false,
            isPartial: false,
            indexVersion: 2,
            direction: 'forward',
            indexBounds: {
              actor_login: [ '["EleisonC", "EleisonC"]' ],
              author_id: [ '[32985500, 32985500]' ],
              _id: [ '[MinKey, MaxKey]' ],
              commit_date: [
                '[new Date(-9223372036854775808), new Date(1558371377000)]'
              ]
            },
            keysExamined: 75,
            seeks: 1,
            dupsTested: 0,
            dupsDropped: 0
          }
        }
      },
      {
        nReturned: 0,
        executionTimeMillisEstimate: 0,
        totalKeysExamined: 74,
        totalDocsExamined: 0,
        executionStages: {
          stage: 'FETCH',
          nReturned: 0,
          executionTimeMillisEstimate: 0,
          works: 75,
          advanced: 0,
          needTime: 75,
          needYield: 0,
          saveState: 0,
          restoreState: 0,
          isEOF: 0,
          docsExamined: 0,
          alreadyHasObj: 0,
          inputStage: {
            stage: 'SORT',
            nReturned: 0,
            executionTimeMillisEstimate: 0,
            works: 75,
            advanced: 0,
            needTime: 75,
            needYield: 0,
            saveState: 0,
            restoreState: 0,
            isEOF: 0,
            sortPattern: { _id: 1 },
            memLimit: 104857600,
            type: 'default',
            totalDataSizeSorted: 6068,
            usedDisk: false,
            inputStage: {
              stage: 'IXSCAN',
              nReturned: 74,
              executionTimeMillisEstimate: 0,
              works: 75,
              advanced: 74,
              needTime: 0,
              needYield: 0,
              saveState: 0,
              restoreState: 0,
              isEOF: 1,
              keyPattern: { actor_login: 1, author_id: 1, commit_date: 1, _id: 1 },
              indexName: 'actor_login_1_author_id_1_commit_date_1__id_1',
              isMultiKey: false,
              multiKeyPaths: {
                actor_login: [],
                author_id: [],
                commit_date: [],
                _id: []
              },
              isUnique: false,
              isSparse: false,
              isPartial: false,
              indexVersion: 2,
              direction: 'forward',
              indexBounds: {
                actor_login: [ '["EleisonC", "EleisonC"]' ],
                author_id: [ '[32985500, 32985500]' ],
                commit_date: [
                  '[new Date(-9223372036854775808), new Date(1558371377000)]'
                ],
                _id: [ '[MinKey, MaxKey]' ]
              },
              keysExamined: 74,
              seeks: 1,
              dupsTested: 0,
              dupsDropped: 0
            }
          }
        }
      }
    ]
  },
  command: {
    find: 'pull_requests',
    filter: {
      '$and': [
        { actor_login: 'EleisonC' },
        { author_id: 32985500 },
        {
          commit_date: { '$lte': ISODate("2019-05-20T16:56:17.000Z") }
        }
      ]
    },
    sort: { _id: 1 },
    '$db': 'github'
  },
  serverInfo: {
    host: 'atlas-u3sxt5-shard-00-01.xy66x.mongodb.net',
    port: 27017,
    version: '5.0.12',
    gitVersion: '79cfcdd83eb6f64e164a588d0daf9bb873328b45'
  },
  serverParameters: {
    internalQueryFacetBufferSizeBytes: 104857600,
    internalQueryFacetMaxOutputDocSizeBytes: 104857600,
    internalLookupStageIntermediateDocumentMaxSizeBytes: 104857600,
    internalDocumentSourceGroupMaxMemoryBytes: 104857600,
    internalQueryMaxBlockingSortMemoryUsageBytes: 104857600,
    internalQueryProhibitBlockingMergeOnMongoS: 0,
    internalQueryMaxAddToSetBytes: 104857600,
    internalDocumentSourceSetWindowFieldsMaxMemoryBytes: 104857600
  },
  ok: 1,
  '$clusterTime': {
    clusterTime: Timestamp({ t: 1663178353, i: 1 }),
    signature: {
      hash: Binary(Buffer.from("1834928e18dd57e09c13189fd097e110caac7d85", "hex"), 0),
      keyId: Long("7142807608573820932")
    }
  },
  operationTime: Timestamp({ t: 1663178353, i: 1 })
}

The compound indexes are good for queries that are executed very often and selected fields are stable.

This article was written as some kind of working note for me, but I hope somebody will find it useful :-)

Sources:

Performance Best Practices: Indexing

Fix Query Issues

The ESR (Equality, Sort, Range) Rule

GitLab Merge Review with Ollama [Gitlab CI/CD]

Pepe — Tue, 25 Feb 2025 13:22:11 +0000

In our last article, we created a simple script designed to review our code in GitLab. Now, we're taking the next step by integrating that very script into our CI/CD pipeline. By embedding it into GitLab's workflow, we can automatically review each merge request before it goes into production.

Sourcing Essential Environment Variables

For the review bot to interact with external services like GitLab and Jira, several environment variables must be configured. Here’s a breakdown of what you need and where to get each one:

GitLab Variables:

GITLAB_URL: The URL of your GitLab instance (e.g., https://gitlab.com or your self-hosted GitLab URL). This is usually known within your organization. For self-hosted instances, it will be your domain.

GITLAB_PRIVATE_TOKEN: A personal access token used for authenticating API calls. Go to your GitLab profile settings. Navigate to Access Tokens.Generate a token with the required scopes (e.g., api).

GITLAB_PROJECT_ID can be found in the project’s settings or as as $CI_PROJECT_ID.

GITLAB_MR_ID is available in merge request pipelines as $CI_MERGE_REQUEST_IID.

Jira Variables:

JIRA_URL: The URL of your Jira instance (e.g., https://yourcompany.atlassian.net). This is provided by your Jira administrator or can be seen in your Jira login URL.

JIRA_USER: Your Jira username or email address.

JIRA_API_TOKEN: An API token for Jira to authenticate REST API requests. Log in to your Atlassian account. Go to the API tokens section. Generate a new token.

JIRA_PROJECT_KEY: The key for the Jira project that you want to link with the merge request or issue. This can be found on the Jira project’s overview page. Can be found in user stories too.

Storing the Variables in GitLab

Once you have all these values, add them as CI/CD variables in your GitLab project:

Go to your project’s Settings > CI/CD.
Expand the Variables section.
Add each variable securely by clicking on Add Variable.

This ensures that your sensitive data is not hard-coded into the repository and is only accessible during pipeline execution.

Overview of the Pipeline Configuration

Below is an example GitLab pipeline configuration for a review bot:

variables:
  GITLAB_RUNNER: "gitlab-docker"
  LLM_MODEL: "mistral"

stages:
  - gitlab_mr_review

review:
  stage: gitlab_mr_review
  tags:
    - $RUNNER_TAG
  services:
    - name: ollama/ollama:latest
      alias: ollama
      command: ["serve"]
  image: python:3.11
  before_script:
    - curl ollama:11434/api/pull -sS -X POST -d "{\"model\":\"$LLM_MODEL\",\"stream\":false}"
  script:
    - curl -sS ollama:11434
    - pip install -r requirements.txt
    - python review-bot/main.py

What’s Happening Here?

Stages & Job:

The pipeline defines a single stage gitlab_mr_review with one job called review. This job is responsible for interacting with the review bot.
Tags & Services:
- Tags: The job uses a runner tag specified by the environment variable $RUNNER_TAG, ensuring that it runs on an appropriate runner.
- Services: A service container (ollama/ollama:latest) is spun up to run the model serving environment. It’s aliased as ollama for easy reference within the job.
Image & Scripts:
- The job uses a Python 3.11 image.
- Before Script: A curl command is executed to pull a specific model ($LLM_MODEL) from the service.
- Script: After pulling the model, it tests ollama response, then the job installs dependencies and runs the review bot located at review-bot/main.py.

Integrating the Review Bot into Different Pipelines

To allow various projects or pipelines to trigger the review bot, you can use GitLab’s trigger functionality. This trigger needs to be implemented in .gitlab-ci.yml of the project where you want to have the merge request review. Here’s how you can set it up:

trigger_review_bot:
  trigger:
    project: "group/review-bot-repo"
    branch: "main"
    strategy: depend
  variables:
    GITLAB_PROJECT_ID: $CI_PROJECT_ID
    GITLAB_MR_ID: $CI_MERGE_REQUEST_IID
  only:
    - merge_requests

In the end the response of the gitlab review bot might look like this (example for dummy project with python inch to cm converter):

I hope you found this useful and that it helps you improve your CI/CD pipeline. Happy coding!

GitLab Merge Review with Ollama [Python]

Pepe — Thu, 20 Feb 2025 17:54:24 +0000

In today's fast-paced software development world, ensuring high-quality code while keeping up with rapid release cycles can be a challenge. Code reviews play a crucial role in maintaining code quality, but they can also be time-consuming and inconsistent, especially in large teams.

What if you could automate the first layer of code review using AI? By integrating GitLab Merge Request Reviews with Ollama, you can streamline the review process, catch potential issues early, and get AI-powered suggestions for improving your code.

In this blog post, we’ll explore how to build an AI-driven Merge Request reviewer that analyzes code changes, provides structured feedback, and even incorporates context from Jira tasks. Whether you're working with large-scale projects or just looking to enhance your DevOps workflows, this solution can save time and improve overall code quality.

Let’s dive into how you can set up this AI-powered reviewer in GitLab! 🚀

Overview

This setup consists of multiple Python scripts that:

Connect to GitLab to fetch Merge Requests and their diffs.
Extract Jira ticket references from MRs and retrieve ticket details.
Use AI (Ollama) to analyze code changes and generate structured feedback.
Post review comments back to GitLab automatically.

Let’s dive into the components that make this work!

1. Connecting to GitLab

The gitlab_client.py script provides a GitLabClient class that interacts with GitLab using the python-gitlab library.

Features:

Authenticates with GitLab.
Retrieves MR details and diffs.
Adds review comments to an MR.

import logging
import gitlab
from config import GITLAB_URL, GITLAB_PRIVATE_TOKEN, GITLAB_PROJECT_ID

class GitLabClient:
    def __init__(self):
        """Gitlab login."""
        self.gl = gitlab.Gitlab(GITLAB_URL, private_token=GITLAB_PRIVATE_TOKEN)
        self.gl.auth()
        self.project = self.gl.projects.get(int(GITLAB_PROJECT_ID))
        logging.info(f"Connected to GitLab project {GITLAB_PROJECT_ID}")

    def get_merge_request(self, mr_id):
        """Fetching Merge Request information."""
        return self.project.mergerequests.get(int(mr_id))

    def get_merge_request_diff(self, mr_id):
        """Fetching diff for the Merge Request."""
        mr = self.get_merge_request(mr_id)
        changes = mr.changes()
        return "\n".join([change["diff"] for change in changes["changes"]])

    def add_comment_to_mr(self, mr_id, comment):
        """Adding a review comment to the Merge Request."""
        mr = self.get_merge_request(mr_id)
        mr.notes.create({"body": comment})
        logging.info(f"Added review comment to MR {mr_id}.")

2. Integrating with Jira

The jira_utils.py script helps extract and retrieve Jira ticket details from MR titles or descriptions.

Features:

Extracts Jira ticket IDs from MR titles/descriptions.
Fetches issue details from Jira.

import logging
import re
from jira import JIRA
from config import JIRA_URL, JIRA_USER, JIRA_API_TOKEN, JIRA_PROJECT_KEY

# Jira Authentication
jira = None
if JIRA_URL and JIRA_USER and JIRA_API_TOKEN:
    try:
        jira_options = {"server": JIRA_URL}
        jira = JIRA(options=jira_options, basic_auth=(JIRA_USER, JIRA_API_TOKEN))
        logging.info("Connected to Jira successfully.")
    except Exception as e:
        logging.error(f"Failed to connect to Jira: {e}")

# Extract Jira Ticket ID
def extract_jira_ticket(mr_title, mr_description):
    pattern = rf"{JIRA_PROJECT_KEY}-\d+"
    match = re.search(pattern, f"{mr_title} {mr_description}")
    return match.group(0) if match else None

# Get Jira Ticket Details
def get_jira_ticket_details(ticket_id):
    if not jira:
        return None
    try:
        issue = jira.issue(ticket_id)
        return f"Jira Ticket: {ticket_id}\nSummary: {issue.fields.summary}\nDescription: {issue.fields.description}"
    except Exception as e:
        logging.error(f"Failed to fetch Jira details for {ticket_id}: {e}")
        return None

3. AI-Powered Code Review

The main.py script brings everything together. It fetches the MR diff, retrieves Jira context, and uses Ollama to analyze code changes.

Features:

Validates required configuration variables.
Retrieves MR details and diffs.
Fetches Jira ticket context.
Generates structured review feedback using AI.
Posts comments back to GitLab.

import logging
import ollama
from config import LLM_MODEL, GITLAB_MR_ID
from gitlab_client import GitLabClient
from jira_utils import extract_jira_ticket, get_jira_ticket_details

def review_code(diff, jira_context=None):
    """Analyze code changes with AI."""
    prompt = f"""
    You are a senior software engineer reviewing a code change.
    Analyze the following changes and provide a structured review:
    {diff}
    {f'Jira Context: {jira_context}' if jira_context else ''}
    """
    response = ollama.chat(model=LLM_MODEL, messages=[{"role": "user", "content": prompt}])
    return response["message"]["content"] if response else "No response from AI."

def main():
    logging.info(f"Starting review for MR {GITLAB_MR_ID}.")
    gitlab_client = GitLabClient()
    diff = gitlab_client.get_merge_request_diff(GITLAB_MR_ID)
    if diff:
        jira_ticket = extract_jira_ticket(mr.title, mr.description)
        jira_context = get_jira_ticket_details(jira_ticket) if jira_ticket else None
        review = review_code(diff, jira_context)
        gitlab_client.add_comment_to_mr(GITLAB_MR_ID, review)

4. Configuration & Setup

Environment variables are loaded using config.py, making configuration flexible and secure.

import os
from dotenv import load_dotenv

load_dotenv()

LLM_MODEL = os.getenv("LLM_MODEL")
GITLAB_URL = os.getenv("GITLAB_URL")
GITLAB_PRIVATE_TOKEN = os.getenv("GITLAB_PRIVATE_TOKEN")
GITLAB_PROJECT_ID = os.getenv("GITLAB_PROJECT_ID")
JIRA_URL = os.getenv("JIRA_URL")
JIRA_USER = os.getenv("JIRA_USER")
JIRA_API_TOKEN = os.getenv("JIRA_API_TOKEN")
JIRA_PROJECT_KEY = os.getenv("JIRA_PROJECT_KEY")

Installing Dependencies

pip install -r requirements.txt

Running the Bot

python main.py

Conclusion

This AI-powered GitLab bot streamlines code reviews by automating feedback generation using AI while integrating Jira for issue tracking. With this setup, teams can save time, improve code quality, and maintain better project visibility.

In the next post, we’ll dive into integrating these Python scripts with GitLab CI/CD to automate the review process. We’ll set up a .gitlab-ci.yml pipeline to trigger AI-powered code reviews on every merge request, fetch Jira context, and post structured feedback directly in GitLab. This will streamline the development workflow, ensuring automated and consistent code quality checks.

Aliases git vs. bash

Pepe — Tue, 18 Feb 2025 09:29:15 +0000

Git alias and Bash alias serve different purposes and function differently:

Git alias

Defines shortcuts for Git commands.
Configured in Git configuration (~/.gitconfig).
Used exclusively within git commands.

Syntax:

git config --global alias.s status

Then you can use:

git s # instead of git status

Bash alias

Defines shortcuts for any shell commands, not just Git.
Configured in .bashrc or .bash_profile (or .zshrc for Zsh).
Works in the entire shell, not just in Git.

Syntax:

alias gs='git status'

Then you can use:

gs # instead of git status

Main differences:

Feature	Git alias	Bash alias
Scope	Git commands only	Any shell command
Configuration	~/.gitconfig	.bashrc, .bash_profile, .zshrc
Usage	Within git commands	Anywhere in the shell

When to use which?

Git alias is better if you want to create shortcuts only for Git commands and you use Git in multiple environments.
Bash alias is more flexible, as it allows you to combine multiple commands, modify the output or use other tools besides Git.

If you want more advanced aliases with parameters, Git aliases also support ! to execute shell commands:

git config --global alias.last '!git log -1 --stat'

But then it may be more advantageous to use a Bash alias or a script.

My favorite aliases:

#git
git config --global alias.s = 'status -sb'
git config --global alias.c = !git add -A && git commit -m
git config --global alias.p = push
git config --global alias.change = commit --amend
git config --global alias.last = 'log -1 HEAD --stat'
git config --global alias.co checkout
git config --global alias.br branch
git config --global alias.unstage 'reset HEAD --'

#bash
alias ..='cd ..'
alias ...='cd ../..' 
alias dt='date "+%F %T"'
alias df="df -Tha --total"
alias lf='ls -alF'  
alias la='ls -A'  
alias ls='ls -CF'  
alias lu='du -sh * | sort -h'  
alias lt='ls -t -1 -long'  
alias lc='find . -type f | wc -l'  
alias ld='ls -d */'  
alias ll='ls -lAh --color=auto'
alias c="clear"
alias cls="clear;ls;pwd"
alias path='echo -e ${PATH//:/\\n}'
alias h='history'
alias hg='history | grep $1' 
alias cron='crontab -l'
alias croned='crontab -e'
alias rm='rm -rfI'
alias bh="cat ~/.bash_history | grep"
alias meminfo='free -m -l -t'
alias psmem='ps auxf | sort -nr -k 4'
alias psmem10='ps auxf | sort -nr -k 4 | head -10'
alias pscpu='ps auxf | sort -nr -k 3'
alias pscpu10='ps auxf | sort -nr -k 3 | head -10'
alias cpuinfo='lscpu'
alias ports='netstat -tulanp'
alias ping='ping -c 5'
alias dfc='df -hPT | column -t'
alias mount='mount |column -t'
alias grep='grep --color=auto'
alias egrep='egrep --color=auto'
alias fgrep='fgrep --color=auto'
alias al="echo ------------Your curent aliases are:------------¡';alias"  
alias fh='find . -name ' 
alias untar='tar -zxvf $1'  
alias tar='tar -czvf $1'