DEV Community: Dmitry Romanoff

Building an Interactive S3 Bucket Manager with Bash

Dmitry Romanoff — Mon, 04 Aug 2025 13:38:39 +0000

Managing AWS S3 buckets from the command line can be tedious, especially when you need to perform multiple operations. I created an interactive bash script that simplifies S3 bucket management with a user-friendly menu interface.

Why Build This?

While AWS CLI is powerful, repeatedly typing long commands gets old fast. This script provides:

Interactive menu - No need to remember complex commands
Visual feedback - Clear success/failure indicators
Pattern matching - Find buckets using wildcards
Bulk operations - Upload/download entire folders
Safety checks - Validates bucket names and file existence

Key Features

1. Smart Bucket Pattern Matching

# Find buckets containing "backup"
*backup*

# Find buckets starting with "my"
my%

# Complex patterns
my*prod*2024

The script converts shell wildcards to regex patterns for flexible searching.

2. Visual Status Indicators

Every operation shows clear feedback:

✅ Green checkmark for success
❌ Red X for failures

3. Comprehensive File Operations

Upload/download single files
Recursive folder operations
File deletion with confirmation
Bucket-to-bucket copying

Core Implementation

The script uses a simple menu loop with case statements:

while true; do
    echo "S3 Bucket Manager - Bucket: $BUCKET_NAME"
    echo "1) List buckets by pattern"
    echo "2) Set bucket name"
    # ... more options

    read -p "Select option (1-14): " choice

    case $choice in
        1) # Pattern matching logic
        2) # Bucket name setting
        # ... handle each option
    esac
done

Bucket Name Validation

The script enforces AWS S3 naming rules:

if [[ ! "$BUCKET_NAME" =~ ^[a-z0-9][a-z0-9.-]*[a-z0-9]$ ]] || 
   [[ ${#BUCKET_NAME} -lt 3 ]] || 
   [[ ${#BUCKET_NAME} -gt 63 ]]; then
    echo "❌ Invalid bucket name"
fi

Safe File Handling

File paths are properly escaped and support tilde expansion:

filename=$(echo "$filename" | sed 's/^"\|"$//g')
filename=$(eval echo "$filename")

Advanced Operations

Bucket Renaming

Since S3 doesn't support direct renaming, the script:

Creates new bucket
Syncs all contents
Deletes original bucket
Updates current bucket reference

if aws s3 mb "s3://$new_bucket" 2>/dev/null; then
    if aws s3 sync "s3://$BUCKET_NAME" "s3://$new_bucket" 2>/dev/null && 
       aws s3 rb "s3://$BUCKET_NAME" --force 2>/dev/null; then
        BUCKET_NAME="$new_bucket"
        echo "✅ Bucket renamed to '$new_bucket'"
    fi
fi

Pattern Matching Magic

Converting shell wildcards to regex:

pattern=$(echo "$pattern" | sed 's/%/.*/g' | sed 's/\*/.*/g')
aws s3 ls | grep -E "$pattern" | head -20

Usage Examples

Quick Setup

./s3_manager.sh
# Enter bucket name when prompted
# Use menu to perform operations

Common Workflows

Find and switch to bucket:
- Option 1: List buckets with pattern
- Option 2: Set bucket name
Upload project folder:
- Option 8: Copy folder to bucket
- Enter: /path/to/project
Backup entire bucket:
- Option 12: Copy bucket to new bucket
- Creates complete duplicate

Error Handling

The script handles common issues:

Missing files/folders
Invalid bucket names
Network failures
Permission errors

All AWS CLI operations redirect stderr to /dev/null and check exit codes for clean error reporting.

Prerequisites

AWS CLI installed and configured
Valid AWS credentials with S3 permissions
Bash shell (works on macOS, Linux, WSL)

Code

#!/bin/bash

# Get bucket name from user
read -p "Enter S3 bucket name: " BUCKET_NAME

while true; do
    echo
    echo "S3 Bucket Manager - Bucket: $BUCKET_NAME"
    echo "----------------------------------------------------------"
    echo "1) List buckets by pattern"
    echo "2) Set bucket name"
    echo "3) Check if bucket exists"
    echo "4) Create bucket"
    echo "5) List files in bucket"
    echo "6) Copy file to bucket"
    echo "7) Copy file from bucket"
    echo "8) Copy folder to bucket"
    echo "9) Copy folder from bucket"
    echo "10) Remove file from bucket"
    echo "11) Remove bucket"
    echo "12) Copy bucket to new bucket"
    echo "13) Rename bucket"
    echo "14) Exit"
    echo
    read -p "Select option (1-14): " choice

    case $choice in
        1)
            read -p "Enter pattern (* for wildcard, % for any chars): " pattern
            pattern=$(echo "$pattern" | sed 's/%/.*/g' | sed 's/\*/.*/g')
            aws s3 ls | grep -E "$pattern" | head -20 || echo "No buckets match pattern"
            ;;
        2)
            read -p "Enter new S3 bucket name: " BUCKET_NAME
            echo "✅ Bucket name set to: $BUCKET_NAME"
            ;;
        3)
            if aws s3 ls "s3://$BUCKET_NAME" >/dev/null 2>&1; then
                echo "✅ Bucket '$BUCKET_NAME' exists"
            else
                echo "❌ Bucket '$BUCKET_NAME' does not exist"
            fi
            ;;
        4)
            if aws s3 ls "s3://$BUCKET_NAME" >/dev/null 2>&1; then
                echo "✅ Bucket '$BUCKET_NAME' already exists"
            else
                # Validate bucket name
                if [[ ! "$BUCKET_NAME" =~ ^[a-z0-9][a-z0-9.-]*[a-z0-9]$ ]] || [[ ${#BUCKET_NAME} -lt 3 ]] || [[ ${#BUCKET_NAME} -gt 63 ]]; then
                    echo "❌ Invalid bucket name. Must be 3-63 chars, lowercase, start/end with letter/number"
                elif aws s3 mb "s3://$BUCKET_NAME" 2>/dev/null; then
                    echo "✅ Bucket '$BUCKET_NAME' created"
                else
                    echo "❌ Failed to create bucket '$BUCKET_NAME' (may already exist globally)"
                fi
            fi
            ;;
        5)
            aws s3 ls "s3://$BUCKET_NAME"
            ;;
        6)
            read -p "Enter filename to upload: " filename
            filename=$(echo "$filename" | sed 's/^"\|"$//g')
            filename=$(eval echo "$filename")
            if [ -f "$filename" ]; then
                if aws s3 cp "$filename" "s3://$BUCKET_NAME/" 2>/dev/null; then
                    echo "✅ File '$filename' uploaded to bucket"
                else
                    echo "❌ Failed to upload file '$filename'"
                fi
            else
                echo "❌ File '$filename' not found"
            fi
            ;;
        7)
            read -p "Enter filename to download: " filename
            filename=$(echo "$filename" | sed 's/^"\|"$//g')
            if aws s3 cp "s3://$BUCKET_NAME/$filename" "./" 2>/dev/null; then
                echo "✅ File '$filename' downloaded to current directory"
            else
                echo "❌ Failed to download file '$filename'"
            fi
            ;;
        8)
            read -p "Enter folder path to upload: " foldername
            foldername=$(echo "$foldername" | sed 's/^"\|"$//g')
            if [ -d "$foldername" ]; then
                if aws s3 cp "$foldername" "s3://$BUCKET_NAME/" --recursive 2>/dev/null; then
                    echo "✅ Folder '$foldername' uploaded to bucket"
                else
                    echo "❌ Failed to upload folder '$foldername'"
                fi
            else
                echo "❌ Folder '$foldername' not found"
            fi
            ;;
        9)
            read -p "Enter folder name to download: " foldername
            foldername=$(echo "$foldername" | sed 's/^"\|"$//g')
            if aws s3 cp "s3://$BUCKET_NAME/$foldername" "./$foldername" --recursive 2>/dev/null; then
                echo "✅ Folder '$foldername' downloaded to current directory"
            else
                echo "❌ Failed to download folder '$foldername'"
            fi
            ;;
        10)
            read -p "Enter filename to remove: " filename
            if aws s3 rm "s3://$BUCKET_NAME/$filename" 2>/dev/null; then
                echo "✅ File '$filename' removed from bucket"
            else
                echo "❌ Failed to remove file '$filename'"
            fi
            ;;
        11)
            if aws s3 rb "s3://$BUCKET_NAME" --force 2>/dev/null; then
                echo "✅ Bucket '$BUCKET_NAME' removed"
            else
                echo "❌ Failed to remove bucket '$BUCKET_NAME'"
            fi
            ;;
        12)
            read -p "Enter new bucket name: " new_bucket
            if aws s3 mb "s3://$new_bucket" 2>/dev/null; then
                if aws s3 sync "s3://$BUCKET_NAME" "s3://$new_bucket" 2>/dev/null; then
                    echo "✅ Bucket '$BUCKET_NAME' copied to '$new_bucket'"
                else
                    echo "❌ Failed to copy bucket contents"
                fi
            else
                echo "❌ Failed to create new bucket '$new_bucket'"
            fi
            ;;
        13)
            read -p "Enter new bucket name: " new_bucket
            if aws s3 mb "s3://$new_bucket" 2>/dev/null; then
                if aws s3 sync "s3://$BUCKET_NAME" "s3://$new_bucket" 2>/dev/null && aws s3 rb "s3://$BUCKET_NAME" --force 2>/dev/null; then
                    BUCKET_NAME="$new_bucket"
                    echo "✅ Bucket renamed to '$new_bucket'"
                else
                    echo "❌ Failed to rename bucket"
                fi
            else
                echo "❌ Failed to create new bucket '$new_bucket'"
            fi
            ;;
        14)
            echo "Exiting..."
            exit 0
            ;;
        *)
            echo "Invalid option. Please select 1-14."
            ;;
    esac
done

Conclusion

This interactive S3 manager transforms tedious AWS CLI commands into a streamlined workflow. The menu-driven interface makes it perfect for:

DevOps automation
Quick file transfers
Bucket management tasks
Learning S3 operations

The script demonstrates how bash can create powerful, user-friendly tools for cloud management. With just 150 lines of code, you get a comprehensive S3 management interface.

Export AWS Untagged Resources to CSV for Analysis and Reporting

Dmitry Romanoff — Sat, 02 Aug 2025 22:00:07 +0000

Managing AWS resources without proper tagging can quickly become a nightmare for cost allocation, compliance, and governance. This Python script scans all AWS regions and exports untagged resources to a structured CSV file for easy analysis and reporting.

Why Export to CSV?

While console output is useful for quick checks, CSV export provides:

Structured data for spreadsheet analysis
Audit trails with timestamped files
Reporting capabilities for management
Integration with other tools and systems
Historical tracking of untagged resources over time

The Solution

This script builds on basic resource scanning by adding structured data export with proper ARN construction for each resource type.

#!/usr/bin/env python3
import boto3
import csv
import os
from datetime import datetime
from concurrent.futures import ThreadPoolExecutor, as_completed

def get_untagged_resources_in_region(region, account_id):
    resources = []
    try:
        session = boto3.Session()

        # EC2 Instances
        ec2 = session.client('ec2', region_name=region)
        instances = ec2.describe_instances()
        for reservation in instances['Reservations']:
            for instance in reservation['Instances']:
                if not instance.get('Tags'):
                    resources.append({
                        'Account': account_id,
                        'Region': region,
                        'Resource': 'EC2 Instance',
                        'ARN': f"arn:aws:ec2:{region}:{account_id}:instance/{instance['InstanceId']}"
                    })

        # EBS Volumes
        volumes = ec2.describe_volumes()
        for volume in volumes['Volumes']:
            if not volume.get('Tags'):
                resources.append({
                    'Account': account_id,
                    'Region': region,
                    'Resource': 'EBS Volume',
                    'ARN': f"arn:aws:ec2:{region}:{account_id}:volume/{volume['VolumeId']}"
                })

        # VPCs
        vpcs = ec2.describe_vpcs()
        for vpc in vpcs['Vpcs']:
            if not vpc.get('Tags'):
                resources.append({
                    'Account': account_id,
                    'Region': region,
                    'Resource': 'VPC',
                    'ARN': f"arn:aws:ec2:{region}:{account_id}:vpc/{vpc['VpcId']}"
                })

        # Security Groups
        security_groups = ec2.describe_security_groups()
        for sg in security_groups['SecurityGroups']:
            if not sg.get('Tags'):
                resources.append({
                    'Account': account_id,
                    'Region': region,
                    'Resource': 'Security Group',
                    'ARN': f"arn:aws:ec2:{region}:{account_id}:security-group/{sg['GroupId']}"
                })

        # Subnets
        subnets = ec2.describe_subnets()
        for subnet in subnets['Subnets']:
            if not subnet.get('Tags'):
                resources.append({
                    'Account': account_id,
                    'Region': region,
                    'Resource': 'Subnet',
                    'ARN': f"arn:aws:ec2:{region}:{account_id}:subnet/{subnet['SubnetId']}"
                })

        # Lambda Functions
        lambda_client = session.client('lambda', region_name=region)
        functions = lambda_client.list_functions()
        for function in functions['Functions']:
            try:
                tags = lambda_client.list_tags(Resource=function['FunctionArn'])
                if not tags.get('Tags'):
                    resources.append({
                        'Account': account_id,
                        'Region': region,
                        'Resource': 'Lambda Function',
                        'ARN': function['FunctionArn']
                    })
            except:
                resources.append({
                    'Account': account_id,
                    'Region': region,
                    'Resource': 'Lambda Function',
                    'ARN': function['FunctionArn']
                })

        # RDS Instances
        rds = session.client('rds', region_name=region)
        instances = rds.describe_db_instances()
        for instance in instances['DBInstances']:
            try:
                tags = rds.list_tags_for_resource(ResourceName=instance['DBInstanceArn'])
                if not tags.get('TagList'):
                    resources.append({
                        'Account': account_id,
                        'Region': region,
                        'Resource': 'RDS Instance',
                        'ARN': instance['DBInstanceArn']
                    })
            except:
                resources.append({
                    'Account': account_id,
                    'Region': region,
                    'Resource': 'RDS Instance',
                    'ARN': instance['DBInstanceArn']
                })

    except:
        pass

    return resources

def main():
    session = boto3.Session()

    # Get account ID
    sts = session.client('sts')
    account_id = sts.get_caller_identity()['Account']

    # Get regions
    regions = [r['RegionName'] for r in session.client('ec2').describe_regions()['Regions']]

    all_resources = []

    with ThreadPoolExecutor(max_workers=15) as executor:
        futures = [executor.submit(get_untagged_resources_in_region, region, account_id) for region in regions]

        for future in as_completed(futures):
            resources = future.result()
            all_resources.extend(resources)

    # S3 Buckets (global)
    try:
        s3 = session.client('s3')
        buckets = s3.list_buckets()
        for bucket in buckets['Buckets']:
            try:
                s3.get_bucket_tagging(Bucket=bucket['Name'])
            except:
                all_resources.append({
                    'Account': account_id,
                    'Region': 'Global',
                    'Resource': 'S3 Bucket',
                    'ARN': f"arn:aws:s3:::{bucket['Name']}"
                })
    except:
        pass

    # Create output directory and filename
    os.makedirs('output', exist_ok=True)
    timestamp = datetime.now().strftime('%Y%m%d_%H%M%S')
    filename = f'output/untagged_resources_{timestamp}.csv'

    # Export to CSV
    if all_resources:
        with open(filename, 'w', newline='') as csvfile:
            fieldnames = ['Account', 'Region', 'Resource', 'ARN']
            writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
            writer.writeheader()
            writer.writerows(all_resources)
        print(f"Exported {len(all_resources)} untagged resources to {filename}")
    else:
        print("No untagged resources found")

if __name__ == "__main__":
    main()

Key Features

Account identification: Uses STS to get current AWS account ID
Proper ARN construction: Builds correct ARNs for each resource type
Timestamped output: Creates unique files with YYYYMMDD_HHMMSS format
Organized storage: Saves files in output/ directory
CSV format: Easy to import into Excel, Google Sheets, or databases

CSV Output Structure

The exported CSV contains four columns:

Account	Region	Resource	ARN
123456789012	us-east-1	EC2 Instance	arn:aws:ec2:us-east-1:123456789012:instance/i-1234567890abcdef0
123456789012	us-west-2	S3 Bucket	arn:aws:s3:::my-untagged-bucket
123456789012	eu-west-1	RDS Instance	arn:aws:rds:eu-west-1:123456789012:db:mydb

Setup and Usage

Install boto3:

pip install boto3

Configure AWS credentials:

aws configure

Run the script:

python get_untagged_resources_per_region_excel.py

Find your report:

output/untagged_resources_20241220_143052.csv

Required IAM Permissions

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "sts:GetCallerIdentity",
                "ec2:DescribeInstances",
                "ec2:DescribeVolumes",
                "ec2:DescribeVpcs",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeSubnets",
                "ec2:DescribeRegions",
                "s3:ListAllMyBuckets",
                "s3:GetBucketTagging",
                "lambda:ListFunctions",
                "lambda:ListTags",
                "rds:DescribeDBInstances",
                "rds:ListTagsForResource"
            ],
            "Resource": "*"
        }
    ]
}

Use Cases for CSV Export

Compliance reporting: Generate monthly untagged resource reports
Cost analysis: Import into BI tools for cost allocation analysis
Automation: Feed data into tagging automation scripts
Tracking progress: Compare reports over time to measure tagging improvements
Multi-account analysis: Combine reports from multiple AWS accounts

Advanced Analysis

Once you have the CSV, you can analyze it without external dependencies:

#!/usr/bin/env python3
import csv
from collections import Counter

def analyze_csv(filename):
    """Analyze untagged resources CSV without pandas"""
    resources = []

    try:
        with open(filename, 'r') as csvfile:
            reader = csv.DictReader(csvfile)
            resources = list(reader)
    except FileNotFoundError:
        print(f"File {filename} not found")
        return

    if not resources:
        print("No data found in CSV")
        return

    print(f"Total untagged resources: {len(resources)}")

    # Count by resource type
    resource_counts = Counter(row['Resource'] for row in resources)
    print("\nResources by type:")
    for resource_type, count in resource_counts.most_common():
        print(f"  {resource_type}: {count}")

    # Count by region
    region_counts = Counter(row['Region'] for row in resources)
    print("\nResources by region:")
    for region, count in region_counts.most_common():
        print(f"  {region}: {count}")

    # Filter EC2 instances
    ec2_instances = [row for row in resources if row['Resource'] == 'EC2 Instance']
    if ec2_instances:
        print(f"\nEC2 Instances ({len(ec2_instances)}):")
        for instance in ec2_instances[:5]:  # Show first 5
            print(f"  {instance['Region']}: {instance['ARN']}")
        if len(ec2_instances) > 5:
            print(f"  ... and {len(ec2_instances) - 5} more")

if __name__ == "__main__":
    # Find the latest CSV file
    import os
    import glob

    csv_files = glob.glob('output/untagged_resources_*.csv')
    if csv_files:
        latest_file = max(csv_files, key=os.path.getctime)
        print(f"Analyzing: {latest_file}\n")
        analyze_csv(latest_file)
    else:
        print("No CSV files found in output/ directory")
        print("Run get_untagged_resources_per_region_excel.py first")

Or use the included analysis script:

python advanced_analysis.py

Conclusion

Structured data export transforms untagged resource discovery from a one-time check into a systematic governance process. The CSV format enables integration with existing reporting workflows and provides the foundation for automated remediation efforts.

Regular exports help track tagging compliance over time and provide the data needed for informed decisions about AWS resource governance policies.

Find Untagged AWS Resources Across All Regions with Python

Dmitry Romanoff — Sat, 02 Aug 2025 21:26:51 +0000

Resource tagging is crucial for AWS cost management, security, and governance. Untagged resources can lead to unexpected costs and make it difficult to track resource ownership. Here's a Python script that scans all AWS regions to find untagged resources.

The Problem

AWS resources without proper tags create several issues:

Cost tracking: Can't allocate costs to projects or teams
Security: Difficult to identify resource owners
Compliance: Many organizations require mandatory tags
Cleanup: Hard to identify unused resources

The Solution

This Python script uses boto3 and concurrent processing to efficiently scan all AWS regions for untagged resources.

#!/usr/bin/env python3
import boto3
from concurrent.futures import ThreadPoolExecutor, as_completed

def get_untagged_resources_in_region(region):
    untagged = []
    try:
        session = boto3.Session()

        # EC2 Instances
        ec2 = session.client('ec2', region_name=region)
        instances = ec2.describe_instances()
        for reservation in instances['Reservations']:
            for instance in reservation['Instances']:
                if not instance.get('Tags'):
                    untagged.append(f"EC2 Instance: {instance['InstanceId']}")

        # EBS Volumes
        volumes = ec2.describe_volumes()
        for volume in volumes['Volumes']:
            if not volume.get('Tags'):
                untagged.append(f"EBS Volume: {volume['VolumeId']}")

        # Lambda Functions
        lambda_client = session.client('lambda', region_name=region)
        functions = lambda_client.list_functions()
        for function in functions['Functions']:
            try:
                tags = lambda_client.list_tags(Resource=function['FunctionArn'])
                if not tags.get('Tags'):
                    untagged.append(f"Lambda Function: {function['FunctionName']}")
            except:
                untagged.append(f"Lambda Function: {function['FunctionName']}")

        # RDS Instances
        rds = session.client('rds', region_name=region)
        instances = rds.describe_db_instances()
        for instance in instances['DBInstances']:
            try:
                tags = rds.list_tags_for_resource(ResourceName=instance['DBInstanceArn'])
                if not tags.get('TagList'):
                    untagged.append(f"RDS Instance: {instance['DBInstanceIdentifier']}")
            except:
                untagged.append(f"RDS Instance: {instance['DBInstanceIdentifier']}")

        # VPCs
        vpcs = ec2.describe_vpcs()
        for vpc in vpcs['Vpcs']:
            if not vpc.get('Tags'):
                untagged.append(f"VPC: {vpc['VpcId']}")

        # Security Groups
        security_groups = ec2.describe_security_groups()
        for sg in security_groups['SecurityGroups']:
            if not sg.get('Tags'):
                untagged.append(f"Security Group: {sg['GroupId']}")

        # Subnets
        subnets = ec2.describe_subnets()
        for subnet in subnets['Subnets']:
            if not subnet.get('Tags'):
                untagged.append(f"Subnet: {subnet['SubnetId']}")

    except:
        pass

    return region, untagged

def main():
    session = boto3.Session()
    regions = [r['RegionName'] for r in session.client('ec2').describe_regions()['Regions']]

    with ThreadPoolExecutor(max_workers=15) as executor:
        futures = [executor.submit(get_untagged_resources_in_region, region) for region in regions]

        for future in as_completed(futures):
            region, untagged = future.result()
            if untagged:
                print(f"\n{region} ({len(untagged)} untagged resources):")
                for resource in untagged:
                    print(f"  - {resource}")

    # S3 Buckets (global)
    try:
        s3 = session.client('s3')
        buckets = s3.list_buckets()
        untagged_buckets = []
        for bucket in buckets['Buckets']:
            try:
                s3.get_bucket_tagging(Bucket=bucket['Name'])
            except:
                untagged_buckets.append(f"S3 Bucket: {bucket['Name']}")

        if untagged_buckets:
            print(f"\nGlobal S3 ({len(untagged_buckets)} untagged buckets):")
            for bucket in untagged_buckets:
                print(f"  - {bucket}")
    except:
        pass

if __name__ == "__main__":
    main()

Key Features

Multi-region scanning: Checks all enabled AWS regions
Concurrent processing: Uses ThreadPoolExecutor for fast execution
Multiple resource types: EC2, EBS, VPCs, Security Groups, Subnets, Lambda, RDS, and S3
Error handling: Gracefully handles permission issues

Setup

Install boto3:

pip install boto3

Configure AWS credentials:

aws configure

Required IAM Permissions

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeInstances",
                "ec2:DescribeVolumes",
                "ec2:DescribeVpcs",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeSubnets",
                "ec2:DescribeRegions",
                "s3:ListAllMyBuckets",
                "s3:GetBucketTagging",
                "lambda:ListFunctions",
                "lambda:ListTags",
                "rds:DescribeDBInstances",
                "rds:ListTagsForResource"
            ],
            "Resource": "*"
        }
    ]
}

Sample Output

us-east-1 (5 untagged resources):
  - EC2 Instance: i-1234567890abcdef0
  - EBS Volume: vol-1234567890abcdef0
  - VPC: vpc-1234567890abcdef0
  - Security Group: sg-1234567890abcdef0
  - Lambda Function: my-function

us-west-2 (2 untagged resources):
  - RDS Instance: my-database
  - Subnet: subnet-1234567890abcdef0

Global S3 (2 untagged buckets):
  - S3 Bucket: my-bucket-1
  - S3 Bucket: my-bucket-2

Next Steps

Once you identify untagged resources, you can:

Tag them manually through the AWS console
Use AWS CLI to bulk tag resources
Implement tagging policies to prevent future untagged resources
Set up AWS Config rules for automated compliance checking

Conclusion

Regular auditing of untagged resources is essential for AWS governance. This script provides a quick way to identify resources that need attention across your entire AWS infrastructure.

The concurrent approach makes it efficient even for accounts with resources across many regions, helping you maintain better control over your AWS environment.

Document_DB_Command_Describer: GenAI-Powered Document Database Command Helper

Dmitry Romanoff — Thu, 17 Jul 2025 12:32:30 +0000

Working with document databases like MongoDB or Couchbase can be tricky—especially when it comes to remembering command syntax or generating complex queries. Imagine having an AI assistant that not only describes what a command does, but can generate new commands for you, all powered by advanced generative models. That’s the magic behind Document_DB_Command_Describer!

What is Document_DB_Command_Describer?

Document_DB_Command_Describer is an open-source tool that uses Generative AI (GenAI) models to help developers interact with document databases more intuitively. The project harnesses the capabilities of AI to analyze, explain, and even construct database commands based on natural language or structured input.

GenAI Under the Hood

At its core, Document_DB_Command_Describer uses GenAI models—think large language models (LLMs) like GPT—to:

Parse and interpret database commands
Generate human-readable descriptions of what each command does
Construct new commands based on user intent or parameters

This means you’re not just relying on static templates or old-school regex parsing. Instead, the tool leverages modern AI to understand context, intent, and database semantics, making its explanations and command generation more accurate and flexible.

Key Features

AI-Powered Command Description: Get concise, context-aware explanations for any command.
Natural Language Command Generation: Describe what you want in plain English; let GenAI build the command for you.
Cross-Database Support: Works with MongoDB, Couchbase, and other document-oriented databases.
Documentation Aid: Automatically generate code snippets and explanations for docs or onboarding.
Extensible: Easily add new commands, databases, or custom AI models.

Getting Started

Clone the repository and install dependencies:

git clone https://github.com/dm8ry/Document_DB_Command_Describer.git
cd Document_DB_Command_Describer
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt  # For Python projects

You may need an API key or endpoint for the GenAI backend. Check the repo’s README for setup details!

Example Usage

Describe a Command

Input:

db.users.find({ "age": { "$gt": 25 } })

Output (via GenAI):

Finds all documents in the users collection where the age field is greater than 25.

Generate a Command

Input (Natural Language):

Find users older than 25.

Output (via GenAI):

db.users.find({ "age": { "$gt": 25 } })

Python API Example

from describer import describe_command, generate_command

desc = describe_command('db.users.find({ "age": { "$gt": 25 } })')
print(desc)

cmd = generate_command(intent="Find users older than 25")
print(cmd)

Why GenAI?

The use of GenAI models means the tool can:

Understand natural language requests and map them to database commands
Offer explanations tailored to your context
Adapt to new database syntaxes or custom extensions
Improve over time as models are updated

Final Thoughts

GenAI is transforming how developers interact with databases. With Document_DB_Command_Describer, you can work smarter—not harder—by letting AI handle the heavy lifting of command comprehension and generation.

Repo: dm8ry/Document_DB_Command_Describer

Document_DB_Command_Describer: Simplifying Document Database Commands

Dmitry Romanoff — Thu, 17 Jul 2025 12:23:33 +0000

Are you working with document databases and often find yourself struggling to remember the exact syntax for different commands? Whether it’s MongoDB, Couchbase, or another NoSQL solution, command variations and options can get overwhelming. That’s where the Document_DB_Command_Describer comes in—a handy tool to help developers quickly understand, generate, and document commands for popular document databases.

What is Document_DB_Command_Describer?

Document_DB_Command_Describer is an open-source utility designed to help developers describe, explain, and generate commands for document databases. It streamlines the process of learning, using, and documenting database commands, making it easier for newcomers and seasoned professionals alike to work efficiently.

Key Features

Command Description: Enter a database command and get a clear, concise description of what it does.
Command Generation: Generate commands based on simple input (e.g., collection name, fields, conditions).
Documentation Aid: Use this tool to create code snippets and explanations for documentation or onboarding manuals.
Extensible: Supports common document database systems and can be adapted for more.

Getting Started

To use Document_DB_Command_Describer, clone the repository:

git clone https://github.com/dm8ry/Document_DB_Command_Describer.git
cd Document_DB_Command_Describer

Install dependencies (if required):

# Example for Python projects
pip install -r requirements.txt

Example Usage

Suppose you want to understand the following MongoDB command:

db.users.find({ "age": { "$gt": 25 } })

Pass it to Document_DB_Command_Describer, and you might get:

Description:

Finds all documents in the users collection where the age field is greater than 25.

Or, generate a command:

from describer import generate_command

command = generate_command(
    collection="users",
    action="find",
    conditions={"age": {"$gt": 25}}
)
print(command)

Output:

db.users.find({ "age": { "$gt": 25 } })

Why Use Document_DB_Command_Describer?

Reduce Cognitive Load: No need to memorize every database command—get instant explanations.
Boost Productivity: Quickly generate commands for common tasks.
Improve Documentation: Create educational resources for your team or users.
Learn by Example: See how database commands map to real-world actions.

Contributing

Document_DB_Command_Describer is open-source and welcomes contributions! If you have ideas for new databases, commands, or features, submit a pull request or open an issue.

Final Thoughts

If you often work with document databases, give Document_DB_Command_Describer a try. It’s a small tool that can make a big difference in your daily workflow, especially when onboarding new team members or documenting database operations for your projects.

Repo: dm8ry/Document_DB_Command_Describer

When Not to Use Machine Learning (and Why It Matters)

Dmitry Romanoff — Thu, 17 Jul 2025 12:13:34 +0000

Machine learning (ML) is everywhere—from recommending your next movie to powering self-driving cars. It’s tempting to think ML is a silver bullet for all problems involving data. But in reality, there are many scenarios where applying ML is not just a poor fit—it can be dangerous, unethical, or simply ineffective.

Let’s explore six key situations where machine learning should be avoided, along with real-world examples to illustrate why.

1. Rapidly Evolving or Unpredictable Environments

ML models learn patterns from historical data. But what happens when the world they operate in changes faster than they can learn?

Example:
A startup tries to use ML to predict the success of TikTok videos. But trends on TikTok evolve daily—what’s viral today is passé tomorrow. By the time the model trains on past data, those patterns are obsolete.

In such environments, rule-based systems or human judgment might be more adaptable.

2. Safety-Critical Applications

If the cost of failure is human life, think twice.

Example:
Self-driving cars and autonomous drones are examples where ML is promising—but risky. A misclassified object could lead to fatal consequences. In 2018, an autonomous Uber vehicle struck and killed a pedestrian due to such a failure.

ML systems lack the predictability and accountability required in safety-critical domains. Traditional engineering with exhaustive testing and verification is still the gold standard here.

3. Strict Regulatory or Legal Constraints

Many industries are governed by rules requiring full transparency—something most ML models, especially deep learning ones, can’t provide.

Example:
In finance, regulators often require firms to explain how a loan decision was made. If a neural network denies a mortgage and can’t explain why, that’s a compliance violation.

In such domains, interpretable models or rule-based systems are often preferred—even if they are less accurate.

4. Highly Sensitive or Ethical Decision-Making

ML can inadvertently reinforce biases present in training data, leading to unfair or unethical outcomes.

Example:
An ML system used for hiring might learn to favor male applicants if historical hiring data was biased. This can perpetuate discrimination and expose the company to lawsuits and public backlash.

In cases involving race, gender, criminal justice, or employment, it's often better to rely on human oversight and transparent criteria.

5. Situations Requiring 100% Accuracy

Machine learning is inherently probabilistic. Even the best models make mistakes. If your application cannot tolerate errors, ML is likely the wrong tool.

Example:
In medical diagnosis, missing a rare but fatal disease can have dire consequences. Even with 99% accuracy, a model that misclassifies 1 in 100 patients could be catastrophic.

When perfect accuracy is non-negotiable, traditional deterministic systems or expert human review are safer.

6. Insufficient or Low-Quality Data

ML is data-hungry. Training a useful model typically requires large, clean, and labeled datasets. Without this, even the most sophisticated algorithms will fail.

Example:
A small e-commerce startup wants to predict customer churn. With only 200 historical customer records, there’s just not enough data to train a reliable model.

In such cases, heuristics, customer interviews, or rule-based analytics can offer more actionable insights.

Final Thoughts

Machine learning is a powerful tool—but it’s not a universal one. Like any tool, its effectiveness depends on the context. Blindly applying ML where it doesn’t fit can lead to wasted time, lost money, or worse—harm to real people.

Before jumping into model training, ask yourself:

Is the environment stable?
Is human safety involved?
Are there ethical or legal constraints?
Do I have enough data?
Is absolute accuracy required?

If the answer to any of these is yes, step back and rethink. Sometimes, the smartest solution isn’t artificial intelligence—it’s intelligent decision-making.

Boosting Performance with AWS DocumentDB: Exercises in Data Modeling and Optimization

Dmitry Romanoff — Sat, 12 Jul 2025 13:04:54 +0000

When working with AWS DocumentDB, it's essential to understand how to efficiently query and manipulate large datasets. In this article, we'll go through several practical exercises, showcasing common use cases, and providing examples of how to optimize your database performance using indexes. We’ll focus on operations like inserting documents, performing aggregations, and comparing query performance before and after adding indexes. Let’s dive in!

1. Inserting Multiple Documents into AWS DocumentDB

In many real-world applications, you’ll need to handle large datasets, such as orders and customer transactions. Below, we simulate inserting 1000 orders, each containing a list of items purchased by the customer. The goal here is to bulk-insert data into AWS DocumentDB using the insertMany function.

Example: Bulk Inserting Orders

let orders = [];

for (let i = 1; i <= 1000; i++) {
  let numItems = Math.floor(Math.random() * 301); // Random number of items between 0 and 300

  let items = [];
  for (let j = 0; j < numItems; j++) {
    items.push({
      "product_id": "prod_" + String(Math.floor(Math.random() * 10) + 1).padStart(3, '0'),
      "quantity": Math.floor(Math.random() * 5) + 1, // Random quantity between 1 and 5
      "price": Math.floor(Math.random() * 50) + 10  // Random price between 10 and 50
    });
  }

  let order = {
    "_id": "order" + String(i).padStart(3, '0'),
    "customer_id": "cust_" + String(Math.floor(Math.random() * 100) + 1).padStart(3, '0'),
    "order_date": new ISODate("2023-07-" + String(i % 31 + 1).padStart(2, '0') + "T00:00:00Z"),
    "items": items // The dynamically generated items array
  };

  orders.push(order);
}

// Now insert all orders into the collection
db.orders.insertMany(orders);

In this script:

We generate 1000 orders, each with a random number of items.
The items have random product IDs, quantities, and prices.
After creating the orders, we use insertMany to bulk insert the documents into the orders collection.

Once the insert operation completes, you can check the document count using:

db.orders.countDocuments();

2. Querying Specific Product in Orders

Let’s now query for orders that contain a specific product, such as prod_006. This is a typical operation when you need to analyze how many times a product appears across all orders.

Example: Query with \$elemMatch

db.orders.countDocuments({
  "items": {
    $elemMatch: {
      "product_id": "prod_006"
    }
  }
});

In this query:

We use $elemMatch to find documents where at least one item in the items array matches the product ID prod_006.

3. Querying for Products with Specific Quantities

Another common query involves filtering based on both product and quantity. Let’s say we want to find orders that contain prod_007 with quantities greater than 1.

Example: Query with Multiple Conditions

db.orders.countDocuments({
  "items": {
    $elemMatch: {
      "product_id": "prod_007",
      "quantity": { $gt: 1 }
    }
  }
});

This query retrieves orders where:

The items array contains prod_007, and
The quantity of prod_007 is greater than 1.

4. Aggregation: Finding Min and Max Quantities

Aggregations are powerful tools for summarizing data. Let’s say we need to find the minimum and maximum quantities of a specific product, like prod_010, across all orders.

Example: Aggregation Pipeline with \$unwind

db.orders.aggregate([
  { 
    $unwind: "$items"  // Flatten the items array
  },
  { 
    $match: { 
      "items.product_id": "prod_010"  // Match documents where product_id is 'prod_010'
    }
  },
  { 
    $group: { 
      _id: null,  // Grouping all documents together
      minQuantity: { $min: "$items.quantity" },  // Find the minimum quantity
      maxQuantity: { $max: "$items.quantity" }   // Find the maximum quantity
    }
  }
]);

In this aggregation:

$unwind is used to flatten the items array, making it easier to work with individual products.
$match filters for orders containing prod_010.
$group then calculates the minimum and maximum quantities for prod_010.

5. Aggregating Quantities for Specific Product and Condition

Let’s take it further and calculate the total quantity of prod_007 when the quantity is less than 2.

Example: Aggregation with \$sum

db.orders.aggregate([
  { 
    $unwind: "$items"  // Flatten the items array
  },
  { 
    $match: { 
      "items.product_id": "prod_007",  // Match product_id = 'prod_007'
      "items.quantity": { $lt: 2 }     // Match quantity < 2
    }
  },
  { 
    $group: { 
      _id: null,  // Grouping all documents together
      totalQuantity: { $sum: "$items.quantity" }  // Sum the quantities of matching items
    }
  }
]);

Here:

We again use $unwind to flatten the items.
$match filters for items where the product is prod_007 and the quantity is less than 2.
Finally, $group sums the matching quantities.

6. Querying Specific Orders with \$unwind

Sometimes you might want to query a specific order based on its ID, customer, or order date, and also unwind the items array for further analysis. Here’s how you can do it.

Example: Querying Specific Orders

db.orders.aggregate([
  { 
    $match: { 
      "_id": "order007",  // Match the specific order by _id
      "customer_id": "cust_057",  // Optionally match by customer_id if needed
      "order_date": ISODate('2023-07-08T00:00:00.000Z')  // Match by order_date if needed
    }
  },
  { 
    $unwind: "$items"  // Unwind the items array
  }
]);

This query matches an order with the ID order007, belonging to customer cust_057, and orders placed on 2023-07-08. Then, it unwinds the items array so each item is returned as a separate document.

7. Comparing Query Performance: With and Without Indexes

Lastly, let’s discuss performance optimization. Without an index, searching through large arrays in documents can be slow. Let’s compare the performance of querying for prod_007 with and without an index on product_id and quantity.

Example: Query Performance Without Index

// Without index
const startWithoutIndex = new Date().getTime();  // Capture start time
const resultWithoutIndex = db.orders.find({
  "items": {
    $elemMatch: {
      "product_id": "prod_007",
      "quantity": { $gt: 1 }
    }
  }
});

const endWithoutIndex = new Date().getTime();  // Capture end time
const executionTimeWithoutIndex = endWithoutIndex - startWithoutIndex;  // Execution time in ms

print("Execution Time Without Index: " + executionTimeWithoutIndex + " ms");

Example: Creating an Index and Querying Again

// Create index
db.orders.createIndex({
  "items.product_id": 1,
  "items.quantity": 1
});

// With index
const startWithIndex = new Date().getTime();  // Capture start time
const resultWithIndex = db.orders.find({
  "items": {
    $elemMatch: {
      "product_id": "prod_007",
      "quantity": { $gt: 1 }
    }
  }
});
const endWithIndex = new Date().getTime();  // Capture end time
const executionTimeWithIndex = endWithIndex - startWithIndex;  // Execution time in ms

print("Execution Time With Index: " + executionTimeWithIndex + " ms");

// Compare the difference
const timeDifference = executionTimeWithoutIndex - executionTimeWithIndex;
print("Difference in Execution Time: " + timeDifference + " ms");

In this example:

We first capture the execution time of a query without an index.
Then, we create an index on product_id and quantity.
Finally, we compare the execution times, and you’ll see how much faster the query runs with the index in place.

Conclusion

Optimizing queries and understanding how to structure your data in AWS DocumentDB can make a significant difference in performance. Through these exercises, we've learned how to insert large datasets, perform common queries, aggregate data, and optimize performance with indexing. Always test performance before and after adding indexes to ensure your queries run efficiently at scale.

Querying a Nested Array with `elemMatch` in Amazon DocumentDB

Dmitry Romanoff — Sat, 12 Jul 2025 12:24:24 +0000

Let's say you have a MongoDB-like database (Amazon DocumentDB) where your collections include documents with nested arrays. You want to query documents where an element in the nested array matches a specific condition. One of the most efficient ways to do this is by using the elemMatch operator, and also indexing the fields involved to improve query performance.

Scenario

You're working with a collection called orders in your DocumentDB database, which stores customer orders. Each order document contains an array of items, and each item has several fields such as product_id, quantity, and price.

Example Document

{
  "_id": "order123",
  "customer_id": "cust_001",
  "order_date": "2023-07-10",
  "items": [
    {
      "product_id": "prod_001",
      "quantity": 2,
      "price": 20
    },
    {
      "product_id": "prod_002",
      "quantity": 1,
      "price": 40
    }
  ]
}

Problem

You want to efficiently query all orders that contain an item with a specific product_id and quantity that meets certain conditions, for example, orders that contain at least one item with product_id of "prod_001" and quantity greater than 1.

db.orders.find({
  "items": {
    "$elemMatch": {
      "product_id": "prod_001",
      "quantity": { "$gt": 1 }
    }
  }
})

This query will return orders where at least one of the items has product_id of "prod_001" and quantity greater than 1.

Solution

To improve the performance of the query, you'll need to ensure that DocumentDB can use an index to speed up lookups. Here's how you can index it.

Step 1: Indexing with `elemMatch` and Compound Indexes

When querying nested arrays like this, a compound index on the array field is critical for performance. For your orders collection, you can create an index that includes the items.product_id and items.quantity fields. This will allow DocumentDB to quickly find the specific items within the array that match your elemMatch query condition.

Index Creation

db.orders.createIndex({
  "items.product_id": 1,
  "items.quantity": 1
})

Explanation:

items.product_id: 1: This creates an index on the product_id field inside the items array.
items.quantity: 1: This creates an index on the quantity field inside the items array.
The compound index on both fields will help optimize the query that looks for documents containing items with specific product IDs and quantities.

Step 2: Query with `elemMatch`

Now you can run your elemMatch query, and DocumentDB will use the compound index to speed up access to the matching documents.

db.orders.find({
  "items": {
    "$elemMatch": {
      "product_id": "prod_001",
      "quantity": { "$gt": 1 }
    }
  }
})

This query will return all orders that contain at least one item with a product_id of "prod_001" and a quantity greater than 1. Because of the compound index on items.product_id and items.quantity, DocumentDB can efficiently perform the query even if the orders collection is large.

Step 3: Index Monitoring and Optimization

Once the index is created, it's important to monitor the query performance over time and adjust your indexes as needed. You can do this using the following:

Profile the query: Use the explain() method to check how DocumentDB is executing your query.

db.orders.find({
  "items": {
    "$elemMatch": {
      "product_id": "prod_001",
      "quantity": { "$gt": 1 }
    }
  }
}).explain("executionStats")

Adjust indexes: If the query still isn't fast enough, consider:

Adding additional fields to the index if you frequently query by other attributes (e.g., order_date or customer_id).
Ensure indexes are balanced and not too many to avoid overhead.

Bonus: Additional Indexing Considerations

Partial Index: If you're only interested in certain types of orders (e.g., only orders from a specific customer_id or within a date range), you can use a partial index that includes only those documents that meet certain criteria. This can save space and improve query performance.

Example:

db.orders.createIndex(
  { "items.product_id": 1, "items.quantity": 1 },
  { partialFilterExpression: { "order_date": { "$gte": new ISODate("2023-01-01") } } }
)

This index would only be used for orders with an order_date after January 1, 2023, improving query performance for those specific orders.

Conclusion

To summarize:

Use elemMatch for querying nested arrays in DocumentDB.
Create a compound index on the fields inside the array that are frequently queried.
Monitor performance using explain() and adjust indexes as necessary.
Consider partial indexes for more specific query patterns to optimize storage and performance further.

Working with AWS DocumentDB: Indexing, Performance, and Stats at Scale

Dmitry Romanoff — Sat, 12 Jul 2025 10:05:24 +0000

Amazon DocumentDB is a fully managed document database service that supports MongoDB workloads. In this post, we’ll walk through a practical, large-scale indexing and performance analysis using DocumentDB. We’ll simulate inserting 100,000 documents, creating indexes, measuring stats, and analyzing the impact of updates on performance.

We'll use raw shell commands with real outputs — no shortcuts. Let’s get started.

🔧 Step 1: Insert 100,000 Documents into DocumentDB

// Choose the database
use mylibrary;

// Define helper functions (basic randomizer)
function getRandomElement(arr) {
  return arr[Math.floor(Math.random() * arr.length)];
}

function getRandomInt(min, max) {
  return Math.floor(Math.random() * (max - min + 1)) + min;
}

var genres = ['Fiction', 'Non-Fiction', 'Science', 'Fantasy', 'Biography', 'History', 'Mystery', 'Romance'];
var formats = ['epub', 'pdf', 'txt', 'audio'];
var firstNames = ['John', 'Mary', 'Alice', 'Robert', 'Linda', 'Michael', 'Sarah', 'David'];
var lastNames = ['Smith', 'Johnson', 'Williams', 'Brown', 'Jones', 'Miller', 'Davis', 'Garcia'];

for (var i = 0; i < 100000; i++) {
  var author = getRandomElement(firstNames) + " " + getRandomElement(lastNames);
  var title = "Book Title " + (i + 1);
  var book = {
    author: author,
    title: "title,"
    genre: getRandomElement(genres),
    format: getRandomElement(formats),
    num_of_pages: getRandomInt(50, 1000),
    year_when_published: getRandomInt(1900, 2025)
  };

  db.books.insertOne(book);
}

db.books.find().count();

Output:

🔍 Step 2: Create Indexes on Author and Title + Year

db.books.createIndex({ author: 1 })
db.books.createIndex({ title: 1, year_when_published: 1 })

📊 Step 3: Get Collection Stats

db.books.stats()

Output:

{
  ns: 'mylibrary;.books',
  count: 100000,
  size: 20600000,
  storageSize: 24150016,
  nindexes: 3,
  totalIndexSize: 14106624,
  indexSizes: {
    _id_: 4079616,
    author_1: 4104192,
    title_1_year_when_published_1: 5922816
  },
  ...
}

📌 Step 4: Print Only Index Sizes and OpCounter

const stats = db.books.stats();
printjson({
  ns: stats.ns,
  indexSizes: stats.indexSizes,
  opCounter: stats.opCounter
});

Output:

{
  ns: 'mylibrary;.books',
  indexSizes: {
    _id_: 4079616,
    author_1: 4104192,
    title_1_year_when_published_1: 5922816
  },
  opCounter: {
    numDocsIns: 100000,
    numDocsUpd: 0,
    numDocsDel: 0
  }
}

📂 Step 5: Check Index Metadata

db.books.getIndexes()

Output:

[
  { key: { _id: 1 }, name: '_id_' },
  { key: { author: 1 }, name: 'author_1' },
  { key: { title: 1, year_when_published: 1 }, name: 'title_1_year_when_published_1' }
]

🔄 Step 6: Perform 10 Updates and Check `opCounter` Diff

const before = db.books.stats().opCounter;

db.books.find().limit(10).forEach(doc => {
  db.books.updateOne({ _id: doc._id }, { $inc: { num_of_pages: 1 } });
});

const after = db.books.stats().opCounter;

const diff = {
  numDocsIns: after.numDocsIns - before.numDocsIns,
  numDocsUpd: after.numDocsUpd - before.numDocsUpd,
  numDocsDel: after.numDocsDel - before.numDocsDel
};

printjson(diff);

Output:

{
  numDocsIns: 0,
  numDocsUpd: 10,
  numDocsDel: 0
}

📈 Step 7: Perform 1000 Updates and Check Index Growth

const beforeIndexes = db.books.stats().indexSizes;

db.books.find().limit(1000).forEach(doc => {
  db.books.updateOne({ _id: doc._id }, { $inc: { num_of_pages: 1 } });
});

const afterIndexes = db.books.stats().indexSizes;

const diffIndexes = {};
for (const key in beforeIndexes) {
  diffIndexes[key] = afterIndexes[key] - beforeIndexes[key];
}

printjson(diffIndexes);

Output:

{
  _id_: 0,
  author_1: 16384,
  title_1_year_when_published_1: 0
}

🔁 Step 8: Rebuild `author_1` Index

db.books.dropIndex("author_1")
db.books.createIndex({ author: 1 }, { background: true })

⚡ Step 9: Compare Execution Plan With and Without Index

// 1. Query using index
const indexedPlan = db.books.find({ author: "Sarah Williams" })
  .explain("executionStats");
const indexedTime = indexedPlan.executionStats.executionTimeMillis;

// 2. Query forcing collection scan
const collScanPlan = db.books.find({ author: "Sarah Williams" })
  .hint({ $natural: 1 })
  .explain("executionStats");
const collScanTime = collScanPlan.executionStats.executionTimeMillis;

// 3. Print both times and their difference
print("Execution Time With Index (ms):", indexedTime);
print("Execution Time Without Index (ms):", collScanTime);
print("Difference (Without - With):", collScanTime - indexedTime);

Output:

Execution Time With Index (ms): 3.716
Execution Time Without Index (ms): 90.578
Difference (Without - With): 86.862

🧪 Step 10: Update Author Field and Rerun the Same Query

// STEP 1: Query before updates
const beforePlan = db.books.find({ author: "Sarah Williams" })
  .explain("executionStats");
const beforeTime = beforePlan.executionStats.executionTimeMillis;

// STEP 2: Perform 1000 updates that affect the `author` index
let updated = 0;
db.books.find().limit(1000).forEach(doc => {
  const newAuthor = doc.author + "_v2_" + updated;
  db.books.updateOne({ _id: doc._id }, { $set: { author: newAuthor } });
  updated++;
});

// STEP 3: Query again after updates
const afterPlan = db.books.find({ author: "Sarah Williams" })
  .explain("executionStats");
const afterTime = afterPlan.executionStats.executionTimeMillis;

// STEP 4: Compare timings
print("Execution Time With Index (Before Updates):", beforeTime, "ms");
print("Execution Time With Index (After Updates):", afterTime, "ms");
print("Time Difference (After - Before):", afterTime - beforeTime, "ms");

Output:

Execution Time With Index (Before Updates): 2.208 ms
Execution Time With Index (After Updates): 2.431 ms
Time Difference (After - Before): 0.223 ms

🧾 Conclusion

In this walkthrough, we:

Created 100K documents in AWS DocumentDB.
Built and measured the impact of indexes.
Used opCounter and stats() to track insert and update behavior.
Compared index performance vs. collection scans.
Saw how index size and query performance can shift under load.

This kind of data instrumentation is critical when working with large-scale workloads in DocumentDB — especially since it's a managed service where you don’t have access to low-level server controls.

✅ Tip: Use these techniques to continuously monitor, tune, and optimize your DocumentDB performance — especially when dealing with write-heavy workloads or evolving query patterns.

Monitoring and Managing Heavy Aggregations in AWS DocumentDB

Dmitry Romanoff — Sat, 12 Jul 2025 09:08:14 +0000

Amazon DocumentDB is a scalable, fully managed document database service designed to work with MongoDB workloads. While it behaves similarly to MongoDB in many ways, it has some critical limitations and behavioral nuances — especially when you're dealing with large datasets and heavy aggregations.

In this post, I’ll walk you through how to:

Populate a large collection with test data
Run a heavy $lookup operation multiple times
Monitor long-running operations with db.currentOp()
Filter and analyze slow queries
Kill operations directly from the shell

📚 Step 1: Insert 100,000 Books into DocumentDB

To simulate load and test joins and performance, we first populate the books collection with randomized data.

// Choose the database
use mylibrary;

// Define helper functions (basic randomizer)
function getRandomElement(arr) {
  return arr[Math.floor(Math.random() * arr.length)];
}

function getRandomInt(min, max) {
  return Math.floor(Math.random() * (max - min + 1)) + min;
}

var genres = ['Fiction', 'Non-Fiction', 'Science', 'Fantasy', 'Biography', 'History', 'Mystery', 'Romance'];
var formats = ['epub', 'pdf', 'txt', 'audio'];
var firstNames = ['John', 'Mary', 'Alice', 'Robert', 'Linda', 'Michael', 'Sarah', 'David'];
var lastNames = ['Smith', 'Johnson', 'Williams', 'Brown', 'Jones', 'Miller', 'Davis', 'Garcia'];

for (var i = 0; i < 100000; i++) {
  var author = getRandomElement(firstNames) + " " + getRandomElement(lastNames);
  var title = "Book Title " + (i + 1);
  var book = {
    author: author,
    title: "title,"
    genre: getRandomElement(genres),
    format: getRandomElement(formats),
    num_of_pages: getRandomInt(50, 1000),
    year_when_published: getRandomInt(1900, 2025)
  };

  db.books.insertOne(book);
}

After running the script:

db.books.find().count();
100000

🔁 Step 2: Run a Heavy Self-Join Using `$lookup` Three Times

We now run a self-join query that joins the books collection to itself three times based on the title field.

db.books.aggregate([
  {
    $lookup: {
      from: "books",
      localField: "title",
      foreignField: "title",
      as: "join1"
    }
  },
  {
    $lookup: {
      from: "books",
      localField: "title",
      foreignField: "title",
      as: "join2"
    }
  },
  {
    $lookup: {
      from: "books",
      localField: "title",
      foreignField: "title",
      as: "join3"
    }
  }
])

This can result in long-running operations, especially on large collections like ours. Now let's inspect how to monitor them.

🔍 Step 3: Use `db.currentOp()` to Monitor Active Queries

db.currentOp()

Example Output:

rs0 [direct: primary] libraries> db.currentOp()
{
  inprog: [
    {
      active: true,
      killPending: false,
      effectiveUsers: [ { user: 'myadmin', db: 'admin' } ],
      opid: 3427458,
      ns: 'mylibrary;.books',
      command: { cursor: Long('337448504967') },
      lsid: {
        id: UUID('54c4f794-c707-4779-8e37-be9ff53a1d3d'),
        uid: Binary.createFromBase64('H8snps2xdBYX5lwliJwTwIf7Mh1/ajgry8q8kHMbKvs=', 0)
      },
      originatingCommand: {
        aggregate: 'books',
        pipeline: [
          {
            '$lookup': {
              from: 'books',
              localField: 'title',
              foreignField: 'title',
              as: 'join1'
            }
          },
          {
            '$lookup': {
              from: 'books',
              localField: 'title',
              foreignField: 'title',
              as: 'join2'
            }
          },
          {
            '$lookup': {
              from: 'books',
              localField: 'title',
              foreignField: 'title',
              as: 'join3'
            }
          }
        ],
        cursor: {},
        lsid: { id: UUID('54c4f794-c707-4779-8e37-be9ff53a1d3d') },
        '$readPreference': { mode: 'primaryPreferred' },
        '$db': 'mylibrary;'
      },
      op: 'command',
      '$db': 'mylibrary;',
      secs_running: 15,
      microsecs_running: Long('15646511'),
      threadId: 48599,
      desc: 'Cursor',
      WaitState: 'Other'
    },
    {
      client: '172.31.80.163:56514',
      desc: 'Conn',
      active: true,
      killPending: false,
      effectiveUsers: [ { user: 'myadmin', db: 'admin' } ],
      opid: 3427498,
      ns: 'admin.$cmd',
      command: {
        aggregate: 1,
        pipeline: [
          {
            '$currentOp': {
              allUsers: true,
              idleConnections: false,
              truncateOps: false
            }
          },
          { '$match': {} }
        ],
        cursor: {},
        lsid: { id: UUID('d2c06e64-4793-4e0a-88de-e98fb24eb98e') },
        '$readPreference': { mode: 'primaryPreferred' },
        '$db': 'admin'
      },
      lsid: {
        id: UUID('d2c06e64-4793-4e0a-88de-e98fb24eb98e'),
        uid: Binary.createFromBase64('H8snps2xdBYX5lwliJwTwIf7Mh1/ajgry8q8kHMbKvs=', 0)
      },
      op: 'command',
      '$db': 'admin',
      secs_running: 0,
      microsecs_running: Long('220'),
      threadId: 51335,
      clientMetaData: {
        application: { name: 'mongosh 2.5.1' },
        driver: { name: 'nodejs|mongosh', version: '6.16.0|2.5.1' },
        platform: 'Node.js v20.19.1, LE',
        os: {
          name: 'linux',
          architecture: 'x64',
          version: '3.10.0-327.22.2.el7.x86_64',
          type: 'Linux'
        }
      }
    }
  ],
  ok: 1
}
rs0 [direct: primary] libraries>

⏱️ Step 4: Filter for Long-Running Queries (>10 Seconds)

db.currentOp({
  active: true,
  secs_running: { $gt: 10 }
})

This is useful if you want to isolate only the slow or stuck operations.

🧠 Step 5: Extract Only Key Fields from Long Queries

To get a more readable view, you can map just the relevant fields:

db.currentOp({
  active: true,
  secs_running: { $gt: 10 }
}).inprog.map(op => ({
  active: op.active,
  opid: op.opid,
  ns: op.ns,
  originatingCommand: op.originatingCommand,
  secs_running: op.secs_running
}));

🛑 Step 6: Kill a Long-Running Operation

Once you identify the opid of the slow query, you can terminate it with:

db.killOp(opid)

Example:

db.killOp(3427458)

This immediately stops the operation and frees up system resources.

🚀 Final Thoughts

Amazon DocumentDB can handle large-scale document operations, but heavy aggregation pipelines (especially self-joins with $lookup) can easily cause bottlenecks. By combining $lookup, db.currentOp(), and db.killOp(), you can simulate and manage real-world production issues like slow queries and runaway operations.

Use this workflow to monitor your workloads and keep your DocumentDB clusters responsive.

Playing with Indexes in MongoDB

Dmitry Romanoff — Sat, 12 Jul 2025 07:53:33 +0000

Indexes in MongoDB can dramatically improve the performance of your queries—if used right. In this article, we'll walk through a practical example of how indexes affect query performance, using a simple book database.

We'll go from inserting mock data to analyzing query execution plans, adding indexes, and understanding index limitations.

1. Generate Sample Data

Let's start by populating a collection with mock book data:

// Choose the database
use mylibrary;

// Define helper functions (basic randomizer)
function getRandomElement(arr) {
  return arr[Math.floor(Math.random() * arr.length)];
}

function getRandomInt(min, max) {
  return Math.floor(Math.random() * (max - min + 1)) + min;
}

var genres = ['Fiction', 'Non-Fiction', 'Science', 'Fantasy', 'Biography', 'History', 'Mystery', 'Romance'];
var formats = ['epub', 'pdf', 'txt', 'audio'];
var firstNames = ['John', 'Mary', 'Alice', 'Robert', 'Linda', 'Michael', 'Sarah', 'David'];
var lastNames = ['Smith', 'Johnson', 'Williams', 'Brown', 'Jones', 'Miller', 'Davis', 'Garcia'];

for (var i = 0; i < 1000; i++) {
  var author = getRandomElement(firstNames) + " " + getRandomElement(lastNames);
  var title = "Book Title " + (i + 1);
  var book = {
    author: author,
    title: "title,"
    genre: getRandomElement(genres),
    format: getRandomElement(formats),
    num_of_pages: getRandomInt(50, 1000),
    year_when_published: getRandomInt(1900, 2025)
  };

  db.books.insertOne(book);
}

2. Check Query Execution Without Index

Let’s look at the query plan for a simple .find() operation:

db.books.find().explain("executionStats")

{
  queryPlanner: {
    plannerVersion: 1,
    namespace: 'mylibrary;.books',
    winningPlan: { stage: 'COLLSCAN' }
  },
  executionStats: {
    executionSuccess: true,
    executionTimeMillis: '0.794',
    planningTimeMillis: '0.079',
    executionStages: {
      stage: 'COLLSCAN',
      nReturned: '1915',
      executionTimeMillisEstimate: '0.614'
    }
  },
  ...
}

This uses a COLLSCAN (collection scan), which means MongoDB is scanning every document—inefficient for large datasets.

3. Create an Index on the `author` Field

Let’s improve performance by indexing the author field:

db.books.createIndex({ author: 1 })

4. List All Indexes

db.books.getIndexes()

[
  { v: 4, key: { _id: 1 }, name: '_id_', ns: 'mylibrary;.books' },
  {
    v: 4,
    key: { author: 1 },
    name: 'author_1',
    ns: 'mylibrary;.books'
  }
]

5. Query by Author

db.books.find({ author: "John Smith" }).limit(5).pretty()

6. Query Plan for Author Search

db.books.find({ author: "John Smith" }).explain("executionStats")

{
  queryPlanner: {
    ...
    winningPlan: { stage: 'IXSCAN', indexName: 'author_1', direction: 'forward' }
  },
  executionStats: {
    executionSuccess: true,
    executionTimeMillis: '0.293',
    ...
  }
}

✅ The query now uses an IXSCAN (index scan) which is much faster!

7. Query by Title (No Index)

db.books.find({ title: "Book Title 10" }).explain("executionStats")

{
  queryPlanner: {
    ...
    winningPlan: { stage: 'COLLSCAN' }
  },
  executionStats: {
    executionSuccess: true,
    executionTimeMillis: '1.775',
    ...
  }
}

Still doing a full collection scan—not ideal.

8. Query by Author + Title

db.books.find({ author: "John Smith", title: "Book Title 10" }).explain("executionStats")

{
  queryPlanner: {
    ...
    winningPlan: { stage: 'IXSCAN', indexName: 'author_1', direction: 'forward' }
  },
  ...
}

This is better than a full scan, but still not ideal if you're frequently querying by title.

9. Compound Index on `year_when_published` and `title`

db.books.createIndex({ year_when_published: 1, title: 1 })

db.books.getIndexes()

[
  ...
  {
    v: 4,
    key: { year_when_published: 1, title: 1 },
    name: 'year_when_published_1_title_1',
    ns: 'mylibrary;.books'
  }
]

Query plan now improves:

db.books.find({ title: "Book Title 10" }).explain("executionStats")

{
  queryPlanner: {
    ...
    winningPlan: {
      stage: 'IXSCAN',
      indexName: 'year_when_published_1_title_1',
      ...
    }
  },
  ...
}

10. Index Size Limitations

Let's test MongoDB's index size limit by trying to insert a document with a huge title:

var longTitle = "A".repeat(5000); // 5000-character string

db.books.insertOne({
  title: longTitle,
  author: "Test Author",
  year_when_published: 2020,
  genre: "Fiction",
  format: "pdf",
  num_of_pages: 300
});

Error:

MongoServerError: key too large to index. Index name: year_when_published_1_title_1

🚫 MongoDB’s index key size is limited (1024 bytes). Long strings in indexed fields can break inserts if the index becomes too large.

🔚 Conclusion

Indexes can drastically improve query performance.
Use .explain("executionStats") to analyze and optimize.
Compound indexes are powerful, but mindful of their field sizes.
Avoid indexing large strings unless absolutely necessary.

Randomized Data Sampling and Aggregation in MongoDB

Dmitry Romanoff — Sat, 12 Jul 2025 07:13:41 +0000

As developers, working with large datasets is often a part of our daily routines. MongoDB, a popular NoSQL database, offers powerful aggregation and sampling capabilities that help us interact with large datasets efficiently. In this article, we will go through various MongoDB commands that demonstrate random sampling, data aggregation, and sorting in MongoDB, focusing on how we can utilize them to generate, query, and analyze data in creative ways.

1. Inserting Randomized Data

Let's start by inserting a set of random books into a books collection. We'll simulate a library database, where each book has an author, title, genre, format, page count, and year of publication.

// Choose the database
use mylibrary;

// Define helper functions (basic randomizer)
function getRandomElement(arr) {
  return arr[Math.floor(Math.random() * arr.length)];
}

function getRandomInt(min, max) {
  return Math.floor(Math.random() * (max - min + 1)) + min;
}

var genres = ['Fiction', 'Non-Fiction', 'Science', 'Fantasy', 'Biography', 'History', 'Mystery', 'Romance'];
var formats = ['epub', 'pdf', 'txt', 'audio'];
var firstNames = ['John', 'Mary', 'Alice', 'Robert', 'Linda', 'Michael', 'Sarah', 'David'];
var lastNames = ['Smith', 'Johnson', 'Williams', 'Brown', 'Jones', 'Miller', 'Davis', 'Garcia'];

for (var i = 0; i < 1000; i++) {
  var author = getRandomElement(firstNames) + " " + getRandomElement(lastNames);
  var title = "Book Title " + (i + 1);
  var book = {
    author: author,
    title: title,
    genre: getRandomElement(genres),
    format: getRandomElement(formats),
    num_of_pages: getRandomInt(50, 1000),
    year_when_published: getRandomInt(1900, 2025)
  };

  db.books.insertOne(book);
}

In this script, we populate a MongoDB collection called books with 1000 randomly generated entries. Each book's data is randomized for the author name, genre, format, number of pages, and publication year.

2. Counting Documents in a Collection

Once the data is inserted, we may want to know how many documents exist in the books collection. MongoDB provides an easy-to-use count() method to get the total number of documents.

db.books.count()

This command will return the total count of documents in the books collection. It’s a simple way to verify that 1000 books were inserted successfully.

3. Sampling Random Books

What if you want to randomly sample a set of books? The $sample stage in the aggregation pipeline is perfect for this.

db.books.aggregate([
  { $sample: { size: 10 } },        // Randomly sample 10 documents (1% of 1000)
  { $sort: { author: 1 } }          // Sort the sampled documents by author (ascending)
])

In this example, we're randomly selecting 10 books and sorting them by author in ascending order. The $sample stage allows us to work with a random subset of the data, which can be useful for testing, statistics, or when you want to generate random previews.

4. Randomly Skipping and Limiting Results

If you need to fetch a single random document, you can use a combination of $skip and $limit. Here's how:

db.books.aggregate([ 
  { 
    $skip : db.books.countDocuments() * Math.random() 
  }, 
  { 
    $limit: 1
  }
]);

This query will randomly skip a certain number of documents, based on a random fraction of the total document count, and then return just one document. This approach ensures that you are retrieving a completely random book from the collection.

5. Aggregating Data by Format

To analyze how many books exist in each format (e.g., pdf, epub, audio), you can use the $group stage to aggregate and the $sort stage to order the results.

db.books.aggregate([
  { $sample: { size: 50 } },               // Take a 5% sample (assuming ~1000 docs)
  { $group: { 
      _id: "$format", 
      count: { $sum: 1 } 
    } 
  },
  { $sort: { count: -1 } }                 // Sort by most common format
])

This query randomly samples 50 books and then groups them by format. The result shows the most common formats in the collection, sorted in descending order of frequency.

6. Aggregating Data by Genre

Similar to the format-based aggregation, we can also aggregate books based on their genre to determine which genres are most popular.

db.books.aggregate([
  { $sample: { size: 100 } },                // Take a 10% sample (assuming ~1000 docs)
  { $group: { 
      _id: "$genre", 
      count: { $sum: 1 }
    } 
  },
  { $sort: { count: -1 } }                   // Sort by most common genre
])

This aggregation pipeline randomly samples 100 books, groups them by genre, and counts the occurrences of each genre. The results will show the most common genres in the database.

7. Dynamic Sampling Based on Document Count

In some cases, you may want to sample a dynamic percentage of documents, say 10% of the total documents. You can calculate the sample size dynamically as shown here:

var totalCount = db.books.count();

// Calculate 10% of the total (you can change 0.1 to any fraction you need)
var sampleSize = Math.floor(totalCount * 0.1);

// Perform the aggregation with the calculated sample size
db.books.aggregate([
  { $sample: { size: sampleSize } },        // Randomly sample 10% of the documents
  { $sort: { author: 1 } }                  // Sort by author (ascending)
])

In this query, we dynamically calculate 10% of the total document count and then randomly sample that many books. This allows for flexible sampling based on the collection size.

8. Combining Genre and Format Aggregation

Finally, let’s combine both genre and format in our aggregation to understand the most popular format within each genre.

db.books.aggregate([
  { $sample: { size: 100 } },  // Sample 10% (assuming ~1000 docs)
  { 
    $group: {
      _id: { genre: "$genre", format: "$format" },  // Group by genre and format
      count: { $sum: 1 }                           // Count how many books in each format within a genre
    }
  },
  { 
    $sort: { "count": -1 }  // Sort by highest count (most popular format per genre)
  },
  {
    $group: {
      _id: "$_id.genre",                  // Group by genre
      mostPopularFormat: { $first: "$_id.format" },  // Get the most popular format per genre
      count: { $first: "$count" }         // Get the count of that most popular format
    }
  },
  { 
    $sort: { "count": -1 }  // Sort by count (most popular genre)
  }
])

This complex aggregation allows you to find the most popular format within each genre and sort genres by their popularity. It's a useful query if you want insights into book preferences across genres and formats.

Conclusion

In this article, we’ve explored a variety of MongoDB commands for sampling and aggregating data. Whether you’re generating random data, sorting by author, or analyzing formats and genres, these MongoDB aggregation commands provide powerful tools for data analysis. As you work with larger datasets, these techniques will help you extract meaningful insights efficiently.

MongoDB’s aggregation framework is highly flexible, and by combining stages like $sample, $group, $sort, and $skip, you can solve a wide range of problems in data processing.