DEV Community: Habil BOZALİ

Setting Up SSL on AWS Elastic Beanstalk Single Instances

Habil BOZALİ — Sat, 10 May 2025 18:08:05 +0000

Photo by Taylor Vick on Unsplash

Deploying applications on AWS Elastic Beanstalk simplifies many aspects of running your web applications, but setting up SSL/TLS on single-instance environments can get a bit tricky. In this article, we’ll explore two practical approaches to implementing SSL for your Elastic Beanstalk single instances.

Why SSL on Single Instances Is Challenging

AWS Elastic Beanstalk single instances don’t come with built-in SSL termination like their load-balanced counterparts. When you deploy a web application in a single-instance environment, you’re essentially working with a standalone EC2 instance without the benefits of an Application Load Balancer to handle SSL termination.

However, there are compelling reasons to use single instances:

Cost efficiency : Avoiding load balancer costs for smaller applications
Simplified architecture : Reduced complexity for development or test environments
Direct instance access : Easier debugging and monitoring
Lower traffic needs : Many applications don’t require the scalability of multiple instances

Let’s dive into two effective solutions for enabling SSL on your Elastic Beanstalk single instances.

Solution 1: Using .ebextensions with Let’s Encrypt

This approach involves configuring your Elastic Beanstalk environment with custom .ebextensions scripts to automatically set up and renew SSL certificates using Let's Encrypt.

Step 1: Create the .ebextensions Directory

In your application’s root directory, create a .ebextensions folder if it doesn't already exist:

mkdir -p .ebextensions

Step 2: Create the Let’s Encrypt Configuration Script

Create a configuration file that will handle the installation and setup of Let’s Encrypt:

# .ebextensions/01_ssl_setup.config
packages:
  yum:
    mod24_ssl: []
    epel-release: []

container_commands:
  10_install_certbot:
    command: "sudo yum install -y certbot python-certbot-apache"
  20_get_certificate:
    command: "sudo certbot certonly --standalone --debug --non-interactive --email ${EMAIL} --agree-tos --domains ${DOMAIN} --keep-until-expiring --pre-hook \"service httpd stop\" --post-hook \"service httpd start\""
    env:
      EMAIL: "your-email@example.com"
      DOMAIN: "yourdomain.com"
  30_link_cert:
    command: "ln -sf /etc/letsencrypt/live/${DOMAIN}/cert.pem /etc/pki/tls/certs/server.crt"
    env:
      DOMAIN: "yourdomain.com"
  40_link_key:
    command: "ln -sf /etc/letsencrypt/live/${DOMAIN}/privkey.pem /etc/pki/tls/certs/server.key"
    env:
      DOMAIN: "yourdomain.com"
  50_enable_ssl_conf:
    command: "sed -i 's/#LoadModule ssl_module/LoadModule ssl_module/' /etc/httpd/conf.modules.d/00-ssl.conf"
  60_create_ssl_conf:
    command: |
      cat > /etc/httpd/conf.d/ssl.conf << 'EOL'
      Listen 443
      <VirtualHost *:443>
        ServerName ${DOMAIN}
        DocumentRoot /var/www/html
        SSLEngine on
        SSLCertificateFile /etc/pki/tls/certs/server.crt
        SSLCertificateKeyFile /etc/pki/tls/certs/server.key
        SSLCertificateChainFile /etc/letsencrypt/live/${DOMAIN}/chain.pem
        <Directory /var/www/html>
          AllowOverride All
          Require all granted
        </Directory>
      </VirtualHost>
      EOL
    env:
      DOMAIN: "yourdomain.com"
files:
  "/etc/cron.d/certbot-renew":
    mode: "000644"
    owner: root
    group: root
    content: |
      0 0,12 * * * root certbot renew --standalone --pre-hook "service httpd stop" --post-hook "service httpd start" --quiet

Make sure to replace your-email@example.com and yourdomain.com with your actual email and domain.

Step 3: Create a Configuration to Handle HTTP to HTTPS Redirect

# .ebextensions/02_ssl_redirect.config
files:
  "/etc/httpd/conf.d/http_redirect.conf":
    mode: "000644"
    owner: root
    group: root
    content: |
      <VirtualHost *:80>
        ServerName yourdomain.com
        Redirect permanent / https://yourdomain.com/
      </VirtualHost>

Again, replace yourdomain.com with your actual domain.

Step 4: Deploy Your Application

Deploy your application with these .ebextensions files:

eb deploy

How This Solution Works

The configuration installs the necessary SSL packages and Let’s Encrypt certbot
It requests a certificate from Let’s Encrypt for your domain
The Apache web server is configured to use the certificate
A cron job is set up to automatically renew the certificate before expiration
HTTP traffic is redirected to HTTPS

Potential Issues and Solutions

Port 80/443 accessibility : Ensure your security group allows inbound traffic on ports 80 and 443.

Certificate renewal failures : The pre-hook and post-hook in the cron job temporarily stop Apache during renewal to free up port 80. Monitor renewal logs at /var/log/letsencrypt/ if you encounter issues.

DNS configuration : Your domain must be correctly pointing to your Elastic Beanstalk instance’s IP address for Let’s Encrypt validation to succeed.

Solution 2: Using a Load Balancer with CNAME

This approach involves creating a single-instance environment but fronting it with a load balancer specifically for SSL termination.

Step 1: Create a Load Balanced Environment

Instead of creating a single-instance environment, create a load-balanced environment with a minimum and maximum of 1 instance:

eb create my-environment --elb-type application --single

Or through the AWS Management Console:

Create a new environment
Choose “Web server environment”
Under “Platform”, select your platform of choice
Under “Configuration presets”, select “Custom configuration”
In the “Capacity” section, select “Load balanced”
Set both “Min instances” and “Max instances” to 1

Step 2: Configure SSL on the Load Balancer

Navigate to the EC2 Console > Load Balancers
Select your environment’s load balancer
Add a listener for HTTPS (port 443)
Add your SSL certificate (you can use AWS Certificate Manager for free public certificates)

{
  "Listeners": [
    {
      "Protocol": "HTTPS",
      "Port": 443,
      "DefaultActions": [
        {
          "Type": "forward",
          "TargetGroupArn": "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/awseb-AWSEB-ABCDEFGHIJ/0123456789abcdef"
        }
      ],
      "SslPolicy": "ELBSecurityPolicy-2016-08",
      "Certificates": [
        {
          "CertificateArn": "arn:aws:acm:us-east-1:123456789012:certificate/12345678-1234-1234-1234-123456789012"
        }
      ]
    }
  ]
}

Step 3: Set Up CNAME Records

In your DNS provider:

Create a CNAME record pointing your domain to your Elastic Beanstalk environment URL
For example: yourdomain.com → my-environment.elasticbeanstalk.com

Step 4: Configure HTTP to HTTPS Redirect (Optional)

Create a listener rule for port 80 that redirects to HTTPS:

In the EC2 Console, go to the Load Balancer
Add a listener for HTTP (port 80)
Add a redirect action to HTTPS (port 443)

How This Solution Works

The load balancer handles SSL termination
Traffic is encrypted between clients and the load balancer
The load balancer forwards traffic to your single EC2 instance
You benefit from AWS Certificate Manager’s free certificates and automatic renewals
The architecture remains simple with just one instance

Considerations for This Approach

Cost implications : This approach adds the cost of a load balancer (approximately $16–20/month)

Advantage : AWS manages certificate renewal automatically

Simplified management : No need to manage SSL configuration on the instance itself

Comparing the Two Approaches

Which Solution Should You Choose?

Choose Solution 1 (.ebextensions with Let’s Encrypt) if:

Cost is a primary concern
You’re comfortable managing Linux configurations
You want to maintain a truly single-instance architecture

Choose Solution 2 (Load Balancer with CNAME) if:

You prefer managed services with less operational overhead
Budget allows for an additional ~$20/month
You might need to scale in the future
You want automatic certificate renewal without custom scripts

Implementation Best Practices

Regardless of which solution you choose, follow these best practices:

Security Considerations

Keep your Elastic Beanstalk platform updated
Use strong SSL/TLS protocols (TLS 1.2 or higher)
Configure security headers in your application
Set appropriate security groups to limit access

Monitoring

Set up CloudWatch alarms for certificate expiration (for Solution 1)
Monitor HTTP status codes to ensure redirects are working properly
Set up alerts for SSL/TLS failures

Documentation

Document your SSL setup thoroughly, including:

Certificate renewal dates and processes
DNS configuration
Load balancer or instance configuration details
Troubleshooting steps for common issues

Conclusion

Setting up SSL on AWS Elastic Beanstalk single instances doesn’t have to be complicated. Both approaches outlined in this article provide secure ways to serve your application over HTTPS.

The .ebextensions approach with Let’s Encrypt offers a cost-effective solution with more hands-on management, while the load balancer approach provides a more managed experience at a higher cost.

Choose the approach that best aligns with your technical comfort level, budget constraints, and operational preferences. Either way, you’ll be providing your users with the secure experience they expect and deserve.

See you in the next article! 👻

The AWS Well-Architected Framework: A Complete Guide

Habil BOZALİ — Sun, 13 Apr 2025 19:18:43 +0000

Photo by Jess Bailey on Unsplash

As more companies pivot towards major providers like AWS in the cloud computing landscape, success isn’t merely about using services — it’s about using them correctly. This is precisely where the AWS Well-Architected Framework comes into play. This framework provides a comprehensive guide to help you evaluate and improve your cloud architecture over time.

What is the AWS Well-Architected Framework?

The AWS Well-Architected Framework is a set of principles and best practices that AWS has codified over years of experience in designing, implementing, and maintaining cloud-based systems. The framework helps cloud architects and application developers build secure, high-performing, resilient, and efficient systems.

The framework is built upon five interrelated pillars:

Operational Excellence : Running and monitoring systems to deliver business value
Security : Protecting information and systems
Reliability : Ensuring a workload performs its intended function correctly and consistently
Performance Efficiency : Using computing resources efficiently
Cost Optimization : Avoiding unnecessary costs

A Deeper Look at the Five Pillars

Each pillar of the Well-Architected Framework focuses on a specific aspect of cloud architecture and provides a set of design principles and best practices:

1. Operational Excellence

This pillar focuses on running and monitoring systems to deliver business value and continually improving processes and procedures.

Key Principles:

Perform operations as code
Make frequent, small, reversible changes
Refine operations procedures frequently
Anticipate failure
Learn from all operational failures

Best Practices:

Implement Infrastructure as Code (IaC) using AWS CloudFormation or Terraform
Establish CI/CD pipelines for automated deployments
Implement comprehensive logging and monitoring with services like CloudWatch
Create runbooks and playbooks for standard procedures and incident response

2. Security

The security pillar protects information, systems, and assets while delivering business value through risk assessments and mitigation strategies.

Key Principles:

Implement a strong identity foundation
Enable traceability
Apply security at all layers
Automate security best practices
Protect data in transit and at rest
Keep people away from data
Prepare for security events

Best Practices:

Use IAM to implement the principle of least privilege
Enable MFA for all users, especially those with privileged access
Implement network security using security groups, NACLs, and VPCs
Encrypt data at rest and in transit
Use AWS GuardDuty for threat detection
Perform regular security assessments and penetration tests

3. Reliability

This pillar emphasizes a system's ability to recover from infrastructure or service disruptions, dynamically acquire computing resources to meet demand, and mitigate disruptions such as misconfigurations or transient network issues.

Key Principles:

Test recovery procedures
Automatically recover from failure
Scale horizontally to increase aggregate system availability
Stop guessing capacity
Manage change in automation

Best Practices:

Design with redundancy and high availability in mind
Use multiple Availability Zones and regions where appropriate
Implement auto-scaling to handle load variations
Create backup and restore strategies
Use fault isolation to protect the entire system
Design with graceful degradation in mind

4. Performance Efficiency

This pillar focuses on using computing resources efficiently to meet system requirements and maintaining that efficiency as demand changes and technologies evolve.

Key Principles:

Democratize advanced technologies
Go global in minutes
Use serverless architectures
Experiment more often
Mechanical sympathy

Best Practices:

Select the right resource types and sizes based on workload requirements
Monitor performance and optimize over time
Use caching to improve performance and reduce database load
Deploy in multiple regions to provide lower latency to global users
Leverage managed services to reduce operational overhead
Experiment with new technologies and approaches

5. Cost Optimization

This pillar focuses on avoiding unnecessary costs by understanding spending over time and controlling fund allocation.

Key Principles:

Adopt a consumption model
Measure overall efficiency
Stop spending money on undifferentiated heavy lifting
Analyze and attribute expenditure

Best Practices:

Implement resource tagging strategies for cost allocation
Use reserved instances and savings plans for predictable workloads
Right-size resources based on actual usage patterns
Automate resource optimization with AWS Trusted Advisor
Implement lifecycle policies for data storage
Review and adjust your architecture as AWS introduces new services

Why is the AWS Well-Architected Framework Important?

In today’s competitive landscape, simply migrating to the cloud isn’t enough. Organizations need to leverage the cloud’s full potential while maintaining security, reliability, and cost-effectiveness. Here’s why the Well-Architected Framework matters:

Risk Identification and Mitigation

The framework helps identify potential issues early in the design process, preventing costly remediation later. By asking critical questions across each pillar, you can discover architectural weaknesses before they become operational problems.

For example, considering the security pillar early might lead you to implement proper encryption and access controls from the start, rather than retrofitting them after a security incident occurs.

Consistency Across Teams

As organizations grow, different teams might adopt varying approaches to cloud architecture. The Well-Architected Framework provides a common language and set of standards that ensures consistency across development teams, resulting in:

Reduced operational overhead
Simplified maintenance
Better knowledge sharing
Lower risk of configuration drift

Continuous Improvement Path

Cloud architecture isn’t static — it evolves with business needs and technological advancements. The framework encourages regular assessments of your workloads, helping you identify areas for improvement as your requirements change.

How to Leverage the Well-Architected Framework

Adopting the framework doesn’t have to be overwhelming. Here’s a practical approach:

Step 1: Conduct a Well-Architected Review

Start by evaluating your existing workloads against the framework’s five pillars. AWS provides a free Well-Architected Tool in the management console that guides you through this process with a series of questions for each pillar.

{
    "WorkloadName": "E-commerce Platform",
    "ReviewDate": "2023-04-13",
    "PillarReviews": {
        "OperationalExcellence": {
            "RiskLevel": "MEDIUM",
            "ImprovementAreas": [
                "Implement infrastructure as code",
                "Enhance monitoring and alerting"
            ]
        },
        "Security": {
            "RiskLevel": "HIGH",
            "ImprovementAreas": [
                "Enable MFA for all IAM users",
                "Implement data encryption at rest"
            ]
        }
        // Other pillars...
    }
}

Step 2: Prioritize Improvements

Once you’ve identified areas for improvement, prioritizing them effectively is crucial for success. Consider the following factors when determining which improvements to tackle first:

Business Impact

Revenue Protection : Prioritize issues that could impact revenue generation or customer retention
Regulatory Compliance : Address improvements needed to maintain compliance with relevant regulations
Brand Protection : Focus on issues that could damage your reputation if left unaddressed

Risk Assessment

High : Critical vulnerabilities or design flaws that could lead to system failure, data breach, or significant financial loss
Medium : Issues that impact performance, efficiency, or partial functionality
Low : Opportunities for optimization that don’t pose immediate threats

Implementation Factors

Effort Required : Consider the time, resources, and expertise needed
Disruption Level : Assess the potential impact on ongoing operations during implementation
Dependencies : Identify improvements that serve as foundations for other changes

Pillar-Based Priority Framework

When prioritizing across pillars, consider this general hierarchy (though this may vary based on your specific business context):

Security issues (especially HIGH risk) typically deserve immediate attention due to potential regulatory, financial, and reputational impacts
Reliability improvements that address potential service disruptions
Operational Excellence enhancements that improve your ability to respond to issues
Performance Efficiency optimizations that affect customer experience
Cost Optimization opportunities (unless your organization is under specific cost pressures)

Prioritization Matrix Example

Here’s a simple matrix approach to visualize priorities:

This approach helps balance urgency, impact, and practical considerations to create a workable improvement roadmap.

Step 3: Implement Changes Iteratively

Don’t try to address everything at once. Instead:

Create a roadmap for improvements
Implement changes in small, manageable increments
Measure the impact of each change
Document lessons learned

Step 4: Make Well-Architected Reviews Regular

Schedule regular reviews (quarterly or bi-annually) to ensure your architecture continues to align with best practices as your workloads evolve.

The Benefits of Following Well-Architected Principles

Organizations that embrace the Well-Architected Framework typically experience several tangible benefits:

Reduced Operational Issues

By designing systems according to well-established principles, you’ll encounter fewer unexpected outages, performance bottlenecks, and security incidents.

Lower Total Cost of Ownership

The Cost Optimization pillar helps identify resource waste and inefficiencies. Organizations often discover they can achieve the same performance at lower costs after applying Well-Architected recommendations.

For example, one e-commerce company reduced their monthly AWS bill by 42% after implementing auto-scaling based on usage patterns and switching from on-demand to reserved instances where appropriate.

Enhanced Security Posture

The Security pillar guides organizations toward comprehensive security practices that protect data, systems, and assets. This proactive approach helps prevent breaches and compliance issues that could otherwise be costly and damage your reputation.

Faster Time to Market

Well-architected systems are easier to maintain and extend. Teams spend less time troubleshooting issues and more time delivering new features and capabilities.

The Risks of Ignoring Well-Architected Principles

Conversely, organizations that neglect architectural best practices often face several challenges:

Technical Debt Accumulation

Without a framework for evaluation, architectural shortcuts and compromises can accumulate, making systems increasingly difficult and expensive to maintain over time.

Unpredictable Costs

Poor architectural decisions can lead to resource inefficiencies and unexpected cost spikes. For instance, improperly configured storage or databases might grow unchecked, resulting in escalating expenses.

Security Vulnerabilities

Neglecting security considerations can expose organizations to data breaches, unauthorized access, and compliance violations. The average cost of a data breach now exceeds $4.5 million, making this risk particularly significant.

Reliability Issues

Systems designed without reliability in mind are prone to outages, data loss, and inconsistent performance — all of which can directly impact customer satisfaction and revenue.

Consider a financial services company that experienced a four-hour outage due to a single point of failure in their architecture. The incident resulted in:

$2.3 million in lost transactions
Customer compensation costs
Reputational damage
Regulatory scrutiny

A Well-Architected review would have identified this vulnerability before it became a costly problem.

Practical Implementation Tips

To make the most of the Well-Architected Framework, consider these practical tips:

Start Small

If you’re new to the framework, begin by applying it to a single, important workload rather than attempting to assess everything at once.

Involve Cross-Functional Teams

Well-Architected reviews are most effective when they involve perspectives from development, operations, security, and business stakeholders.

Automate Compliance Checks

Use tools like AWS Config Rules, CloudFormation Guard, or third-party solutions to automatically check your infrastructure against Well-Architected best practices.

# Example CloudFormation Guard rule to ensure encryption
let s3_buckets = Resources.*[Type == 'AWS::S3::Bucket']
rule s3_buckets_encrypted when %s3_buckets !empty {
  %s3_buckets.Properties.BucketEncryption.ServerSideEncryptionConfiguration[*].ServerSideEncryptionByDefault.SSEAlgorithm == "AES256" or
  %s3_buckets.Properties.BucketEncryption.ServerSideEncryptionConfiguration[*].ServerSideEncryptionByDefault.SSEAlgorithm == "aws:kms"
}

Document Decisions

When you choose to deviate from Well-Architected best practices, document your reasoning. This helps future team members understand the context behind architectural decisions.

Conclusion

The AWS Well-Architected Framework isn’t just a theoretical construct — it’s a practical tool that translates AWS’s vast experience into actionable guidance. By systematically applying these principles to your cloud workloads, you can build systems that are more secure, reliable, efficient, and cost-effective.

Remember that Well-Architected is a journey, not a destination. Cloud best practices continue to evolve, and your architecture should evolve alongside them. Regular reviews, incremental improvements, and a commitment to excellence will ensure your cloud infrastructure remains a competitive advantage rather than a liability.

Whether you’re just starting your cloud journey or looking to optimize existing workloads, the Well-Architected Framework provides a valuable compass to guide your way. The time and resources invested in aligning with these principles will pay dividends in reduced incidents, lower costs, and greater business agility.

Happy architecting! 👻

Automating Pi-hole Updates with Ansible

Habil BOZALİ — Sun, 16 Mar 2025 23:24:57 +0000

Photo by Ant Rozetsky on Unsplash

Automating Pi-hole Updates with Ansible

Managing multiple Pi-hole instances can become a time-consuming task, especially when it comes to regular updates. In this article, we’ll explore how to use Ansible to automate the process of updating Pi-hole installations across your network. This approach will save you time and ensure consistency across all your Pi-hole servers.

What is Pi-hole?

Pi-hole is a network-wide ad blocker that acts as a DNS sinkhole. It intercepts DNS requests on your network and blocks requests to known advertising and tracking domains, preventing ads from being downloaded. This not only improves your browsing experience but also:

Reduces bandwidth usage
Increases browsing speed
Enhances privacy by blocking tracking domains
Works on all devices on your network without needing to install software on each device

Pi-hole is typically installed on a Raspberry Pi (hence the name), but it can run on virtually any Linux distribution with minimal resources. It’s an excellent solution for home networks or small businesses looking to reduce ad traffic.

Why Ansible for Pi-hole Management?

When you’re managing one Pi-hole, manual updates are straightforward. However, as your infrastructure grows or if you maintain Pi-hole instances across different locations, the manual approach becomes:

Time-consuming
Error-prone
Difficult to track
Inconsistent

Ansible provides a solution with these benefits:

Automation : Execute the same tasks across multiple servers with a single command
Idempotency : Run playbooks multiple times without causing issues
Consistency : Ensure all systems are updated using the same procedure
Documentation : Your playbooks serve as living documentation of your update process
Scalability : Easily add new Pi-hole instances to your inventory

Setting Up the Environment

Let’s break down the process into clear steps:

Step 1: Install Ansible

First, ensure you have Ansible installed on your control node:

# On Debian/Ubuntu
sudo apt update
sudo apt install ansible

# On macOS with Homebrew
brew install ansible

# Verify installation
ansible --version

Step 2: Create Your Ansible Structure

Create a basic directory structure for your Ansible project:

mkdir -p pihole-ansible/inventory
mkdir -p pihole-ansible/playbooks
cd pihole-ansible

Step 3: Configure Your Inventory

Create an inventory file that lists your Pi-hole servers:

# inventory/hosts
[pizeros]
pihole1 ansible_host=192.168.1.100
pihole2 ansible_host=192.168.1.101
pihole3 ansible_host=192.168.1.102

[pizeros:vars]
ansible_user=pi

Step 4: Create the Group Variables

Create a group variables file to apply settings to all Pi-hole instances:

# inventory/group_vars/pizeros.yml
ansible_python_interpreter: /usr/bin/python3
ansible_become: yes
ansible_become_method: sudo

Step 5: Create the Update Playbook

Create a playbook that handles the Pi-hole update process:

# playbooks/update_pihole.yml
---
- hosts: pizeros
  become: true
  become_method: sudo
  become_user: root
  tasks:
    - name: Update package lists
      apt:
        update_cache: yes
      changed_when: false
- name: Upgrade all packages
      apt:
        upgrade: dist
        autoremove: yes
        autoclean: yes
    - name: Update Pi-hole
      command: pihole -up
      register: pihole_update_result
      changed_when: "'Everything is already up to date' not in pihole_update_result.stdout"
    - name: Display Pi-hole update results
      debug:
        var: pihole_update_result.stdout_lines

Step 6: Create a Convenience Script

For even easier updates, create a simple shell script:

# update.sh
#!/bin/bash
ansible-playbook -i inventory/hosts playbooks/update_pihole.yml

Make it executable:

chmod +x update.sh

Running the Update Process

Now that everything is set up, you can update all your Pi-hole instances with a single command:

./update.sh

Or, if you prefer to run the playbook directly:

ansible-playbook -i inventory/hosts playbooks/update_pihole.yml

Understanding the Playbook in Detail

Let’s break down what our update playbook does:

1. Package Updates

- name: Update package lists
  apt:
    update_cache: yes
  changed_when: false
- name: Upgrade all packages
  apt:
    upgrade: dist
    autoremove: yes
    autoclean: yes

These tasks:

Update the APT package cache
Perform a full distribution upgrade
Remove unnecessary packages
Clean the APT cache

2. Pi-hole Specific Update

- name: Update Pi-hole
  command: pihole -up
  register: pihole_update_result
  changed_when: "'Everything is already up to date' not in pihole_update_result.stdout"

This task:

Runs the Pi-hole update command (pihole -up)
Captures the output in a variable
Only registers as “changed” if an actual update occurred

3. Result Display

- name: Display Pi-hole update results
  debug:
    var: pihole_update_result.stdout_lines

This task displays the full output of the Pi-hole update process, making it easy to review what happened.

Advanced Customizations

Once you have the basic update process working, you can enhance your Ansible setup with these additional features:

Schedule Regular Updates

Use cron on your control node to schedule regular updates:

# Run updates every Sunday at 3:00 AM
0 3 * * 0 /path/to/pihole-ansible/update.sh > /path/to/logs/pihole-update.log 2>&1

Add Health Checks

Enhance your playbook with health checks after updates:

- name: Check Pi-hole status
  command: pihole status
  register: pihole_status
  changed_when: false
- name: Verify DNS resolution is working
  command: dig @localhost google.com
  register: dns_test
  changed_when: false
  failed_when: "'ANSWER SECTION' not in dns_test.stdout"

Add Notification System

Add tasks to notify you when updates are complete:

- name: Send update completion notification
  mail:
    host: smtp.gmail.com
    port: 587
    username: your_email@gmail.com
    password: "{{ email_password }}"
    to: admin@example.com
    subject: "Pi-hole update completed"
    body: "Updates have been applied to all Pi-hole instances.\n\n{{ pihole_update_result.stdout }}"
  when: pihole_update_result.changed
  no_log: true
  vars:
    ansible_python_interpreter: /usr/bin/python3
  delegate_to: localhost

Note: Store sensitive information like passwords in an encrypted Ansible vault.

Troubleshooting Common Issues

When using this automation, you might encounter some issues:

SSH Connection Problems

If you have SSH connection issues:

Verify your inventory has the correct IP addresses and usernames

Test the connection manually:

ansible pizeros -i inventory/hosts -m ping

Ensure SSH key authentication is set up:

ssh-copy-id pi@your_pihole_ip

Update Failures

If Pi-hole updates fail:

Ensure your Pi-hole instances have internet connectivity

Review Pi-hole logs for specific errors:

- name: Check Pi-hole logs   
  command: cat /var/log/pihole.log   
  register: pihole_logs   
  changed_when: false

Check disk space on your Pi-hole instances:

- name: Check available disk space
  shell: df -h /   
  register: disk_space   
  changed_when: false

Conclusion

Using Ansible to automate Pi-hole updates significantly improves manual processes, especially when managing multiple instances. This approach not only saves time but also ensures consistent updates across your entire network.

The playbooks and configurations in this article provide a solid foundation that you can customize to meet your specific needs. As you become more familiar with Ansible, you can expand your automation to include other aspects of Pi-hole management such as configuration changes, blocklist updates, or even full system backups.

Remember that automation is an investment that pays dividends over time. The initial setup may take some effort, but the long-term benefits of time savings and consistency are well worth it.

Happy automating and see you in the next article! 👻

Analyzing CloudFront Logs with Amazon Athena

Habil BOZALİ — Mon, 03 Mar 2025 19:48:34 +0000

Photo by Adrien on Unsplash

You are actively using the CloudFront service and noticed an increase in 4xx records in CloudFront statistics. This is usually due to incorrect configurations. However, for more details, you can open CloudFront logs and examine incoming requests. In this article, we will focus on how to perform log review most practically. Let’s start.

Why Athena for CloudFront Logs?

CloudFront generates detailed access logs that contain valuable information about requests made to your distribution. However, these logs are:

Delivered as compressed files to your S3 bucket
Generated in large volumes
Written in a specific format that’s not immediately queryable

Amazon Athena provides a serverless solution to analyze these logs using standard SQL, without the need to set up complex data processing pipelines.

Setting Up the Environment

Let’s break down the process into clear steps:

Step 1: Enable CloudFront Logging

First, ensure that CloudFront logging is enabled for your distribution:

Navigate to the CloudFront console
Select your distribution
Click on the “Behaviors” tab
Edit the behaviour settings and enable logging
Specify an S3 bucket for your logs

{  
 “Logging”: {  
 “Enabled”: true,  
 “IncludeCookies”: false,  
 “Bucket”: “your-logs-bucket.s3.amazonaws.com”,  
 “Prefix”: “cloudfront-logs/”  
 }  
}

Step 2: Create Athena Database and Table

Once logs are being delivered to your S3 bucket, you need to create a database and table in Athena:

Navigate to the Athena console
Create a new database:

CREATE DATABASE cloudfront_logs;

Create a table that maps to the CloudFront log format:

CREATE EXTERNAL TABLE IF NOT EXISTS cloudfront_logs.cf_access_logs (  
 `date` DATE,  
 time STRING,  
 location STRING,  
 bytes BIGINT,  
 request_ip STRING,  
 method STRING,  
 host STRING,  
 uri STRING,  
 status INT,  
 referrer STRING,  
 user_agent STRING,  
 query_string STRING,  
 cookie STRING,  
 result_type STRING,  
 request_id STRING,  
 host_header STRING,  
 request_protocol STRING,  
 request_bytes BIGINT,  
 time_taken FLOAT,  
 xforwarded_for STRING,  
 ssl_protocol STRING,  
 ssl_cipher STRING,  
 response_result_type STRING,  
 http_version STRING,  
 fle_status STRING,  
 fle_encrypted_fields INT,  
 c_port INT,  
 time_to_first_byte FLOAT,  
 x_edge_detailed_result_type STRING,  
 sc_content_type STRING,  
 sc_content_len BIGINT,  
 sc_range_start BIGINT,  
 sc_range_end BIGINT  
)  
ROW FORMAT DELIMITED   
FIELDS TERMINATED BY ‘\t’  
LOCATION ‘s3://your-logs-bucket/cloudfront-logs/’  
TBLPROPERTIES (‘skip.header.line.count’=’2');

Note: The table structure follows CloudFront’s log format. Adjust the LOCATION to match your S3 bucket path.

Analyzing 4xx Errors

Now that your environment is set up, let’s write some queries to analyze those 4xx errors:

Query 1: Count of 4xx Errors by Status Code

SELECT   
 status,  
 COUNT(*) as error_count  
FROM   
 cloudfront_logs.cf_access_logs  
WHERE   
 status BETWEEN 400 AND 499  
 AND date BETWEEN DATE ‘2023–01–01’ AND DATE ‘2023–01–31’  
GROUP BY   
 status  
ORDER BY   
 error_count DESC;

This query will show you the distribution of different 4xx error codes.

Query 2: Top URIs Generating 4xx Errors

SELECT   
 uri,  
 status,  
 COUNT(*) as error_count  
FROM   
 cloudfront_logs.cf_access_logs  
WHERE   
 status BETWEEN 400 AND 499  
 AND date BETWEEN DATE ‘2023–01–01’ AND DATE ‘2023–01–31’  
GROUP BY   
 uri, status  
ORDER BY   
 error_count DESC  
LIMIT 20;

This helps identify problematic endpoints or resources.

Query 3: Error Distribution by Time

SELECT   
 date,  
 HOUR(from_iso8601_timestamp(concat(date, ‘T’, time, ‘Z’))) as hour,  
 COUNT(*) as error_count  
FROM   
 cloudfront_logs.cf_access_logs  
WHERE   
 status BETWEEN 400 AND 499  
 AND date BETWEEN DATE ‘2023–01–01’ AND DATE ‘2023–01–31’  
GROUP BY   
 date, HOUR(from_iso8601_timestamp(concat(date, ‘T’, time, ‘Z’)))  
ORDER BY   
 date, hour;

This query helps identify patterns or spikes in errors by time.

Query 4: 4xx Errors by User Agent

SELECT   
 REGEXP_EXTRACT(user_agent, ‘([^/]+)’) as browser,  
 COUNT(*) as error_count  
FROM   
 cloudfront_logs.cf_access_logs  
WHERE   
 status BETWEEN 400 AND 499  
 AND date BETWEEN DATE ‘2023–01–01’ AND DATE ‘2023–01–31’  
GROUP BY   
 REGEXP_EXTRACT(user_agent, ‘([^/]+)’)  
ORDER BY   
 error_count DESC  
LIMIT 10;

This can help identify if errors are related to specific browsers or bots.

Query 5: Geographic Distribution of Errors

If you have enabled location fields in your logs:

SELECT   
 location,  
 COUNT(*) as error_count  
FROM   
 cloudfront_logs.cf_access_logs  
WHERE   
 status BETWEEN 400 AND 499  
 AND date BETWEEN DATE ‘2023–01–01’ AND DATE ‘2023–01–31’  
GROUP BY   
 location  
ORDER BY   
 error_count DESC  
LIMIT 10;

Advanced Analysis: Finding Root Causes

Let’s dig deeper to find patterns that might explain the increase in 4xx errors:

Query 6: Analyze Referrer Patterns

SELECT   
 referrer,  
 COUNT(*) as error_count  
FROM   
 cloudfront_logs.cf_access_logs  
WHERE   
 status BETWEEN 400 AND 499  
 AND date BETWEEN DATE ‘2023–01–01’ AND DATE ‘2023–01–31’  
 AND referrer != ‘-’  
GROUP BY   
 referrer  
ORDER BY   
 error_count DESC  
LIMIT 20;

This can identify broken links from other websites.

Query 7: Identify Common Query Parameters in Failed Requests

SELECT   
 query_string,  
 COUNT(*) as error_count  
FROM   
 cloudfront_logs.cf_access_logs  
WHERE   
 status BETWEEN 400 AND 499  
 AND date BETWEEN DATE ‘2023–01–01’ AND DATE ‘2023–01–31’  
 AND query_string != ‘-’  
GROUP BY   
 query_string  
ORDER BY   
 error_count DESC  
LIMIT 20;

This may reveal issues with specific parameters being passed to your application.

Creating Dashboards with QuickSight

For ongoing monitoring, you can connect Amazon QuickSight to your Athena queries:

Navigate to the QuickSight console
Create a new analysis
Select Athena as the data source
Choose your cloudfront_logs database and cf_access_logs table
Build visualizations based on the queries we’ve explored

Some useful visualizations include:

Line chart of 4xx errors over time
Bar chart of top error-generating URIs
Pie chart of error status code distribution
Heat map of errors by hour and day

Optimizing Athena Queries

CloudFront logs can grow large, making Athena queries expensive. Here are some optimization tips:

Partitioning Your Table

For more efficient queries, consider partitioning your table by date:

CREATE EXTERNAL TABLE IF NOT EXISTS cloudfront_logs.cf_access_logs_partitioned (  
 `time` STRING,  
 location STRING,  
 bytes BIGINT,  
 — other fields as in the previous definition  
)  
PARTITIONED BY (`date` DATE)  
ROW FORMAT DELIMITED   
FIELDS TERMINATED BY ‘\t’  
LOCATION ‘s3://your-logs-bucket/cloudfront-logs/’  
TBLPROPERTIES (‘skip.header.line.count’=’2');

Then load partitions:

MSCK REPAIR TABLE cloudfront_logs.cf_access_logs_partitioned;

Or add partitions manually:

ALTER TABLE cloudfront_logs.cf_access_logs_partitioned ADD  
PARTITION (`date`=’2023–01–01') LOCATION ‘s3://your-logs-bucket/cloudfront-logs/2023–01–01/’  
PARTITION (`date`=’2023–01–02') LOCATION ‘s3://your-logs-bucket/cloudfront-logs/2023–01–02/’;

Convert to Columnar Format

Convert your data to a columnar format like Parquet for better performance:

CREATE TABLE cloudfront_logs.cf_access_logs_parquet  
WITH (  
 format = ‘PARQUET’,  
 parquet_compression = ‘SNAPPY’,  
 external_location = ‘s3://your-logs-bucket/cloudfront-logs-parquet/’  
) AS  
SELECT * FROM cloudfront_logs.cf_access_logs  
WHERE date BETWEEN DATE ‘2023–01–01’ AND DATE ‘2023–01–31’;

Common Troubleshooting Patterns

Based on the analysis of 4xx errors, here are common issues to check:

404 errors: Check for recently removed resources or broken links in your application
403 errors: Review CloudFront distribution settings, especially origin access identity configuration
400 errors: Look for malformed requests, possibly from outdated clients or bots
401 errors: Check authentication mechanisms and token expiration settings

Conclusion

By using Amazon Athena to analyze CloudFront logs, you can quickly identify the root causes of 4xx errors and take corrective actions. This serverless approach eliminates the need for complex ETL processes while providing powerful SQL-based analysis capabilities.

Remember to optimize your queries and table structure as your log volume grows to keep costs manageable and queries performant.

I hope this article helps you diagnose and resolve your CloudFront 4xx errors efficiently. Happy troubleshooting! 👻

Automated GitLab Artifact Cleanup with AWS Lambda

Habil BOZALİ — Mon, 24 Feb 2025 08:12:04 +0000

Photo by Siarhei Palishchuk on Unsplash

GitLab CI/CD pipelines generate artifacts and build logs that accumulate over time, consuming valuable storage space. While these artifacts are crucial for debugging and deployment, keeping them indefinitely is neither practical nor cost-effective. Let’s explore how to automate the cleanup process using AWS Lambda and EventBridge.

AWS Lambda is a serverless compute service that lets you run code without provisioning or managing servers. Combined with EventBridge for scheduling and SNS for notifications, we can create a robust solution for maintaining our GitLab instance’s hygiene.

Let’s break down the solution into its core components and understand how they work together.

Core Components

Our cleanup solution consists of several key elements:

AWS Lambda function for artifact cleanup
GitLab API integration for accessing projects and jobs
AWS SNS integration for notifications
Environment variables for secure credential management
Error handling and logging

Code Analysis

Let’s analyze the main components of our cleanup function:

Configuration and Setup:

const gitlabHost = 'https://gitlab.myinstance.com';
const token = process.env.GITLAB_TOKEN;
const date = new Date();
date.setMonth(date.getMonth() - 3); //Can be change

This section establishes our GitLab connection details and sets up a three-month threshold for artifact retention.

Main Cleanup Function:

async function deleteOldArtifacts() {
    try {
        let page = 1;
        while (true) {
            const projectsUrl = `${gitlabHost}/api/v4/projects?private_token=${token}&per_page=100&page=${page}`;
            const projectsResponse = await axios.get(projectsUrl);
            const projects = projectsResponse.data;
            if (projects.length === 0) break;
            for (const project of projects) {
                        await deleteArtifactsForJob(project.id);
                        await deleteArtifactsForProject(project.id);
                    }
                    page++;
                }
                sendSnsMessage('Old artifacts from all projects deleted successfully.');
            } catch (error) {
                sendSnsMessage(error.message);
            }
}

This function iterates through all GitLab projects, handling pagination, and processing each project’s artifacts.

SNS Notification:

async function sendSnsMessage(message) {
    const sns = new AWS.SNS();
    const topicArn = 'arn:aws:sns:eu-central-1:00000000000:arn'; 

    sns.publish({
        TopicArn: topicArn,
        Message: JSON.stringify(message)
    });
}

This component sends notifications about the cleanup process status.

Setting Up the Solution

Create a new Lambda function and set these environment variables:

GITLAB_TOKEN
AWS_A_KEY
AWS_S_KEY

Configure the EventBridge rule with a cron expression to schedule the cleanup:

0 0 1 * ? * // Runs monthly on the 1st

Set up an SNS topic and subscribe to receive notifications about the cleanup process.

Complete Code

Here’s the complete Lambda function code:

const axios = require('axios');
const AWS = require('aws-sdk');
require('dotenv').config();
const gitlabHost = 'https://gitlab.myinstance.com';
const token = process.env.GITLAB_TOKEN;
const date = new Date();
date.setMonth(date.getMonth() - 3);
const formattedThreeMonthsAgo = date.toISOString();
AWS.config.update({
    region: 'eu-central-1', 
    accessKeyId: process.env.AWS_A_KEY,
    secretAccessKey: process.env.AWS_S_KEY
});
async function deleteOldArtifacts() {
    try {
        let page = 1;
        while (true) {
            const projectsUrl = `${gitlabHost}/api/v4/projects?private_token=${token}&per_page=100&page=${page}`;
            const projectsResponse = await axios.get(projectsUrl);
            const projects = projectsResponse.data;
            if (projects.length === 0) break;
            for (const project of projects) {
                console.log(`Processing project ID: ${project.id}`);
                await deleteArtifactsForJob(project.id);
                await deleteArtifactsForProject(project.id);
            }
            page++;
        }
        sendSnsMessage('Old artifacts from all projects deleted successfully.');
    } catch (error) {
        console.log(error.message);
        sendSnsMessage(error.message);
    }
}
async function deleteArtifactsForProject(projectId) {
    try {
        const projectArtifactsUrl = `${gitlabHost}/api/v4/projects/${projectId}/artifacts?private_token=${token}`;
        await axios.delete(projectArtifactsUrl);   
    } catch (error) {
        console.error(`Error deleting artifacts for project ${projectId}:`, error.message);
    }
}
async function deleteArtifactsForJob(projectId) {
    try {
        const jobsUrl = `${gitlabHost}/api/v4/projects/${projectId}/jobs?private_token=${token}`;
        const jobsResponse = await axios.get(jobsUrl);
        const jobs = jobsResponse.data;
        for (const job of jobs) {
            try {
                if(typeof job.artifacts !== 'undefined' && job.artifacts.length > 0 && job.finished_at && job.finished_at < formattedThreeMonthsAgo) {
                    const deleteUrl = `${gitlabHost}/api/v4/projects/${projectId}/jobs/${job.id}/artifacts?private_token=${token}`;
                    await axios.delete(deleteUrl);
                    console.log(`Artifact ${job.id} deleted successfully.`);
                }
            } catch (error) {
                console.error(`Error deleting artifacts for job ${projectId}:`, error.message);
            }
        }
    } catch (error) {
        console.error(`Error deleting artifacts for job ${projectId}:`, error.message);
    }
}
async function sendSnsMessage(message) {
    const sns = new AWS.SNS();
    const topicArn = 'arn:aws:sns:eu-central-1:00000000000:arn'; 
    sns.publish({
    TopicArn: topicArn,
    Message: JSON.stringify(message)
    }, (err, data) => {
    if (err) {
        console.error('Error sending message:', err);
    } else {
        console.log('Message sent successfully:', data);
    }
    });
}
exports.handler = async (event) => {
    await deleteOldArtifacts();
    const response = {
      statusCode: 200,
      body: JSON.stringify('Done!'),
    };
    return response;
};

Benefits

Automated cleanup reduces manual maintenance
Configurable retention period
Notification system for monitoring
Cost-effective storage management
Scalable solution for growing GitLab instances

Conclusion

This automated solution helps maintain a clean and efficient GitLab instance by regularly removing old artifacts. The combination of AWS Lambda, EventBridge, and SNS creates a reliable, hands-off maintenance system that can be easily adapted to your specific needs.

See you in the next article! 👻

How to List AWS S3 Directory Contents Using Python and Boto3

Habil BOZALİ — Mon, 24 Feb 2025 07:53:45 +0000

Photo by Luke Chesser on Unsplash

When working with AWS S3, you might need to get a list of all files in a specific bucket or directory. This is particularly useful for inventory management, backup operations, or content synchronization. In this article, we’ll explore how to use Python and boto3 to list directory contents in an S3 bucket.

Common Use Cases

Creating inventory reports of S3 bucket contents
Verifying uploaded files after bulk transfers
Monitoring content changes in specific folders
Synchronizing content between different environments
Automated file management and cleanup operations

Prerequisites

Python 3.x installed
AWS account with appropriate permissions
boto3 library installed (pip install boto3)
AWS credentials configured

Understanding the Code

Let’s break down a simple yet effective solution:

import boto3
import csv
def list_bucket():
  # Configuration
  bucket = "your-bucket-name"
  folder = "path/to/folder"
  # Initialize S3 client
  s3 = boto3.resource("s3",
    aws_access_key_id="YOUR_ACCESS_KEY",
    aws_secret_access_key="YOUR_SECRET_KEY"
  )
  # Get bucket reference
  s3_bucket = s3.Bucket(bucket)

  # List files and extract relative paths
  files_in_s3 = [
      f.key.split(folder + "/")[1] 
      for f in s3_bucket.objects.filter(Prefix=folder).all()
  ]

  # Write results to file
  with open('bucket-contents.txt', 'w', encoding='UTF8') as file:
      file.write(str(files_in_s3))
if name == 'main':
  list_bucket()

Code Breakdown

Client Initialization

s3 = boto3.resource("s3",
    aws_access_key_id="YOUR_ACCESS_KEY",
    aws_secret_access_key="YOUR_SECRET_KEY"
)

This section establishes a connection to AWS S3 using your credentials. For better security, consider using AWS CLI profiles or environment variables instead of hardcoded credentials.

File Listing

files_in_s3 = [
    f.key.split(folder + "/")[1] 
    for f in s3_bucket.objects.filter(Prefix=folder).all()
]

This part:

Uses filter(Prefix=folder) to list only files in the specified folder
Splits the full path to get relative file paths
Creates a list using list comprehension

Output Generation

with open('bucket-contents.txt', 'w', encoding='UTF8') as file:
    file.write(str(files_in_s3))

Writes the results to a text file using proper UTF-8 encoding.

Enhanced Version

Here’s an improved version with additional features:

import boto3
import csv
from datetime import datetime
def list_bucket_contents(bucket_name, folder_prefix, output_format='txt'):
    try:
        # Initialize S3 client using AWS CLI profile
        session = boto3.Session(profile_name='default')
        s3 = session.resource('s3')
        bucket = s3.Bucket(bucket_name)

        # Get file listing
        files = []
        for obj in bucket.objects.filter(Prefix=folder_prefix):
            files.append({
                'name': obj.key.split(folder_prefix + '/')[1],
                'size': obj.size,
                'last_modified': obj.last_modified
            })

        # Generate output filename
        timestamp = datetime.now().strftime('%Y%m%d_%H%M%S')
        output_file = f'bucket_contents_{timestamp}.{output_format}'

        # Write output
        if output_format == 'csv':
            with open(output_file, 'w', newline='', encoding='UTF8') as f:
                writer = csv.DictWriter(f, fieldnames=['name', 'size', 'last_modified'])
                writer.writeheader()
                writer.writerows(files)
        else:
            with open(output_file, 'w', encoding='UTF8') as f:
                f.write(str(files))

        return True, output_file

    except Exception as e:
        return False, str(e)
if __name__ == ' __main__':
    success, result = list_bucket_contents(
        'your-bucket-name',
        'path/to/folder',
        'csv'
    )
    print(f"Operation {'successful' if success else 'failed'}: {result}")

This enhanced version includes:

Support for multiple output formats (CSV/TXT)
Additional file metadata (size, last modified date)
Error handling
Timestamp-based output files
AWS CLI profile support

Best Practices

Never hardcode AWS credentials in your code
Use error handling to manage potential failures
Consider pagination for large buckets
Include relevant metadata in the output
Use appropriate file encoding (UTF-8)

Conclusion

Directory listing in AWS S3 using Python and boto3 is a powerful tool for managing your cloud storage. Whether you’re doing inventory management, migrations, or routine maintenance, this script provides a solid foundation that you can build upon.

See you in the next article! 👻

Automated Java Application Deployment to AWS Elastic Beanstalk using GitHub Actions

Habil BOZALİ — Mon, 24 Feb 2025 07:51:14 +0000

Photo by Roman Synkevych on Unsplash

When developing modern applications, having a robust CI/CD pipeline is crucial. In this guide, we’ll walk through setting up automated deployments to AWS Elastic Beanstalk using GitHub Actions for both staging and production environments.

Prerequisites

Java application with Maven
GitHub repository
AWS Elastic Beanstalk environment
AWS IAM credentials

Setting Up GitHub Actions

We’ll create two workflow files: one for staging and one for production. These files should be placed in the .github/workflows/ directory of your repository.

Staging Deployment Configuration

name: Deploy staging
on:
  push:
    branches:
    - staging
jobs:
  build-and-deploy:
    runs-on: ubuntu-latest
    steps:
    - name: Checkout source code
      uses: actions/checkout@v4
    - name: Setup Java
      uses: actions/setup-java@v4
      with:
        distribution: 'temurin'
        java-version: '21'
    - name: Setup Maven
      uses: stCarolas/setup-maven@v5
      with:
        maven-version: 3.8.2
    - name: Build project
      run: mvn clean package -DskipTests
    - name: Deploy to EB
      uses: einaregilsson/beanstalk-deploy@v22
      with:
        aws_access_key: ${{ secrets.AWS_ACCESS_KEY_ID }}
        aws_secret_key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        application_name: 'your-app-name-staging'
        environment_name: 'your-environment-name-staging'
        version_label: ${{ github.sha}}
        region: eu-central-1
        existing_bucket_name: 'elasticbeanstalk-eu-central-1-your-bucket'
        deployment_package: target/your-application.jar

Production Deployment Configuration

name: Deploy production
on:
  push:
    branches:
    - production
jobs:
  build-and-deploy:
    runs-on: ubuntu-latest
    steps:
    - name: Checkout source code
      uses: actions/checkout@v4
    - name: Setup Java
      uses: actions/setup-java@v4
      with:
        distribution: 'temurin'
        java-version: '21'
    - name: Setup Maven
      uses: stCarolas/setup-maven@v5
      with:
        maven-version: 3.8.2
    - name: Build project
      run: mvn clean package -DskipTests
    - name: Deploy to EB
      uses: einaregilsson/beanstalk-deploy@v22
      with:
        aws_access_key: ${{ secrets.AWS_ACCESS_KEY_ID }}
        aws_secret_key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        application_name: 'your-app-name-prod'
        environment_name: 'your-environment-name-prod'
        version_label: ${{ github.sha}}
        region: eu-central-1
        existing_bucket_name: 'elasticbeanstalk-eu-central-1-your-bucket'
        deployment_package: target/your-application.jar

Setting Up GitHub Secrets

To securely manage AWS credentials, you must add them as GitHub secrets. Here’s how:

Navigate to your GitHub repository
Go to Settings > Secrets and Variables> Actions
Click “New repository secret”
Add the following secrets:

AWS_ACCESS_KEY_ID: Your AWS access key
AWS_SECRET_ACCESS_KEY: Your AWS secret key

How It Works

Let’s break down the key components of our workflow:

1. Trigger Configuration

on:
  push:
    branches:
    - main # or production for prod deployment

The workflow triggers when code is pushed to the specified branch.

2. Environment Setup

- name: Setup Java
  uses: actions/setup-java@v4
  with:
    distribution: 'temurin'
    java-version: '21'
- name: Setup Maven
  uses: stCarolas/setup-maven@v5
  with:
    maven-version: 3.8.2

Sets up the Java and Maven environment for building the application.

3. Build Process

- name: Build project
  run: mvn clean package -DskipTests

Builds the Java application and creates the JAR file.

4. Deployment

- name: Deploy to EB
  uses: einaregilsson/beanstalk-deploy@v22
  with:
    aws_access_key: ${{ secrets.AWS_ACCESS_KEY_ID }}
    aws_secret_key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
    application_name: 'your-app-name'
    environment_name: 'your-environment-name'
    version_label: ${{ github.sha}}
    region: eu-central-1
    existing_bucket_name: 'elasticbeanstalk-eu-central-1-your-bucket'
    deployment_package: target/your-application.jar

Deploys the built application to AWS Elastic Beanstalk.

Important Notes

Security : Never commit AWS credentials directly in the workflow files.
Version Label : We use github.sha it for version tracking, ensuring unique deployments.
Skip Tests : The current configuration skips tests during the build (-DskipTests). Consider enabling tests for production deployments.
Region : Use the correct AWS region where your Elastic Beanstalk environment is hosted.
Bucket Name : The existing_bucket_name should match your Elastic Beanstalk S3 bucket.

Benefits

Automated deployments triggered by code pushes
Separate workflows for staging and production environments
Secure credential management using GitHub secrets
Consistent build and deployment process
Version tracking using Git SHA

Conclusion

This setup provides a robust CI/CD pipeline for Java applications, automating the deployment process to AWS Elastic Beanstalk. By maintaining separate workflows for staging and production, we ensure a proper deployment strategy while keeping our sensitive credentials secure.

See you in the next article! 👻

How to Clone AWS Cognito User Pools Using Python

Habil BOZALİ — Mon, 03 Feb 2025 15:48:05 +0000

Photo by marc belver colomer on Unsplash

Amazon Cognito is a powerful user authentication and authorization service provided by AWS. It helps you manage user sign-up, sign-in, and access control for your web and mobile applications. One common requirement when working with Cognito is the need to clone a User Pool, especially when setting up different environments (development, staging, production) or creating backups.

In this article, we’ll explore how to clone an AWS Cognito User Pool programmatically using Python and the boto3 library. We’ll create a script that copies all essential components, including app clients, groups, and schema attributes.

Prerequisites

Python 3.x installed
AWS account with appropriate permissions
boto3 library installed (pip install boto3)
AWS credentials configured

Understanding the Code Structure

Let’s break down our solution into manageable parts:

1. Setting Up AWS Client

import boto3
from botocore.exceptions import ClientError

def get_client(aws_profile=None, region_name='eu-central-1'):
    if aws_profile:
        boto3.setup_default_session(profile_name=aws_profile)
    return boto3.client('cognito-idp', region_name=region_name)

This section initializes the AWS client using boto3. It allows you to specify an AWS profile and region, making it flexible for different environments.

2. Main Cloning Function

def copy_user_pool(source_user_pool_id, new_user_pool_name, aws_profile=None):
    client = get_client(aws_profile)

    try:
        response = client.describe_user_pool(UserPoolId=source_user_pool_id)
        user_pool_details = response['UserPool']
        print(f"Source User Pool details for ID {source_user_pool_id} retrieved successfully.")

        new_user_pool_response = client.create_user_pool(
            PoolName=new_user_pool_name,
            Policies=user_pool_details['Policies'],
            LambdaConfig=user_pool_details.get('LambdaConfig', {}),
            AutoVerifiedAttributes=user_pool_details.get('AutoVerifiedAttributes', []),
            # ... other configuration parameters
        )

        new_user_pool_id = new_user_pool_response['UserPool']['Id']
        return new_user_pool_id

    except Exception as e:
        print(f"An error occurred: {e}")
        return None

This function handles the main cloning process. It:

Retrieves the source User Pool details
Creates a new User Pool with the same configuration
Returns the new User Pool ID if successful

3. Copying App Clients

def copy_app_clients(client, source_user_pool_id, new_user_pool_id, user_pool_details):
    try:
        app_clients_response = client.list_user_pool_clients(UserPoolId=source_user_pool_id)
        app_clients = app_clients_response['UserPoolClients']

        for app_client in app_clients:
            client.create_user_pool_client(
                UserPoolId=new_user_pool_id,
                ClientName=app_client['ClientName'],
                GenerateSecret=True,
                RefreshTokenValidity=86400,
                # ... other client configurations
            )
    except Exception as e:
        print(f"An error occurred while copying app clients: {e}")

This function copies all app clients from the source User Pool to the new one, maintaining their configurations.

4. Copying User Pool Groups

def copy_user_pool_groups(client, source_user_pool_id, new_user_pool_id):
    try:
        groups_response = client.list_groups(UserPoolId=source_user_pool_id)
        groups = groups_response['Groups']

        for group in groups:
            client.create_group(
                UserPoolId=new_user_pool_id,
                GroupName=group['GroupName'],
                Description=group.get('Description', ''),
                Precedence=group.get('Precedence', 0)
            )
    except Exception as e:
        print(f"An error occurred while copying groups: {e}")

This function replicates all user groups from the source User Pool to the new one.

Usage Example

if __name__ == " __main__":
    source_user_pool_id = 'eu-central-1_XXXXXXXX' # Replace with your source pool ID
    new_user_pool_name = 'my-new-user-pool'
    aws_profile = 'YOUR_AWS_PROFILE'

new_pool_id = copy_user_pool(source_user_pool_id, new_user_pool_name, aws_profile)
    if new_pool_id:
        print(f"User Pool copied successfully. New Pool ID: {new_pool_id}")
    else:
        print("User Pool copying failed.")

Important Notes

The script maintains the same schema attributes, policies, and configurations as the source User Pool
App clients are created with new client IDs and secrets
User data is not copied — only the User Pool structure and configuration
Make sure you have appropriate AWS permissions before running the script

Conclusion

This Python script provides a convenient way to clone AWS Cognito User Pools, which can be particularly useful when setting up new environments or creating backups. The modular structure makes it easy to modify and extend based on your specific needs.

Remember to handle sensitive information carefully and never commit AWS credentials to version control systems.

See you in the next article! 👻

Publish static website with S3 + Github + Cloudfront

Habil BOZALİ — Sun, 27 Aug 2023 18:40:14 +0000

A static website is a type of website that consists of fixed content and resources that do not change dynamically. This means that the content is pre-built and remains the same for all users who visit the website. Static websites are typically written in HTML, CSS, and JavaScript, and they can be hosted on a variety of platforms, including cloud services like Amazon Web Services (AWS).

Amazon S3 (Simple Storage Service) is a scalable cloud storage service provided by Amazon Web Services. It allows you to store and retrieve large amounts of data, including files like images, videos, documents, and more. S3 is designed to be highly reliable, secure, and cost-effective. It's commonly used to store a wide range of data, from backups to media files.

Using AWS S3 to host a static website can provide you with a cost-effective, reliable, and scalable solution that is well-suited for websites with content that doesn't change frequently.

I assume, you have a static website and want to deploy your changes with GitHub Actions and distribute your website with CloudFront CDN network.

Also, I assume, you already have an S3 bucket and CloudFront Distribution.

Create Secrets

In order to run your action, you need to create some secret for it. Let's create 3 required secrets for the action.

AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
DISTRIBUTION_ID

Create GitHub Action

Create a new directory named .github and workflow in your project root directory.

After that, you need to create a main.yml file.

mkdir .github && cd .github && mkdir workflow && cd workflow
touch main.yml

Populate the main.yml file with the following content

name: Upload Website to S3

on:
  push:
    branches:
    - main

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
    - name: Checkout
      uses: actions/checkout@v1

    - name: Configure AWS Credentials
      uses: aws-actions/configure-aws-credentials@v1
      with:
        aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
        aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        aws-region: eu-central-1

    - name: Deploy static site to S3 bucket
      run: aws s3 sync . s3://my-static-website-bucket --delete
    - name: Invalidate CloudFront Distribution
      run: aws cloudfront create-invalidation --distribution-id ${{ secrets.DISTRIBUTION_ID }} --paths "/*"

Conclusion

We created a Github Pipeline and uploaded our static files to the s3 service, invalidating the CloudFront service, and loading the updated version of our page.

See you in the next article. 👻

Create virtual machines with MacOS (m1 and m2 included)

Habil BOZALİ — Sat, 08 Jul 2023 17:50:46 +0000

Introduction

A virtual machine (VM) is a software emulation of a computer system that behaves like a separate physical machine. It enables the creation and execution of multiple operating systems or applications on a single physical computer. VMs provide isolation, allowing different software environments to run independently of each other.

Problem

You are using a Macbook as the development device and you want to use an application that only has a Windows or Linux version available.

My scenario is as follows; I wanted to install Power BI Client on my computer and saw that only the Windows version is available. I set out thinking that I could solve this problem by setting up a virtual machine. Unfortunately, it is very difficult to find free solutions in the macOS world, but you can solve this problem with open-source software.

The first solution that comes to mind is Oracle VM VirtualBox, but Oracle VM VirtualBox does not yet support Apple Silicon processors. (Currently in Developer Preview), My second solution is UTM. UTM is a full-featured system emulator and virtual machine host for iOS and macOS. It is based on QEMU. In short, it allows you to run Windows, Linux, and more on your Mac, iPhone, and iPad. Let's start.

Requirements

brew

Installation

Step 1: You need to install UTM. You can download UTM from the GitHub page.

Step 2: Install the required libs.

brew install cabextract wimlib cdrtools minacle/chntpw/chntpw aria2

Step 3: Download the installer creator. To do this, you need to one of the following addresses.

Once you downloaded and extracted the installer creator, you need to run uup_download_macos.sh from Terminal to generate the ISO.

Step 4: Open the UTM and start the Windows installation.

Following steps are required for the successful installation.

Open UTM and click the “+” button to open the VM creation wizard.

Select “Virtualize”.
Select “Windows”.
Make sure “Import VHDX Image” is unchecked and “Install Windows 10 or higher” is checked. Also, make sure “Install drivers and SPICE tools” is checked. Press “Browse” and select the ISO you built in Step 3.
Pick the amount of RAM and CPU cores you wish to give access to the VM. Press “Next” to continue.
Specify the maximum amount of drive space to allocate. Press “Next” to continue.
If you have a directory you want to mount in the VM, you can select it here. Alternatively, you can skip this and select the directory later from the VM window’s toolbar. The shared directory will be available after installing SPICE tools (see below). Press “Next” to continue.
Press “Save” to create the VM. Wait for the guest tools to finish downloading and press the Run button to start the VM.
Follow the Windows installer. If you have issues with the mouse, press the mouse capture button in the toolbar to send mouse input directly. Press Control+Option together to exit mouse capture mode. Sometimes, due to driver issues, you can enter and exit capture mode and the mouse cursor works normally again.

Conclusion

In this article, we talked about how to install Windows with the help of a virtual machine on the MacOS operating system. I hope it was useful.

See you in the next article. 👻

How to use Lens IDE with two different AWS profiles

Habil BOZALİ — Sat, 01 Jul 2023 19:24:24 +0000

What is Lens IDE?

The Lens IDE is a desktop application with a user-friendly interface for managing and interacting with Kubernetes clusters. It offers a centralized dashboard to view and manage multiple clusters, provides real-time metrics and logs, allows for easy deployment and management of applications, and offers various other features to simplify working with Kubernetes.

The Problem

If there is only 1 profile in the AWS CLI, Lens IDE will not give you a problem, but when you start working with more than one profile, you begin to encounter problems.

Editing the active profile manually or rotating the access keys will start to tire you after a while.

I can offer you a solution to overcome this problem. Let's start.

Requirements

Installation

Before starting the installation, let's ensure everything works with a single profile. To achieve this, let's complete the AWS CLI, then kubectl and finally Lens IDE installations.

After the initial single profile setup, you need to create AWS Profile with the following command:

aws configure --profile PROFILE_NAME

Example AWS CLI Config File:

[profile DEMO_1]
region = eu-central-1
output = json
[profile DEMO_2]
region = eu-central-1
output = json

Example AWS CLI Credential File:

[DEMO_1]
aws_access_key_id = XYZ
aws_secret_access_key = 123
[DEMO_2]
aws_access_key_id = ABC
aws_secret_access_key = 321

Install AWSP:

npm install -g awsp

Add the following to your .bashrc or .zshrc config

alias awsp="source _awsp"

Now you have installed awsp, if you execute awsp command in your terminal you can switch in your AWS Profiles.

One more step to achieve the goal. We need to add some environment variables to the kubectl config file.

Open the file and add the following environments for every profile.

users:
- name: demo@demo.eu-central-1.eksctl.io
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      args:
      - token
      - -i
      - dec14
      command: aws
      env:
      - name: AWS_PROFILE
        value: DEMO_1

Conclusion

In this article, we learned how to create & manage multiple aws profiles and how to use these profiles with Lens IDE. I hope it was useful.

See you in the next article. 👻

Transfer your Gmail messages to Google Workspace Group

Habil BOZALİ — Tue, 20 Jun 2023 21:21:33 +0000

Introduction

Google Groups is an online service provided by Google that allows people with shared interests or goals to communicate and collaborate in a group setting. It serves as a platform for discussions, email-based communication, file sharing, and organizing events within a group of individuals.

There are several reasons why you might consider using Google Groups:

Communication and Collaboration
Community Building
Email Distribution
Access Control and Privacy
Integration with other Google Services
Archiving and Searchability

What if you have an old Gmail mailbox and want to transfer all the content into a Google group? Let's solve it together.

GYB

Got Your Back (GYB) is a command line tool that backs up and restores your Gmail account. GYB also works with Google Workspace (formerly G Suite / Google Apps) accounts.

Installation

Windows Users:

https://github.com/GAM-team/got-your-back/releases/Setup.msi.

Mac and Linux Users:

Open a terminal and run:

bash <(curl -s -S -L https://git.io/gyb-install)

this will download GYB, install it and start setup. Follow the instructions and complete the setup.

Backup Gmail

gyb --email myoldgmail@gmail.com --action backup

This command will create a directory and clone all the messages into it.

Google Workspace Setup

Go to the Google Developers Console
Select Yes and click "Agree and continue". It will take a moment for the project to be created.
Click "Go to credentials"
Click "New credentials" and choose "Service account key".
Click "Select..." and choose "New service account".
Give your service account a name like "GYB Service account".
Keep JSON as the key type. Click "Create".
Agree to create the service account without a role. Copy the client ID.
Your browser will download a .json file. Save the file with the name oauth2service.json and put it in the same folder as gyb.py or gyb.exe.
Go to Domain-wide Delegation in your Google Workspace Admin console
Click "Add New".
For Client ID, enter the Client ID from above.
For API Scopes, enter exactly:

https://mail.google.com/,https://www.googleapis.com/auth/apps.groups.migration,https://www.googleapis.com/auth/drive.appdata

Click "Authorize". Now, your service account setup is complete.

Start Migration

gyb --local-folder GYB-GMail-Backup-myoldgmail@gmail.com --action restore-group --use-admin admin@domain.com --service-account --email yourgroup@domain.com

This command syncs your local backup into a specified Google Group.

Conclusion

In this article, we learned how to move the messages in your Gmail account to the Google group using GYB.

See you in the next article. 👻