DEV Community: Uendi Hoxha

My Thoughts on Data Mesh

Uendi Hoxha — Wed, 16 Jul 2025 11:02:57 +0000

Over the past year, I’ve watched the concept of "Data Mesh" evolve from an abstract theory into a serious architectural consideration for modern data teams. As someone who works across both DevOps and Data Engineering, I’m naturally drawn to its promise: domain-oriented data ownership, faster delivery cycles and better alignment between producers and consumers of data. But is Data Mesh the solution to all our scaling problems, or just a temporary trend?

What Is Data Mesh?

At its core, Data Mesh is a decentralized approach to data architecture. It challenges the traditional model where a centralized data team owns and serves all organizational data. Instead, it proposes a model based on four key principles:

Domain-Oriented Ownership: Data should be owned and maintained by the teams who understand it.
Data as a Product: Each data set is treated like a product, with clear documentation, SLAs, quality checks & versioning.
Self-Serve Data Infrastructure: Platform teams provide tooling and infrastructure that domain teams can use without needing deep DevOps skills.
Federated Computational Governance: Governance responsibilities are shared across domains with global standards enforced in a decentralized way.

Why I’m Excited About Data Mesh

Well, first thing is scalability. In large organizations, central data teams often become bottlenecks. With Data Mesh, domains can move independently and scale without overwhelming a single team.

Next, ownership. When the people closest to the data also own its pipelines and quality, the result is more accurate, timely & useful data.

At another point, Data Mesh promotes shorter development cycles. Teams can iterate on their own data products without waiting for centralized coordination.

Challenges I’ve Seen
First things first, it requires a cultural shift. Domain teams must want and be prepared to own their data.

Without strong standards, decentralization can lead to inconsistency, duplication, and poor discoverability.

While some platforms like DataHub and OpenMetadata are improving, many organizations still struggle with unified lineage, quality monitoring & schema tracking.

When contracts aren't enforced, multiple versions of the same data can exist across domains—leading to trust issues.

Real-World Tradeoffs
I believe centralization still has its place. For some use cases, a centralized team may still be more efficient, especially for cross-domain reporting or compliance.

Many successful teams implement a hybrid model: centralized data lake storage with decentralized ownership of pipelines and transformations.

Last but not least are costs considerations. Domain duplication and self-serve infra may increase costs—especially in cloud environments. Observability becomes essential to avoid waste.

Lessons Learned & Best Practices

- Start Small: Pilot Data Mesh with one or two domains. Prove the model before expanding.

- Invest in Metadata & Discovery:
Use tools like OpenMetadata or DataHub to make datasets easily discoverable.

- Automate Data Contracts:
Add contract validation to CI/CD pipelines using tools like Great Expectations or Spectacles.

- Standardize Naming & Schema Conventions:
Avoid inconsistency by enforcing naming rules across domains.

- Establish Cross-Domain Syncs:
Hold regular governance meetings to align on contracts, metrics, and schema evolution.

Conclusion

Data Mesh isn’t a silver bullet. It’s a mindset shift that aligns data architecture with how modern software systems are already built: domain-first, API-driven & self-serve.

If your team is hitting bottlenecks with a centralized data model, or struggling with ownership and scale, Data Mesh is worth exploring. Start small, measure results and evolve...

SQL Query Optimization for Data Engineers

Uendi Hoxha — Tue, 06 May 2025 09:46:14 +0000

Moving data efficiently can make the difference between a smooth system and a frustratingly slow one. Optimizing your SQL queries not only speeds up your jobs, but also reduces cloud costs and improves system scalability.

In this post, I'll share 7 practical SQL optimization tips you can apply immediately, with real-world examples.

I. Always SELECT Only the Columns You Need
It’s easy to get lazy and use SELECT *, especially when you're exploring data.
However, pulling all columns increases the amount of data transferred across the network and the memory needed to process it. On wide tables, this can severely impact performance.

Bad example:

SELECT * FROM orders;

Better:

SELECT order_id, order_date, total_amount FROM orders;

II. Use Proper Indexes
Indexes are critical for query performance, especially when filtering (WHERE), joining (JOIN), or sorting (ORDER BY).

If your query frequently filters on a column, it’s a strong candidate for indexing.

Example:

CREATE INDEX idx_orders_customer_id ON orders(customer_id);

Pro tip: Always check your queries with EXPLAIN to verify whether your indexes are actually being used. A missing or unused index can make queries 10x slower.

III. Avoid Unnecessary JOINs
JOINs are powerful — but they can be costly, especially across large tables.
If you're joining tables just to retrieve a field you don't actually use, or if the JOIN isn't adding value to your result set, rethink the query.

Best practices:

Fetch only what you truly need
Consider denormalization if two tables are always accessed together
Use INNER JOIN instead of LEFT JOIN when you don't need unmatched rows

Example:

Instead of this:

SELECT o.order_id, c.customer_name
FROM orders o
LEFT JOIN customers c ON o.customer_id = c.customer_id;

If you know every order has a customer, prefer:

SELECT o.order_id, c.customer_name
FROM orders o
INNER JOIN customers c ON o.customer_id = c.customer_id;

IV. Filter Early With WHERE Clauses
Always narrow down your data as early as possible.

The earlier you apply your WHERE filters, the less data the database engine needs to process — making the query faster and lighter.

Example:

SELECT customer_id, order_id
FROM orders
WHERE order_date > '2025-01-01';

Filtering after joining or fetching lots of rows will cause unnecessary load. Make filtering a priority.

V. Limit Result Sets When Exploring
When you're writing queries to explore data or debug issues, always add a LIMIT to avoid pulling millions of rows by accident.

Example:

SELECT * FROM orders
WHERE total_amount > 1000
LIMIT 100;

This tiny habit prevents unnecessary load on your database and keeps you from crashing your local environment.

VI. Analyze Execution Plans (EXPLAIN)
Want to know why a query is slow?
Use your database’s execution plan tools.

In PostgreSQL and MySQL, running EXPLAIN shows how the database will execute your query whether it will do a sequential scan (slow) or an index scan (fast).

Example:

EXPLAIN ANALYZE
SELECT * FROM orders WHERE customer_id = 123;

Look out for:

Seq Scan → sequentially scanning the whole table (bad for large tables)
Index Scan → using indexes efficiently (good)
High-cost operations like sorts, nested loops, or large hash joins

Learning to read execution plans is one of the best investments you can make as a data engineer.

VII. Batch Large Updates and Inserts
Updating or inserting millions of rows at once can lock tables and overwhelm resources.
Instead, break large operations into smaller batches.

Example:

Instead of:

INSERT INTO large_table
SELECT * FROM very_large_temp_table;

Use a batching strategy:

INSERT INTO large_table
SELECT * FROM very_large_temp_table
WHERE id BETWEEN 1 AND 10000;

-- Repeat with next batch

This keeps locks short, memory usage reasonable, and reduces the risk of timeouts.

Strategies to Save Costs on AWS Services Without Compromising Performance

Uendi Hoxha — Wed, 02 Apr 2025 14:12:41 +0000

Managing cloud costs effectively is a key challenge for many businesses leveraging AWS. The vast range of services can make it difficult to track expenses, and without careful monitoring, costs can quickly exceed expectations. One of the first steps to saving costs on AWS is identifying which services are driving up your bill. A great tool for this is AWS Cost Explorer, which allows you to visualize and analyze your AWS spending patterns.

By using AWS Cost Explorer, you can easily detect services that have unusually high costs or spikes in usage. This gives you the visibility needed to pinpoint areas for optimization, ensuring you're not overpaying for underutilized resources or inefficient configurations. Once you’ve identified these services, you can take steps to optimize their usage, which is exactly what we’ll cover in this article. Let’s dive into practical strategies to reduce AWS costs without sacrificing performance or reliability.

I. Right-Sizing Your Instances
One of the easiest ways to save money on AWS is by right-sizing your instances. AWS allows you to scale resources up or down based on your application’s needs, so it's important to regularly monitor your usage and adjust the size of your instances accordingly.

How to right-size:

Use AWS Cost Explorer and AWS Trusted Advisor to analyze your current instance usage.
Monitor CloudWatch metrics to assess CPU, memory, and disk utilization.
Switch to smaller instances when underutilized or opt for larger instances only when necessary.

Tip: Consider using AWS EC2 Spot Instances for non-critical workloads. These instances can be up to 90% cheaper than On-Demand instances.

II. Use Reserved Instances and Savings Plans
AWS offers Reserved Instances (RIs) and Savings Plans for long-term commitments that can provide significant savings compared to On-Demand pricing.

Reserved Instances are best for predictable, steady-state workloads, while Savings Plans provide more flexibility across EC2, Lambda, and other services.

Savings Plan vs Reserved Instances:

Savings Plans allow you to commit to a specific amount of usage (measured in $/hour) for a one- or three-year term and can be applied across multiple AWS services.
RIs provide a significant discount on instance pricing if you commit to using specific instance types in a specific region for a longer period (1 or 3 years).

III. Optimize Storage Costs
AWS storage costs can become a significant part of your cloud bill, especially when using services like Amazon S3, EBS, and RDS.

Amazon S3:

Use S3 Intelligent-Tiering to automatically move objects between storage classes based on access patterns.
Set lifecycle policies to transition objects to lower-cost storage tiers (e.g., S3 Glacier for archival data).

Amazon EBS:

Regularly monitor your EBS volumes and delete unused or unnecessary volumes to reduce costs.
Use EBS Snapshots wisely, as frequent snapshots can lead to unnecessary costs.

Amazon RDS:

Move your RDS instances to read replicas if you're serving a read-heavy workload, reducing costs by offloading read traffic.
Switch to RDS Aurora, which is more cost-effective for many workloads compared to traditional RDS engines.

IV. Utilize Auto-Scaling to Adjust to Demand
Auto-scaling helps you automatically scale up or down based on demand, which means you're only paying for the resources you need at any given moment.

**EC2 Auto Scaling: **Automatically adjusts the number of EC2 instances running, ensuring you're not paying for unused resources.

Elastic Load Balancing (ELB): Combined with Auto Scaling, this ensures traffic is distributed across your instances in an optimized manner.

V. Use Lambda for Serverless Architectures
AWS Lambda can help you reduce costs by allowing you to run code without provisioning or managing servers. You only pay for the compute time your code consumes, making it highly cost-effective for certain workloads.

How Lambda saves costs:

No need for running EC2 instances 24/7.
Pay only for actual execution time, reducing costs for sporadic workloads.
Can scale automatically to handle varying workloads.

VI. Leverage CloudWatch for Cost Monitoring
AWS provides tools like CloudWatch to monitor resource usage and set alarms for over-spending. By monitoring your AWS costs with CloudWatch, you can identify where to optimize and avoid unexpected spikes in usage.

CloudWatch Tips:

Set up Cost Anomaly Detection to automatically detect unusual spending patterns.
Use AWS Budgets to set custom cost and usage budgets, and receive alerts when your spending exceeds thresholds.

Take Advantage of S3 Glacier for Long-Term Data Storage
For data that doesn’t need to be accessed frequently, use S3 Glacier or S3 Glacier Deep Archive to store data at a fraction of the cost of standard S3 storage.

Best for: Backup data, historical records, and other infrequently accessed data that still needs to be retained.

VII. Choose the Right AWS Region
AWS services are priced differently depending on the region. By selecting the most cost-effective region that still meets your performance needs, you can reduce costs. However, be mindful of latency and data transfer costs if your users are far from the chosen region.

Tip: Check the AWS Pricing Calculator to estimate costs in different regions before making a decision.

Project Overview: Real-Time Smart Building Monitoring System with Amazon Kinesis

Uendi Hoxha — Mon, 14 Oct 2024 19:10:00 +0000

Architecture Overview

Components
IoT Sensors - High-fidelity sensors monitor environmental variables such as temperature, humidity, light levels and occupancy.
Kinesis Data Stream - Collects real-time data from various IoT sensors deployed in the building.
AWS SQS: Acts as a buffer to handle traffic spikes by queuing incoming sensor data, ensuring reliable message delivery and smoothing out the data flow to the downstream Lambda function.
AWS Lambda - Processes the incoming data, applies transformations and performs analytics.
DynamoDB - Stores processed data for structured queries and historical analysis.
Data Visualization Tools - Grafana of Amazon Athena for analyzing sensor metrics and insights.

Use Case Scenarios

1. Predictive Maintenance
Utilize real-time environmental data and historical trends to predict when equipment (like HVAC systems) may require maintenance. By analyzing temperature fluctuations and operational patterns, the system can forecast potential failures, allowing for proactive maintenance scheduling.

2. Energy Optimization
Collect data on occupancy and environmental conditions to dynamically adjust HVAC systems, optimizing energy consumption and reducing costs. For example, if sensors detect that a room is unoccupied, the HVAC system can be adjusted accordingly.

3. Space Utilization
Monitor occupancy data in real-time to understand space utilization, enabling better planning and resource allocation within the building. Analyzing patterns over time can inform decisions about office layout or space reallocation.

Data Flow and Processing

Data Ingestion
IoT sensors send real-time data (temperature, humidity, light level, occupancy) to the Kinesis Data Stream.
Sensor Data Format:

{
  "sensor_id": "sensor_1",
  "temperature": 22.5,
  "humidity": 45.0,
  "light_level": 70,
  "occupancy": true,
  "timestamp": 1694658000
}

Data Processing with AWS SQS
The Kinesis Data Stream triggers a Lambda function, which sends the data to an SQS queue. Another Lambda function, triggered by the SQS queue processes the messages by applying necessary transformations such as unit conversions or data normalization.

import os
import json
import boto3
from decimal import Decimal

# Use environment variable for the table name
dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table(os.environ['DYNAMODB_TABLE_NAME'])
sqs = boto3.client('sqs')
queue_url = os.environ['SQS_QUEUE_URL']

def lambda_handler(event, context):
    for record in event['Records']:
        payload = json.loads(record['kinesis']['data'])

        # Data validation logic
        if validate_data(payload):
            transformed_data = transform_data(payload)
            # Send data to SQS for further processing
            send_message_to_sqs(transformed_data)
        else:
            print(f"Invalid data: {payload}")

    return {
        'statusCode': 200,
        'body': json.dumps('Data processed successfully')
    }

def validate_data(data):
    return 'sensor_id' in data and 'temperature' in data

def transform_data(data):
    return {
        'sensor_id': data['sensor_id'],
        'temperature': Decimal(data['temperature']),
        'humidity': Decimal(data['humidity']),
        'light_level': data['light_level'],
        'occupancy': data['occupancy'],
        'timestamp': int(data['timestamp'])
    }

def send_message_to_sqs(data):
    # Send transformed data to SQS
    try:
        response = sqs.send_message(
            QueueUrl=queue_url,
            MessageBody=json.dumps(data)
        )
        print(f"Message sent to SQS: {response['MessageId']}")
    except Exception as e:
        print(f"Error sending message to SQS: {e}")

Data Storage
Processed data is stored in DynamoDB for structured querying and historical analysis. The data structure allows efficient retrieval and aggregation of sensor data.
DynamoDB Table Schema:
Table Name: SensorData
Partition Key: sensor_id (String)
Sort Key: timestamp (Number)
Attributes: temperature (Decimal), humidity (Decimal), light_level (Number), occupancy (Boolean)

DynamoDB’s query capability should be able to perform structured queries on the collected data, like this one:

response = table.query(
    KeyConditionExpression=Key('sensor_id').eq('sensor_1'),
    FilterExpression=Attr('occupancy').eq(True)
)

Purposes of Analyzing Collected Data

Analyzing the collected data serves multiple purposes, enhancing the overall efficiency and management of the smart building system. Historical temperature and humidity data, along with occupancy patterns, enable dynamic adjustments to HVAC settings via AWS IoT, ensuring optimal comfort while conserving energy.

By correlating sensor data with equipment operational metrics, the system can identify trends that precede potential failures, facilitating proactive maintenance scheduling.

Implementing thresholds for temperature anomalies in DynamoDB allows for triggering alerts using AWS SNS when limits are exceeded, thus preventing equipment damage.

Additionally, monitoring energy usage patterns relative to occupancy levels drives energy-efficient upgrades, with reports created in Amazon QuickSight to visualize energy consumption against occupancy over time. This analysis also identifies under-utilized areas through aggregation queries in DynamoDB, informing decisions about office layout and resource allocation.

Furthermore, historical data is stored for longitudinal studies, with AWS Glue used to periodically batch process data from DynamoDB into Amazon S3 for deeper analytical queries via Amazon Athena.

Lastly, anomaly detection algorithms can be implemented using Amazon SageMaker, flagging unusual conditions based on historical data patterns to enhance safety and operational reliability.

Time for some demo...

The real-time temperature is streamed via Kinesis and processed by AWS Lambda. The processed temperature data is then queried from DynamoDB by the chatbot, which provides the response.

The historical data for the conference room is stored in DynamoDB. AWS Lambda processed and stored this data when it was collected yesterday. The chatbot queries this stored data to provide the historical temperature.
This scenario aligns with the "Predictive Maintenance" and "Space Utilization" use cases from the architecture, where the system can analyze trends and historical patterns.

This question is outside the scope of the data being collected and analyzed by the system. The chatbot appropriately responds with a fallback message, indicating its primary focus is on sensor-related data.

While this question goes beyond the basic temperature or environmental monitoring capabilities, it can be tied to an extended use case where occupancy sensors (part of the IoT network) could detect whether a room is occupied. This information could then be used to check availability. In this scenario, the chatbot is querying the occupancy data stored in DynamoDB for a booking system.

Best Practices for Securing Amazon S3 Buckets

Uendi Hoxha — Wed, 09 Oct 2024 14:56:34 +0000

###The Risks of Public S3 Buckets
Public S3 buckets can pose significant security risks due to improper configurations. When a bucket is publicly accessible, it allows anyone on the internet to view or manipulate the contents. This misconfiguration can lead to several critical issues.

There are some test buckets you can find here: https://buckets.grayhatwarfare.com/files?bucket=tempdev.s3-us-west-2.amazonaws.com. Notice how the content of the bucket is publicly accessible.

curl https://tempdev.s3-us-west-2.amazonaws.com/
<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>tempdev</Name><Prefix></Prefix><Marker></Marker><MaxKeys>1000</MaxKeys><IsTruncated>true</IsTruncated><Contents><Key>3rdpartylicenses.txt</Key><LastModified>2018-05-03T02:32:47.000Z</LastModified><ETag>&quot;c27a89a617ae0a7660c490a46b8c9486&quot;</ETag><Size>12331</Size><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>AvayaHome.7f45b5641004c88bd0ee.jpg</Key><LastModified>2018-05-03T02:32:50.000Z</LastModified><ETag>&quot;7f45b5641004c88bd0ee9d6b1330b90a&quot;</ETag><Size>850159</Size><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>AvayaOffice.801610a579e808709ae0.jpg</Key><LastModified>2018-05-03T02:32:51.000Z</LastModified><ETag>&quot;801610a579e808709ae0338a3f0c39c1&quot;</ETag><Size>163750</Size><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>assets/bower_components/Ionicons/.bower.json</Key><LastModified>2018-05-03T02:33:14.000Z</LastModified><ETag>&quot;38e89495d6f99665c32e21304ae50d12&quot;</ETag><Size>881</Size><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>assets/bower_components/Ionicons/LICENSE</Key><LastModified>2018-05-03T02:33:16.000Z</LastModified><ETag>&quot;11c960a3f0bc008428616bffe574b258&quot;</ETag><Size>1094</Size><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>assets/bower_components/Ionicons/bower.json</Key><LastModified>2018-05-03T02:33:14.000Z</LastModified><ETag>&quot;238c943fc3d1f3f8e92d75d47ef31ea7&quot;</ETag><Size>691</Size><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>assets/bower_components/Ionicons/cheatsheet.html</Key><LastModified>2018-05-03T02:33:15.000Z</LastModified><ETag>&quot;fb33483329960f43204001c5cf5837c0&quot;</ETag><Size>1276366</Size><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>assets/bower_components/Ionicons/component.json</Key><LastModified>2018-05-03T02:33:16.000Z</LastModified><ETag>&quot;0e29ebf1783312e96becd2fe4f0fe065&quot;</ETag><Size>429</Size><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>assets/bower_components/Ionicons/composer.json</Key><LastModified>2018-05-03T02:33:16.000Z</LastModified><ETag>&quot;de83b956f2f9554252e2c316f6cb0c77&quot;</ETag><Size>887</Size><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>assets/bower_components/Ionicons/css/ionicons.css</Key><LastModified>2018-05-03T02:34:23.000Z</LastModified><ETag>&quot;f27354b28af3cf48d28260c03305d0ce&quot;</ETag><Size>57193</Size><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>assets/bower_components/Ionicons/css/ionicons.min.css</Key><LastModified>2018-05-03T02:34:23.000Z</LastModified><ETag>&quot;0d6763b67616cb9183f3931313d42971&quot;</ETag><Size>51284</Size><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>assets/bower_components/Ionicons/fonts/ionicons.eot</Key><LastModified>2018-05-03T02:34:24.000Z</LastModified><ETag>&quot;2c2ae068be3b089e0a5b59abb1831550&quot;</ETag><Size>120724</Size><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>assets/bower_components/Ionicons/fonts/ionicons.svg</Key><LastModified>2018-05-03T02:34:26.000Z</LastModified><ETag>&quot;621bd386841f74e0053cb8e67f8a0604&quot;</ETag><Size>333834</Size><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>assets/bower_components/Ionicons/fonts/ionicons.ttf</Key><LastModified>2018-05-03T02:34:23.000Z</LastModified><ETag>&quot;24712f6c47821394fba7942fbb52c3b2&quot;</ETag><Size>188508</Size><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>assets/bower_components/Ionicons/fonts/ionicons.woff</Key><LastModified>2018-05-03T02:34:26.000Z</LastModified><ETag>&quot;05acfdb568b3df49ad31355b19495d4a&quot;</ETag><Size>67904</Size><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>assets/bower_components/Ionicons/less/_ionicons-font.less</Key><LastModified>2018-05-03T02:34:27.000Z</LastModified><ETag>&quot;bb570d47b5190b9f55ed9302aac05459&quot;</ETag><Size>880</Size><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>assets/bower_components/Ionicons/less/_ionicons-icons.less</Key><LastModified>2018-05-03T02:34:27.000Z</LastModified><ETag>&quot;9379d6c15ae5bb23c0c0ad5c2901b4b6&quot;</ETag><Size>90037</Size><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>assets/bower_components/Ionicons/less/_ionicons-variables.less</Key><LastModified>2018-05-03T02:34:27.000Z</LastModified><ETag>&quot;572209c81d7e5a82cc4a995d0cc459bf&quot;</ETag><Size>27680</Size><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>assets/bower_components/Ionicons/less/ionicons.less</Key><LastModified>2018-05-03T02:34:27.000Z</LastModified><ETag>&quot;5b6120e1e2a45ba544699d1f6658a20a&quot;</ETag><Size>84</Size><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>assets/bower_components/Ionicons/png/512/alert-circled.png</Key><LastModified>2018-05-03T02:39:02.000Z</LastModified><ETag>&quot;c9f9f9e6871298de4a84fd37c0d88f07&quot;</ETag><Size>2551</Size><StorageClass>STANDARD</StorageClass>

Attackers can enumerate or download all files in the bucket and potentially find sensitive data or vulnerabilities using tools like s3cmd to list and download all files in a loop.
If the bucket policy permits, you can upload malicious files that could harm users or the service. So, just imagine the scenario where attackers may exploit public buckets to store large amounts of data or generate excessive requests, leading to unexpected charges on your AWS bill!

Best Practices for Securing S3 Buckets

To mitigate the risks associated with public S3 buckets, it is essential to follow best practices that ensure the security and privacy of your data:

*I. Set Default Settings to Private *
Ensure that the default settings of your S3 buckets are private. Only grant access to users and services that absolutely need it. Review access settings regularly to ensure no unintended permissions are granted.

*II. Implement Bucket Policies *
Use S3 bucket policies to define who can access your bucket and what actions they can perform. Limit access to specific IAM users, roles, or AWS accounts as necessary.

III. Enable Server Access Logging
Turn on server access logging for your S3 buckets. This feature allows you to log requests made to your bucket, which can help you monitor access patterns and identify unauthorized attempts to access data.

IV. Enable Versioning
Activate versioning on your S3 buckets. This feature allows you to preserve, retrieve, and restore every version of every object stored in the bucket, making it easier to recover from accidental deletions or overwrites.

V. Encrypt Data At Rest
Enable server-side encryption (SSE) for all objects stored in S3. This ensures that your data is encrypted at rest, adding an extra layer of security. You can choose to use Amazon S3-managed keys (SSE-S3), AWS Key Management Service (SSE-KMS), or customer-provided keys (SSE-C).

VI. Encrypt Data In Transit
Always ensure that data transmitted between your application and S3 is encrypted. Use HTTPS to secure data in transit and prevent man-in-the-middle attacks. This guarantees that sensitive data, such as credentials or personally identifiable information (PII), remains protected during transmission.

VII. Use Block Public Access Feature
AWS provides the S3 Block Public Access feature, which helps you quickly identify and prevent public access to S3 buckets. Enable this feature to block all public access at the account or bucket level.

VIII. Track API Calls with CloudTrail
Utilize AWS CloudTrail to track API calls made to S3 buckets, and configure Amazon CloudWatch alarms to notify you of any suspicious activity or unauthorized access attempts.

IX. Implement Lifecycle Policies
Use lifecycle policies to manage the storage of objects in your S3 buckets. These policies can automatically transition objects to less expensive storage classes or delete them after a specified period, helping reduce storage costs and potential exposure of stale data.

X. Combine Access Points with Bucket Policies
Access Points allow for granular permissions tailored to specific applications or teams. For example, you can create separate Access Points for different applications, granting read or write access as needed. Meanwhile, bucket policies enforce broader rules, such as restricting access to certain IP addresses. This layered approach not only minimizes the risk of unauthorized access but also simplifies permission management, allowing for quick adjustments without affecting overall security.

XI. Use Access Points for Data Lakes
Access Points are invaluable when building a data lake in S3, as they enable tailored access for various teams. Each team can have its own Access Point with specific permissions, ensuring they access only the data they need. For instance, X team might have broad read access, Y team has restricted access to sensitive data. This segmentation enhances governance and compliance with regulations, providing clear oversight of who accesses what data. Additionally, Access Points can optimize performance by directing requests more efficiently, leading to faster data retrieval and processing.

Containerization and Deployment Using Amazon ECS and Fargate

Uendi Hoxha — Wed, 09 Oct 2024 13:23:41 +0000

Amazon Elastic Container Service (ECS) is a fully managed container orchestration service that simplifies the deployment and management of containerized applications. AWS Fargate is a serverless compute engine for containers that works with ECS, allowing you to run containers without managing the underlying infrastructure.

In this article, I will explore how to use ECS and Fargate for deploying a sample application while integrating Amazon RDS for database management, using AWS KMS and Secrets Manager to securely handle sensitive information and managing Docker images with Amazon ECR.

I. Setting Up Your Development Environment

To get started, let’s create a simple application that we will containerize and deploy. For this example, we will use a Node.js application with a PostgreSQL database.
Example of simple app.js:

const express = require('express');
const mysql = require('mysql');
const AWS = require('aws-sdk');
const secretsManager = new AWS.SecretsManager();

const app = express();
const port = process.env.PORT || 3000;

async function getDatabaseCredentials() {
    const data = await secretsManager.getSecretValue({ SecretId: 'RDSMasterUserSecret' }).promise();
    return JSON.parse(data.SecretString);
}

app.get('/', async (req, res) => {
    const secret = await getDatabaseCredentials();
    const connection = mysql.createConnection({
        host: 'your-rds-endpoint', // Replace this with your actual RDS endpoint after creation
        user: secret.username,
        password: secret.password,
        database: 'mydatabase'
    });

    connection.query('SELECT * FROM mytable', (error, results) => {
        if (error) throw error;
        res.json(results);
    });
});

app.listen(port, () => {
    console.log(`Server running at http://localhost:${port}`);
});

Initialize `package.json` and Install Dependencies

Command to initialize package.json: npm init -y
Command to install dependencies: npm install express mysql

II. Writing Dockerfile

Next step is creating a Dockerfile to containerize our application:

# Use the official Node.js image
FROM node:14

# Set the working directory
WORKDIR /usr/src/app

# Copy package.json and install dependencies
COPY package*.json ./
RUN npm install

# Copy the application code
COPY . .

# Expose the application port
EXPOSE 3000

# Command to run the application
CMD ["node", "app.js"]

III. Configuring AWS KMS and Secrets Manager

To securely manage your database credentials, we will be using AWS Secrets Manager and KMS.

Create a KMS key (as shown in the CloudFormation template below).
Store RDS credentials in Secrets Manager (as included in the template).

IV. Setting Up Amazon RDS

Create a template.yaml file for your CloudFormation setup, which includes RDS configuration.

AWSTemplateFormatVersion: '2010-09-09'
Resources:
  MyKMSKey:
    Type: AWS::KMS::Key
    Properties:
      KeyPolicy:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal:
              AWS: arn:aws:iam::<your-account-id>:root
            Action: "kms:*"
            Resource: "*"

  MySecret:
    Type: AWS::SecretsManager::Secret
    Properties:
      Name: RDSMasterUserSecret
      Description: RDS Master User Credentials
      SecretString: !Sub |
        {
          "username": "${MasterUsername}",
          "password": "${MasterUserPassword}"
        }
      KmsKeyId: !Ref MyKMSKey

  MyDBInstance:
    Type: AWS::RDS::DBInstance
    Properties:
      DBInstanceIdentifier: mydbinstance
      AllocatedStorage: 20
      DBInstanceClass: db.t2.micro
      Engine: mysql
      MasterUsername: !Join [ "", [ !GetAtt MySecret.SecretString, "username" ] ]
      MasterUserPassword: !Join [ "", [ !GetAtt MySecret.SecretString, "password" ] ]
      DBName: mydatabase
      VPCSecurityGroups:
        - !GetAtt MyDBSecurityGroup.GroupId

Run the following command in your terminal to deploy the stack, ensuring you have the AWS CLI configured. The template.yaml file includes parameters such as MasterUsername and MasterUserPassword, which you must define in the command when deploying the stack. Here’s how to pass these parameters during deployment:

aws cloudformation create-stack --stack-name my-stack --template-body file://template.yaml --parameters ParameterKey=MasterUsername,ParameterValue=admin ParameterKey=MasterUserPassword,ParameterValue=mypassword --capabilities CAPABILITY_NAMED_IAM

These parameters will be used to create the RDS instance and store the credentials securely in AWS Secrets Manager.

V. Building and Pushing Docker Images

Now that we have our Dockerfile ready, let’s build and push our Docker image to ECR.

Step 1: Create an ECR Repository
First, log in to your AWS Management Console and navigate to ECR. Create a new repository named my-ecs-app.

Step 2: Authenticate Docker to ECR
Run the following command to authenticate Docker with your ECR registry (replace REGION with your AWS region):

aws ecr get-login-password --region REGION | docker login --username AWS --password-stdin <your_account_id>.dkr.ecr.<REGION>.amazonaws.com

Step 3: Build the Docker Image
Run the following command to build your Docker image:

docker build -t my-ecs-app .

Step 4: Tag and Push the Image
Tag the image for your ECR repository:

docker tag my-ecs-app:latest <your_account_id>.dkr.ecr.<REGION>.amazonaws.com/my-ecs-app:latest

Step 5: Push the image to ECR

docker push <your_account_id>.dkr.ecr.<REGION>.amazonaws.com/my-ecs-app:latest

VI. Deploying with Amazon ECS and Fargate

Define a task in ECS that references your Docker image stored in ECR. Make sure the image parameter matches the name of the repository you created in ECR:

{
  "family": "my-node-app",
  "containerDefinitions": [
    {
      "name": "my-node-app",
      "image": "<your-account-id>.dkr.ecr.<region>.amazonaws.com/my-node-app:latest",
      "essential": true,
      "memory": 512,
      "cpu": 256,
      "portMappings": [
        {
          "containerPort": 3000,
          "hostPort": 3000
        }
      ],
      "environment": [
        {
          "name": "PORT",
          "value": "3000"
        }
      ]
    }
  ]
}

You can now run your ECS task on Fargate, which manages the compute resources for you.

aws ecs create-service --cluster my-cluster --service-name my-node-app --task-definition my-node-app --desired-count 1 --launch-type FARGATE --network-configuration "awsvpcConfiguration={subnets=[<subnet-id>],securityGroups=[<security-group-id>],assignPublicIp='ENABLED'}"

VII. Permissions and IAM Roles

Last, but not least we have permissions. For the successful deployment of your application using Amazon ECS, RDS and Secrets Manager we must ensure the following IAM roles and permissions are configured:

a. IAM Role for ECS Task Execution
Role Name: ECS-Task-Execution-Role
Permissions:

AmazonECSTaskExecutionRolePolicy (Allows ECS to pull images from ECR)
SecretsManagerReadWrite (Allows access to AWS Secrets Manager)

b. IAM Role for RDS Access
Role Name: RDS-Access-Role
Custom Permissions:

rds:DescribeDBInstances
rds:CreateDBInstance (To create a new RDS instance)
rds:DeleteDBInstance (To delete an existing RDS instance)
rds:ModifyDBInstance (To modify settings of an RDS instance)

c. Amazon ECR Permissions

ecr:GetAuthorizationToken (Required to authenticate Docker with Amazon ECR)
ecr:BatchCheckLayerAvailability (To check layers for Docker images)
ecr:GetDownloadUrlForLayer (To download layers of Docker images)
ecr:BatchGetImage (To retrieve Docker images

d. VPC Permissions (RDS and ECS should be within a VPC)

ec2:DescribeVpcs (To describe VPCs)
ec2:DescribeSubnets (To describe subnets)
ec2:DescribeSecurityGroups (To describe security groups)
ec2:CreateNetworkInterface (If using awsvpc network mode)

e. CloudFormation Permissions

cloudformation:CreateStack
cloudformation:UpdateStack
cloudformation:DescribeStacks
cloudformation:DeleteStack

Adhere to the Principle of Least Privilege! Ensure that you grant only the permissions that are absolutely necessary for users or services to perform their required tasks.

Dockerfile Best Practices: Writing Efficient and Secure Docker Images

Uendi Hoxha — Mon, 07 Oct 2024 23:34:28 +0000

Docker allows developers to package applications with their dependencies into a lightweight, portable container. However, creating efficient and secure Docker images is crucial, especially in production environments where performance and security are paramount. In this article, we’ll explore best practices to help you write optimized and secure Dockerfiles, ensuring your containers are small, fast, and robust.

I. Choose the Right Base Image

The base image sets the foundation of your container. Opting for a lightweight base image can significantly reduce the size of your image and minimize security vulnerabilities.

Use official Docker images whenever possible, as they are maintained and regularly updated.
Prefer lightweight images like alpine over full OS images like ubuntu or debian. Alpine is only around 5 MB compared to 100+ MB for Ubuntu : FROM node:20-alpine

II. Leverage Multistage Builds for Smaller Images

Multistage builds allow you to separate the build environment from the final production image, ensuring the final image only contains the necessary runtime files. This helps in reducing the size of the image and removing build-time dependencies.

Use multistage builds to compile or build your application in one stage and only copy necessary artifacts to the next stage.

III. Minimize Layers

Each command in a Dockerfile adds a new layer to the final image. Reducing the number of layers and consolidating commands can lead to a more efficient image.

Combine multiple RUN commands into a single layer.
Avoid adding unnecessary files to the image.

# Instead of this:
RUN apt-get update
RUN apt-get install -y curl
RUN apt-get clean

# Use this:
RUN apt-get update && \
    apt-get install -y curl && \
    apt-get clean

IV. Use .dockerignore

Just like .gitignore, .dockerignore helps exclude unnecessary files from your Docker image, reducing its size and preventing sensitive files (like env files or Git directories) from being included in the build context.
Add unnecessary files like documentation, .git directories, and local configuration files to .dockerignore.

# .dockerignore
node_modules
.git
.env
README.md

V. Set User Permissions

By default, Docker containers run as the root user, which can pose security risks. It’s a good practice to run your containers with a non-root user wherever possible.

Use the USER directive to switch to a non-root user.
Create a user in the Dockerfile if one doesn’t exist in the base image.

# Add a user and switch to it
RUN addgroup -S appgroup && adduser -S appuser -G appgroup
USER appuser

CMD ["./myapp"]

VI. Optimize Caching with Build Arguments

Docker caches each layer during the build process, which can speed up subsequent builds. However, improper caching can lead to outdated dependencies or inefficient builds. Using build arguments can help control when the cache should be invalidated.

Add frequently changing commands (for example, COPY for source code) after more stable ones (like dependency installation). For example:

# First install dependencies (cacheable)
COPY package.json .
RUN npm install

# Then add source code (likely to change)
COPY . .

CMD ["npm", "start"]

By copying the package.json file before the source code, you allow Docker to cache the dependencies layer, saving time on rebuilds.

VII. Use Official Docker Image Scanning Tools

Docker images can contain security vulnerabilities. Regularly scan your images using tools like Docker Scan or AWS ECR Image Scanning to detect and fix potential issues.

Integrate security scanning into your CI/CD pipeline to catch vulnerabilities early.
Use tools like Docker Scan.

VIII. Avoid Hardcoding Secrets

Avoid adding sensitive information (like API keys, passwords, or tokens) directly into your Dockerfile. Instead, pass them securely using environment variables or Docker Secrets.

Use ARG and ENV for dynamic configurations, but ensure they are passed securely.
Utilize Docker Secrets or other secret management tools for production deployments.

ARG API_KEY
ENV API_KEY=$API_KEY

IX. Clean Up After Installing Dependencies

After installing packages or dependencies, ensure you clean up the temporary files and cache to keep the image lean.

Use apt-get clean or equivalent commands for other package managers.
Remove any temporary files after installation.

RUN apt-get update && \
    apt-get install -y curl && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

X. Use COPY Instead of ADD

While ADD can be used to copy files and fetch remote URLs, it's safer and more explicit to use COPY for local file transfers. Use ADD only when you need to extract tar files or download remote files.

Use COPY for local files to avoid unintended behavior.
Use ADD only for advanced use cases like fetching remote files.

Here’s an example Dockerfile that incorporates the best practices:

# Stage 1: Build Stage - Using multistage builds for smaller images
FROM node:20-alpine AS builder

# Set working directory
WORKDIR /app

# Install dependencies (cacheable layer)
COPY package.json package-lock.json ./
RUN npm install --production && \
    # Clean up npm cache after installing
    npm cache clean --force

# Copy source files
COPY . .

# Build the application
RUN npm run build

# Remove dev dependencies and unnecessary files
RUN rm -rf ./src ./tests ./node_modules && \
    npm install --production && \
    # Clean up any temporary files
    npm cache clean --force && \
    rm -rf /var/cache/apk/* /tmp/*

# Stage 2: Production Stage - Creating a lightweight final image
FROM node:20-alpine

# Set working directory
WORKDIR /app

# Copy necessary files from build stage
COPY --from=builder /app/dist ./dist
COPY --from=builder /app/node_modules ./node_modules

# Add a non-root user for security
RUN addgroup -S appgroup && adduser -S appuser -G appgroup
USER appuser

# Expose the port the app runs on
EXPOSE 3000

# Start the application
CMD ["node", "dist/index.js"]

# .dockerignore
node_modules
.git
.env
README.md
Dockerfile

Project Overview: AWS Inspector in Jenkins Pipeline

Uendi Hoxha — Fri, 04 Oct 2024 15:53:53 +0000

Let’s say you are deploying an application to an EC2 instance. You want to ensure that your infrastructure is secure before deployment. AWS Inspector can scan the EC2 environment for vulnerabilities, missing patches or insecure configurations.

By integrating AWS Inspector into Jenkins, the pipeline will run automated security scans each time a new build is made. If AWS Inspector detects security vulnerabilities, the deployment will be halted, ensuring that insecure code or configurations never reach production.

GOAL

Automate security testing using AWS Inspector during your Jenkins pipeline. After code builds, Jenkins will trigger AWS Inspector to scan the environment for vulnerabilities before deployment.

I. Set Up Jenkins

Ensure you have the following plugins installed in Jenkins:

Git (for version control integration)
Pipeline (to write pipelines as code)
AWS CLI (to interact with AWS services)
AWS Credentials (to securely store access keys)

II. Configure AWS Inspector

If you haven't configured AWS Inspector yet, follow these steps:

Step 1
Set up the necessary AWSInspectorRole role using IAM. This role must have permissions to create and manage findings and initiate security scans, for example:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "inspector:StartAssessmentRun",
        "inspector:ListFindings",
        "inspector:DescribeFindings",
        "inspector:ListAssessmentRuns"
      ],
      "Resource": "*"
    }
  ]
}

Step 2
Create an Assessment Target to specify which resources you want to evaluate in AWS Inspector. Choose Create assessment target and provide a name and select the resources (for example EC2 instances).
After creating the target, create an Assessment Template. Configure the Assessment Template in AWS Inspector to define what type of scans will run (like network, operating system).

Ensure the AWS Inspector has permissions to access your resources (EC2). You can attach a policy similar to the one above to the role assigned to AWS Inspector.

III. Set Up Jenkins Pipeline

Here’s an example Jenkins pipeline that integrates AWS Inspector to trigger a security scan:

pipeline {
    agent any

    environment {
        AWS_ACCESS_KEY_ID = credentials('aws-access-key')   // Store AWS credentials securely in Jenkins
        AWS_SECRET_ACCESS_KEY = credentials('aws-secret-key')
        REGION = 'us-east-1'  // Change to your region
        INSPECTOR_TEMPLATE_ARN = 'arn:aws:inspector:us-east-1:123456789012:template/0-ABCD1234'
    }

    stages {
        stage('Clone Repository') {
            steps {
                git 'https://github.com/your-repo/your-project'
            }
        }

        stage('Build') {
            steps {
                sh 'npm install'  // Example build step for a Node.js project
            }
        }

        stage('Run AWS Inspector') {
            steps {
                script {
                    // Trigger AWS Inspector assessment run
                    sh """
                    aws inspector start-assessment-run \
                        --assessment-template-arn $INSPECTOR_TEMPLATE_ARN \
                        --region $REGION
                    """
                }
            }
        }

        stage('Check Assessment Findings') {
            steps {
                script {
                    // Wait a bit for AWS Inspector to finish
                    sleep(time: 300, unit: 'SECONDS')

                    // Fetch findings from AWS Inspector
                    def findings = sh(script: """
                        aws inspector list-findings \
                        --assessment-run-arns $INSPECTOR_TEMPLATE_ARN \
                        --region $REGION
                    """, returnStdout: true).trim()

                    // Check if findings were detected
                    if (findings) {
                        echo "Security issues detected: $findings"
                        currentBuild.result = 'UNSTABLE'
                    } else {
                        echo "No security issues detected!"
                    }
                }
            }
        }

        stage('Deploy') {
            when {
                expression {
                    return currentBuild.result != 'UNSTABLE'
                }
            }
            steps {
                sh 'npm run deploy'  // Example deployment step
            }
        }
    }

    post {
        always {
            cleanWs()  // Cleanup workspace after run
        }
    }
}

Pipeline Stages

1. Clone Repository: Pulls the source code from a Git repository.
2. Build: Compiles or prepares the project for deployment (e.g., npm install, mvn package).
3. Run AWS Inspector: Starts an AWS Inspector assessment run. It uses the ARN of an existing assessment template.
4. Check Assessment Findings: After some time (around 5 minutes), the pipeline checks for any findings from AWS Inspector. If issues are found, it marks the build as unstable and stops the deployment.
5. Deploy: If no security vulnerabilities are detected, the pipeline proceeds to the deployment stage.

IV. Integrate AWS Credentials in Jenkins

In Jenkins, go to Manage Jenkins → Manage Credentials. Add your AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY as credentials.
Use these to securely interact with AWS services from the Jenkins pipeline.

V. Trigger the Pipeline

Save the pipeline configuration and lick on Build Now to run the pipeline. Monitor the logs to see each stage's progress and check the output of the AWS Inspector findings.

VI. Monitor and Review Findings

After the AWS Inspector stage runs, you can review the findings in the AWS Management Console under Amazon Inspector.

Terraform vs. AWS CloudFormation: A Detailed Comparison

Uendi Hoxha — Thu, 03 Oct 2024 15:29:38 +0000

Two prominent Infrastructure as Code (IaC) tools for automating cloud resources are Terraform and AWS CloudFormation. Both enable you to define, deploy, and manage cloud infrastructure efficiently. However, there are significant differences in terms of usability, multi-cloud capabilities, state management, etc. In this article I will provide an in-depth comparison between the two, including use cases, examples and more technical details.

What is Terraform?
Terraform is an open-source IaC tool developed by HashiCorp. It uses the declarative language HCL (HashiCorp Configuration Language) to define and manage infrastructure. Terraform is multi-cloud—it supports not only AWS, but also other cloud providers like Microsoft Azure, Google Cloud and even on-premise infrastructure.
This is how a simple instance would look like in terraform:

provider "aws" {
  region = "us-east-2"
}

resource "aws_instance" "example" {
  ami           = "ami-0c55b159cbfafe1f0"
  instance_type = "t2.micro"

  tags = {
    Name = "TerraformExample"
  }
}

In the example above, Terraform uses the AWS provider to launch an EC2 instance using a specific Amazon Machine Image (AMI) and instance type. After the script is written, running terraform apply will deploy the instance.

What is AWS CloudFormation?
AWS CloudFormation is Amazon’s native IaC tool, allowing AWS users to automate the deployment of infrastructure using JSON or YAML templates. CloudFormation provides an integration with AWS services, automatically manages dependencies and handles the creation, update, and deletion of resources.
Now let's see how an instance would look like in CloudFormation:

Resources:
  MyEC2Instance:
    Type: "AWS::EC2::Instance"
    Properties:
      InstanceType: "t2.micro"
      ImageId: "ami-0c55b159cbfafe1f0"
      Tags:
        - Key: Name
          Value: CloudFormationExample

The CloudFormation template above defines an EC2 instance using the AWS::EC2::Instance resource type. Similar to Terraform, running the aws cloudformation create-stack command will provision the instance.

Key Differences Between Terraform and AWS CloudFormation
a. Multi-Cloud vs AWS-Specific
Terraform's most significant advantage is its multi-cloud support. You can manage infrastructure across various cloud providers using a single tool and language. This makes it ideal for companies pursuing hybrid or multi-cloud strategies.
At the other hand, CloudFormation is AWS-specific. It’s tailored for AWS services and is integrated with the AWS ecosystem, giving you immediate access to the latest AWS features. If your infrastructure is fully based on AWS, CloudFormation may provide better AWS-specific optimizations and service integration.

b. Language and Syntax
Terraform uses the HCL syntax, designed to be human-readable and intuitive. HCL makes it easier to write infrastructure code, and its modular approach encourages code reuse. Modules in Terraform allow you to organize and standardize your infrastructure deployments.
Example of terraform module:

module "network" {
  source = "./modules/network"
  cidr_block = "10.0.0.0/16"
}

module "ec2" {
  source = "./modules/ec2"
  instance_type = "t2.micro"
}

CloudFormation templates are written in YAML or JSON, both of which are more verbose and can be harder to manage for large templates. However, YAML is still widely used and preferred over JSON for its readability. CloudFormation also offers nested stacks, which allow for some modularity but are more rigid than Terraform’s modules.

c. State Management
Terraform maintains a state file that records the infrastructure’s current status. This state file is critical for determining what changes are needed in the next deployment. However, managing state files especially in team environments can be challenging and requires careful handling (for example storing the state file in a remote backend like S3).

CloudFormation does not expose state to the user. AWS manages the state internally, which simplifies usage. You don’t need to worry about handling state files, which can reduce complexity for simpler deployments. However, for more complex deployments that need granular control over state, Terraform might be the better choice.

d. Error Handling and Rollbacks
Terraform provides detailed and informative error messages, which are helpful for debugging. However, in some cases Terraform might leave infrastructure in a partially deployed or failed state, requiring manual intervention to fix inconsistencies.

Meanwhile, CloudFormation has built-in rollback functionality. If a stack fails to deploy, CloudFormation will automatically attempt to revert to the last known stable state. This makes it more robust in terms of error recovery, especially for large deployments.

e. Provisioners and Extensibility
Terraform has the concept of provisioners, which allow you to execute scripts on your resources after they’re created. This feature makes it possible to configure servers or services in ways that go beyond basic resource creation.

resource "aws_instance" "example" {
  ami           = "ami-0c55b159cbfafe1f0"
  instance_type = "t2.micro"

  provisioner "local-exec" {
    command = "echo Instance created!"
  }
}

CloudFormation doesn’t support provisioners in the same way Terraform does. Instead, AWS recommends using services like AWS Lambda or AWS Systems Manager to execute post-deployment tasks. While these can achieve similar outcomes, they add extra complexity.

f. Compliance and Security
Terraform supports integrations with security and compliance tools like AWS Config and Cloud Custodian, but it requires custom configurations. Terraform is more flexible for companies with complex compliance needs spanning multiple cloud providers.

CloudFormation integrates with AWS Config and AWS Organizations, making it easier to implement compliance rules and security policies directly within AWS. For AWS-centric environments, CloudFormation may be more straightforward for enforcing compliance.

g. Cost
Terraform itself is free and open-source, though you might incur costs for remote state storage, version control, and CI/CD pipelines (e.g., using S3 or Terraform Cloud).

CloudFormation is free to use, as it’s included with AWS services. However, depending on the resources you deploy, there could be indirect costs like storage or execution time for rollback operations.

Here’s an outline I created with the key factors to consider when choosing between Terraform and AWS CloudFormation:

Container Orchestration with Kubernetes on AWS EKS

Uendi Hoxha — Tue, 01 Oct 2024 15:13:19 +0000

As we transition to microservices architectures, container orchestration becomes essential for managing complex application environments. Kubernetes is the leading open-source platform for automating deployment, scaling and operations of containerized applications. Amazon Elastic Kubernetes Service (EKS) simplifies Kubernetes by providing a managed service that automates much of the setup and management process. In this article, we will dive into technical details on how to set up, manage, and scale Kubernetes applications on AWS EKS.

Setting Up Kubernetes on AWS EKS
Let’s walk through the steps of setting up a Kubernetes cluster on EKS.

1. Install AWS CLI and eksctl
First, ensure that you have the necessary tools installed:

AWS CLI: To interact with AWS services.
eksctl: A command-line tool for creating and managing EKS clusters.

Install AWS CLI
$ curl "https://awscli.amazonaws.com/AWSCLIV2.pkg" -o "AWSCLIV2.pkg"
$ sudo installer -pkg AWSCLIV2.pkg -target /

Install eksctl
$ curl --silent --location "https://github.com/weaveworks/eksctl/releases/download/latest_release/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp
$ sudo mv /tmp/eksctl /usr/local/bin

2. Create an EKS Cluster
To create a Kubernetes cluster, use eksctl. This command will create a control plane and worker nodes (EC2 instances) for your cluster.

# Create an EKS Cluster
$ eksctl create cluster \
  --name my-eks-cluster \
  --version 1.25 \
  --region us-east-2 \
  --nodegroup-name my-nodes \
  --node-type t3.medium \
  --nodes 3 \
  --nodes-min 1 \
  --nodes-max 4 \
  --managed

This command will create a managed Kubernetes cluster with 3 EC2 nodes of type t3.medium, automatically scaling between 1 and 4 nodes based on resource requirements.

3. Configure kubectl to Access Your EKS Cluster
After the cluster is created, you’ll need to configure kubectl (Kubernetes CLI) to interact with it.

# Update kubeconfig with EKS cluster details
$ aws eks --region us-east-2 update-kubeconfig --name my-eks-cluster

Deploying Applications on EKS
Now that your Kubernetes cluster is running, let’s deploy a simple containerized application.

1. Create a Deployment
A Deployment is a Kubernetes resource that manages a set of identical pods. Here, we’ll deploy a simple Nginx web server.

# nginx-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80

Apply the deployment:
kubectl apply -f nginx-deployment.yaml
This will create 2 replicas of the Nginx server.

2. Expose the Deployment with a Service
To make the Nginx application accessible from outside the cluster, you need to create a Service.

# nginx-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: nginx-service
spec:
  selector:
    app: nginx
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80
  type: LoadBalancer

Apply the service:
$ kubectl apply -f nginx-service.yaml

This will create an AWS Elastic Load Balancer that routes traffic to your Nginx pods. You can find the external IP address (ELB) of the service:
$ kubectl get services

Managing Scaling with EKS
Kubernetes in EKS automatically handles horizontal scaling based on CPU and memory utilization. Let’s configure Horizontal Pod Autoscaler (HPA) for the Nginx deployment.

1. Enable Metrics Server
First, ensure that the Metrics Server is installed. This is a Kubernetes component required for autoscaling.

$ kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

2. Create Horizontal Pod Autoscaler
Next, create an HPA for the Nginx deployment:

$ kubectl autoscale deployment nginx-deployment --cpu-percent=50 --min=2 --max=10

This command will autoscale the Nginx deployment, ensuring that CPU utilization stays around 50%, and Kubernetes will automatically scale pods between 2 and 10 based on the load.

Securing AWS EKS with IAM and RBAC
1. IAM Roles for Service Accounts (IRSA)
Amazon EKS integrates tightly with AWS IAM to control access to resources. With IAM Roles for Service Accounts (IRSA), you can give specific permissions to pods by associating IAM roles with Kubernetes service accounts.

Here’s how you would set up IRSA for an application that needs to access S3:
Step 1: Create an IAM role with the required S3 permissions.
Step 2: Annotate the Kubernetes service account with the IAM role.

$ eksctl create iamserviceaccount \
  --name my-app-service-account \
  --namespace default \
  --cluster my-eks-cluster \
  --attach-policy-arn arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess \
  --approve

Avoiding Pitfalls: Essential Configuration Tips for AWS Lambda

Uendi Hoxha — Tue, 01 Oct 2024 15:11:45 +0000

In this article, I will cover common misconfigurations in lambda functions, their impact and how to resolve them. Topics will include VPC integration, setting appropriate permissions, connecting to RDS databases, monitoring and operational best practices, provisioned concurrency and also lambda layers.

I. Configuring AWS Lambda in a VPC

Do configure your lambda in a VPC! When you configure Lambda in a VPC, it can access resources like Amazon RDS databases, Amazon ElastiCache clusters or other services that are only available within your private subnet. This is essential for applications that require secure database connections without exposing those databases to the public internet.

When configuring Lambda functions in a VPC, it’s crucial to set up the right security groups to control traffic based on your application’s needs. For example, if your Lambda function needs to access an Amazon RDS instance, the security group associated with the RDS must allow inbound traffic from the security group associated with the Lambda function.

If the Lambda function needs to access other AWS services (e.g., S3, DynamoDB), ensure that the Lambda’s security group allows outbound traffic to those service endpoints. This is usually done by allowing all outbound traffic, as AWS services are designed to communicate within the AWS network securely.

If your Lambda needs to communicate with external APIs, configure the Lambda function to use a NAT Gateway or NAT instance for outbound internet access. The security group should allow outbound traffic to 0.0.0.0/0 for HTTP/HTTPS.

Enable VPC Flow Logs to capture information about the IP traffic going to and from network interfaces in your VPC. This can help you identify and troubleshoot network issues.

II. Setting the Right Permissions with AWS IAM

Lambdas often have overly broad IAM permissions or lack the necessary permissions, leading to either security risks or operational failures. Follow the principle of least privilege and assign only the necessary permissions using granular IAM roles and policies. Use IAM Roles for Lambda instead of attaching permissions directly to the function.

For example, if your Lambda function needs to access an S3 bucket and a DynamoDB table, attach policies that provide read/write access to those resources.

When defining IAM policies, aim for granularity. Instead of using wildcard permissions (e.g., s3:* or dynamodb:*), specify the exact actions your Lambda function needs to perform.
Instead of granting full access to S3, create a policy that only allows specific actions on a designated bucket:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject"
      ],
      "Resource": [
        "arn:aws:s3:::example-bucket/*"
      ]
    }
  ]
}

For a DynamoDB table, allow only the necessary operations:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "dynamodb:GetItem",
        "dynamodb:PutItem"
      ],
      "Resource": [
        "arn:aws:dynamodb:region:account-id:table/my-table"
      ]
    }
  ]
}

III. Connecting Lambda to RDS Databases

Using Amazon RDS Proxy is a recommended solution for alleviating the issues associated with connecting AWS Lambda to RDS databases. RDS Proxy acts as an intermediary between your Lambda function and the RDS instance, managing and pooling connections effectively. RDS Proxy enables your application to scale seamlessly, accommodating sudden spikes in traffic without degrading performance.

First, create an IAM role that grants RDS Proxy permission to connect to your RDS database. The policy should allow the rds-db:connect action on your database resources.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "rds-db:connect",
      "Resource": "arn:aws:rds-db:region:account-id:dbuser/my-db-user"
    }
  ]
}

Next, use the AWS CLI or the AWS Management Console to create the RDS Proxy.

Ensure that the security group associated with your RDS Proxy allows inbound traffic from your Lambda function. You may need to modify the security group rules to allow access on the database port (e.g., 3306 for MySQL).

In your Lambda function code, update the database connection string to point to the RDS Proxy endpoint rather than the RDS instance directly.
Example:

const mysql = require('mysql');
const connection = mysql.createConnection({
  host: 'my-db-proxy.proxy-abcdefghijkl.us-east-1.rds.amazonaws.com',
  user: 'my-db-user',
  password: 'my-db-password',
  database: 'my-database'
});

connection.connect((err) => {
  if (err) {
    console.error('Error connecting to the database:', err.stack);
    return;
  }
  console.log('Connected to the database.');
});

IV. Monitoring and Operations with CloudWatch

Without proper monitoring, it’s challenging to gain insights into your Lambda function’s performance, leading to potential issues like undetected errors, performance bottlenecks, resource wastage, etc.

By default, AWS Lambda automatically integrates with CloudWatch Logs. Each invocation of your function generates log entries that contain details about execution, errors and any output returned.

CloudWatch automatically collects several key metrics for Lambda functions, including:
Invocations: Number of times your function is invoked.
Duration: Time taken to execute the function.
Errors: Count of failed executions.
Throttles: Number of invocation requests that were throttled due to concurrency limits.

To proactively monitor your Lambda functions, configure CloudWatch Alarms to notify you of potential issues. For example, you can set up an alarm to trigger if the error rate exceeds a certain threshold. For example, creating an Alarm for High Error Rate:

aws cloudwatch put-metric-alarm --alarm-name HighErrorRate \
  --metric-name Errors --namespace AWS/Lambda --statistic Sum --period 300 \
  --threshold 5 --comparison-operator GreaterThanThreshold \
  --dimensions Name=FunctionName,Value=myLambdaFunction \
  --evaluation-periods 1 --alarm-actions arn:aws:sns:us-east-1:123456789012:my-sns-topic \
  --unit Count

Use CloudWatch Logs Insights to analyze and query your logs. This is an awesome feature that allows you to run SQL-like queries to find specific logs, helping with debugging and performance analysis. For example:

fields @timestamp, @message
| filter @message like /ERROR/
| sort @timestamp desc
| limit 20

For deeper insights, enable AWS X-Ray to trace requests through your Lambda function. This can help you understand latencies and identify bottlenecks in your application flow.

V. Provisioned Concurrency Configurations

Provisioned Concurrency is a feature that ensures AWS Lambda functions are always ready to respond instantly to incoming requests. By pre-initializing a specified number of function instances, Provisioned Concurrency helps eliminate cold starts, leading to improved performance and reduced latency.

To configure Provisioned Concurrency for a Lambda function, specify the amount of concurrency you want to provision. This ensures that a set number of instances are always warm and ready to handle requests.

Use CloudWatch to monitor metrics related to Provisioned Concurrency, such as the number of provisioned instances and the number of concurrent requests. This data can help you optimize the provisioned level based on usage patterns.

Considerations for Costs: While Provisioned Concurrency reduces cold starts, it comes at an additional cost. Be mindful of your application’s usage patterns to avoid over-provisioning, which can lead to unnecessary expenses.

VI. Using Lambda Layers for Code Reusability

Don't ever repeat your self! AWS lambda layers allows you to package and share common code, libraries, and dependencies across multiple Lambda functions.

To create a Lambda Layer, package the libraries and dependencies you want to reuse into a zip file. This zip file should contain a directory structure that follows the conventions for Lambda Layers. For example, if you’re including a Python library, it should be in the python/lib/python3.8/site-packages directory structure.
Example of packaging a python library:

mkdir -p my-layer/python/lib/python3.8/site-packages
pip install requests -t my-layer/python/lib/python3.8/site-packages/
zip -r my-layer.zip my-layer

Once the zip file is ready, publish it to AWS Lambda using the AWS CLI or the AWS Management Console.

Example using the AWS CLI to Publish a Layer:

aws lambda publish-layer-version --layer-name MyLayer --zip-file fileb://my-layer.zip --compatible-runtimes python3.8

After publishing the layer, you can include it in your Lambda functions. You can do this either when creating a new function or by updating an existing one:

aws lambda update-function-configuration --function-name myLambdaFunction --layers arn:aws:lambda:us-east-1:123456789012:layer:MyLayer:1

Each time you publish a new version of a layer, it gets a unique version ARN. This allows you to manage different versions of libraries independently. Please be cautious about breaking changes when updating layers that are used by multiple functions.

Integrating Data with AWS Glue, Dynamodb, S3 and Amazon Athena

Uendi Hoxha — Tue, 01 Oct 2024 12:39:47 +0000

Overview
AWS Glue is a fully managed ETL (Extract, Transform, Load) service that simplifies data preparation for analytics. This guide details the steps to extract data from two DynamoDB tables, transform it using AWS Glue, load it into Amazon S3, and analyze it using Amazon Athena.

Why Use AWS Glue?
Serverless Architecture: AWS Glue eliminates the need for server management, allowing users to focus on data integration without worrying about underlying infrastructure. This serverless model ensures that resources scale automatically based on workload.

Automated Data Cataloging: AWS Glue’s Data Catalog automatically discovers and stores metadata about data sources, making it easy to manage and access data. The catalog can integrate with various AWS services, providing a unified view of your data landscape.

Seamless Integration: AWS Glue natively integrates with a range of AWS services, such as DynamoDB, S3, and Athena, simplifying the process of moving data across the AWS ecosystem.

Support for Various Data Sources: AWS Glue supports multiple data formats and sources, making it versatile for different use cases. This flexibility allows organizations to centralize their data preparation efforts.

Scenario: Integrating Data from DynamoDB to S3 and Querying with Athena

Step 1

Extracting Data from DynamoDB
To set up AWS Glue for extracting data from DynamoDB, refer to the AWS Glue documentation on creating crawlers. Crawlers will automatically scan your DynamoDB tables to populate the Data Catalog with metadata.

Create Crawlers
Navigate to the AWS Glue Console.
Create a new crawler to scan the Customers and Transactions tables in DynamoDB.

Step 2

Transforming Data with ETL Jobs
Once the Data Catalog is populated, you can create an ETL job to transform the data. The AWS Glue documentation on ETL jobs provides a comprehensive guide.

AWS Glue provides a variety of transformation types to help you prepare and process your data efficiently during the ETL process. For more, check AWS documentation about transforming data with AWS Glue managed transforms.

Define ETL Logic:

Use AWS Glue Studio to create a job that joins the Customers and Transactions tables. Here’s an example snippet:

joined_df = Join.apply(customers_df, transactions_df, 'CustomerID', 'CustomerID')

Load Transformed Data to S3:

Specify an S3 bucket as the output location. AWS Glue can store the data in various formats (e.g., Parquet, CSV), which enhances query performance in Athena.

View in AWS Glue:

Step 3

Querying Data with Amazon Athena
After loading the transformed data into S3, you can use Amazon Athena to query it. Follow the Athena documentation to set up a table that points to your S3 bucket.

Run SQL Queries:
Leverage the power of SQL to analyze your data. For instance:

SELECT Name, SUM(Amount) as TotalSpent
FROM ecommerce_data
GROUP BY Name
ORDER BY TotalSpent DESC;

And there you go—you now have your transformed data queried in Amazon Athena!

DEV Community: Uendi Hoxha

My Thoughts on Data Mesh

SQL Query Optimization for Data Engineers

Strategies to Save Costs on AWS Services Without Compromising Performance

Project Overview: Real-Time Smart Building Monitoring System with Amazon Kinesis

Architecture Overview

Use Case Scenarios

Data Flow and Processing

Purposes of Analyzing Collected Data

Time for some demo...

Best Practices for Securing Amazon S3 Buckets

Best Practices for Securing S3 Buckets

Containerization and Deployment Using Amazon ECS and Fargate

I. Setting Up Your Development Environment

Initialize package.json and Install Dependencies

II. Writing Dockerfile

III. Configuring AWS KMS and Secrets Manager

IV. Setting Up Amazon RDS

V. Building and Pushing Docker Images

VI. Deploying with Amazon ECS and Fargate

VII. Permissions and IAM Roles

Dockerfile Best Practices: Writing Efficient and Secure Docker Images

I. Choose the Right Base Image

II. Leverage Multistage Builds for Smaller Images

III. Minimize Layers

IV. Use .dockerignore

V. Set User Permissions

VI. Optimize Caching with Build Arguments

VII. Use Official Docker Image Scanning Tools

VIII. Avoid Hardcoding Secrets

IX. Clean Up After Installing Dependencies

X. Use COPY Instead of ADD

Project Overview: AWS Inspector in Jenkins Pipeline

GOAL

I. Set Up Jenkins

II. Configure AWS Inspector

III. Set Up Jenkins Pipeline

IV. Integrate AWS Credentials in Jenkins

V. Trigger the Pipeline

VI. Monitor and Review Findings

Terraform vs. AWS CloudFormation: A Detailed Comparison

Container Orchestration with Kubernetes on AWS EKS

Avoiding Pitfalls: Essential Configuration Tips for AWS Lambda

I. Configuring AWS Lambda in a VPC

II. Setting the Right Permissions with AWS IAM

III. Connecting Lambda to RDS Databases

IV. Monitoring and Operations with CloudWatch

V. Provisioned Concurrency Configurations

VI. Using Lambda Layers for Code Reusability

Integrating Data with AWS Glue, Dynamodb, S3 and Amazon Athena

Step 1

Step 2

Step 3

Initialize `package.json` and Install Dependencies