DEV Community: Bernard Chika Uwaezuoke

Building a 7-Service Docker Compose Incident Response Platform

Bernard Chika Uwaezuoke — Fri, 03 Jul 2026 16:03:03 +0000

Introduction

In this project, I built a mini production-style Incident Response Platform using Docker Compose.

The goal was not just to run one container. The goal was to model how real applications are often structured in DevOps and platform engineering environments: multiple services, clear separation of responsibilities, persistent storage, background processing, routing, health checks, and monitoring.

The final platform has seven services:

Nginx edge reverse proxy
Frontend web UI
FastAPI backend API
Background worker
PostgreSQL database
Redis cache and queue
Prometheus monitoring

What Problem Does This Project Solve?

Every engineering team eventually deals with incidents:

An API becomes slow
A payment service fails
A database starts timing out
Error rates increase after a deployment
Users report that an application is unavailable

In many teams, these issues are first tracked manually in chats, spreadsheets, or random notes. That works for very small teams, but it quickly becomes messy.

This project provides a simple incident management workflow:

Create an incident
Assign severity
Assign ownership
Acknowledge the incident
Resolve the incident
Store incident records
Process background notifications
Expose metrics for monitoring

It is intentionally small, but the architecture follows patterns used in real systems.

Project Architecture

The platform runs as a Docker Compose stack. A browser connects to Nginx, and Nginx routes requests either to the frontend or the API.

The API talks to PostgreSQL for persistent storage and Redis for caching and background jobs. A worker consumes jobs from Redis. Prometheus scrapes metrics from the API.

The high-level request flow looks like this:

Browser
  -> Nginx edge service
    -> Frontend web UI
    -> FastAPI backend
      -> PostgreSQL
      -> Redis
        -> Worker
      -> Prometheus metrics endpoint

Project Folder Structure

I organized the project so each concern has its own place.

The important folders are:

incident-response-platform
|-- docker-compose.yml
|-- infra
|   |-- nginx
|   |   `-- nginx.conf
|   `-- prometheus
|       `-- prometheus.yml
|-- services
|   |-- api
|   |   |-- app
|   |   |-- Dockerfile
|   |   |-- requirements.txt
|   |   `-- worker.py
|   `-- frontend
|       |-- Dockerfile
|       |-- index.html
|       |-- app.js
|       `-- styles.css
|-- .env.example
`-- README.md

This structure keeps infrastructure configuration separate from application code.

Defining the Seven Services in Docker Compose

The heart of the project is the docker-compose.yml file.

services:
  edge:
    image: nginx:1.27-alpine
    container_name: incident-edge
    depends_on:
      - frontend
      - api
    ports:
      - "8081:80"
    volumes:
      - ./infra/nginx/nginx.conf:/etc/nginx/nginx.conf:ro
    networks:
      - incident-net
    restart: unless-stopped

  frontend:
    build:
      context: ./services/frontend
    container_name: incident-frontend
    expose:
      - "80"
    networks:
      - incident-net
    restart: unless-stopped

  api:
    build:
      context: ./services/api
    container_name: incident-api
    env_file:
      - .env
    environment:
      DATABASE_URL: postgresql://incident_app:incident_password@postgres:5432/incident_response
      REDIS_URL: redis://redis:6379/0
      ENVIRONMENT: local
    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_healthy
    expose:
      - "8000"
    healthcheck:
      test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8000/health').read()"]
      interval: 15s
      timeout: 5s
      retries: 5
      start_period: 20s
    networks:
      - incident-net
    restart: unless-stopped

  worker:
    build:
      context: ./services/api
    container_name: incident-worker
    command: ["python", "worker.py"]
    env_file:
      - .env
    environment:
      DATABASE_URL: postgresql://incident_app:incident_password@postgres:5432/incident_response
      REDIS_URL: redis://redis:6379/0
      ENVIRONMENT: local
    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_healthy
    networks:
      - incident-net
    restart: unless-stopped

  postgres:
    image: postgres:16-alpine
    container_name: incident-postgres
    environment:
      POSTGRES_DB: incident_response
      POSTGRES_USER: incident_app
      POSTGRES_PASSWORD: incident_password
    ports:
      - "5432:5432"
    volumes:
      - postgres-data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U incident_app -d incident_response"]
      interval: 10s
      timeout: 5s
      retries: 5
    networks:
      - incident-net
    restart: unless-stopped

  redis:
    image: redis:7-alpine
    container_name: incident-redis
    command: ["redis-server", "--appendonly", "yes"]
    ports:
      - "6379:6379"
    volumes:
      - redis-data:/data
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 10s
      timeout: 5s
      retries: 5
    networks:
      - incident-net
    restart: unless-stopped

  prometheus:
    image: prom/prometheus:v2.55.1
    container_name: incident-prometheus
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
      - "--storage.tsdb.path=/prometheus"
      - "--web.enable-lifecycle"
    ports:
      - "9090:9090"
    volumes:
      - ./infra/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prometheus-data:/prometheus
    depends_on:
      - api
    networks:
      - incident-net
    restart: unless-stopped

networks:
  incident-net:
    driver: bridge

volumes:
  postgres-data:
  redis-data:
  prometheus-data:

This Compose file gives the application a production-style shape:

Nginx is the only public application entry point
The frontend and API run as separate services
PostgreSQL and Redis have health checks
Data is stored in named Docker volumes
Services communicate over a private Docker network
Prometheus collects metrics from the API

Running the Stack

To start the platform, I used:

docker compose up --build

This command builds the custom API, worker, and frontend images, pulls the required base images, creates the network, creates volumes, and starts all services.

To check the containers:

docker ps

The main URLs are:

Application: http://localhost:8081
API docs:    http://localhost:8081/api/docs
Prometheus:  http://localhost:9090

I used port 8081 because Jenkins was already running on port 8080.

The Web Application

Once the stack is running, the browser UI is available at:

http://localhost:8081

From the UI, an operator can:

Create a new incident
Set the affected service
Choose the severity
Assign an owner
Acknowledge the incident
Resolve the incident

When an incident is created, the frontend sends a request to the API through Nginx.

The API

The backend API is built with FastAPI.

FastAPI gives the project automatic Swagger documentation, which is useful for testing the API directly from the browser.

The API exposes endpoints like:

GET    /health
GET    /incidents
POST   /incidents
PATCH  /incidents/{incident_id}
GET    /metrics

The /health endpoint is used by Docker health checks.

The /metrics endpoint exposes Prometheus-compatible metrics.

Background Jobs with Redis and Worker

When a new incident is created, the API does not do all the work inside the request.

Instead, it stores the incident in PostgreSQL and pushes a job into Redis.

The worker service listens for jobs from Redis and processes them asynchronously.

In this sample project, the worker simulates notifying the platform team. In a real-world system, this could be replaced with:

Slack notifications
Microsoft Teams notifications
Email alerts
PagerDuty escalation
ServiceNow ticket creation

This pattern is important because web requests should stay fast. Slow or external tasks should usually run in the background.

Monitoring with Prometheus

Prometheus is included as the monitoring service.

It scrapes the API metrics endpoint:

http://api:8000/metrics

Some example metrics exposed by the API are:

incidents_created_total
incidents_resolved_total
open_incidents

Prometheus is available locally at:

http://localhost:9090

This gives the project a real observability layer instead of only relying on logs.

Publishing to GitHub

After building and testing the project locally, I initialized Git, committed the code, created a GitHub repository, and pushed the project.

The basic Git workflow was:

git init
git add .
git commit -m "Initial Docker Compose incident platform"
gh repo create incident-response-platform --private --source . --remote origin --push

If you want to share your own project publicly, create the repository as public or change the visibility later in GitHub settings.

DevOps Skills Practiced

This project helped me practice several real DevOps concepts:

Docker Compose for multi-service applications
Nginx as a reverse proxy
FastAPI backend development
PostgreSQL persistence
Redis caching and background queues
Background worker architecture
Docker health checks
Docker networks
Docker volumes
Prometheus metrics
Git and GitHub workflow
Production-style application thinking

Useful Commands

Start the project:

docker compose up --build

Run in detached mode:

docker compose up -d --build

Check running containers:

docker compose ps

View all logs:

docker compose logs -f

View only API logs:

docker compose logs -f api

View worker logs:

docker compose logs -f worker

Stop the stack:

docker compose down

Stop and remove volumes:

docker compose down -v

What I Would Add Next

This project is a strong foundation, but there are several ways to improve it:

Add Grafana dashboards
Add authentication
Add real Slack or Teams notifications
Add CI/CD with GitHub Actions
Add container image scanning
Add automated tests
Add rate limiting in Nginx
Add TLS for HTTPS
Deploy it to a cloud environment

Conclusion

This project shows how Docker Compose can be used to build more than a simple container demo.

By combining Nginx, FastAPI, PostgreSQL, Redis, a worker, and Prometheus, I created a small but realistic incident response platform.

The biggest lesson is that DevOps is not only about running containers. It is about designing systems that are organized, observable, resilient, and easier to operate.

That is what this project demonstrates in a practical way.

Building Decoupled Event-Driven Microservices on AWS with SNS, SQS, Lambda, and DynamoDB

Bernard Chika Uwaezuoke — Thu, 02 Jul 2026 14:44:49 +0000

Introduction

Modern applications often begin as a single service where one request triggers several operations sequentially. An order-processing application, for example, might receive an order, process its payment, update inventory, and send a notification within the same synchronous request.

This design is simple at the beginning, but it becomes increasingly fragile as the application grows.

What happens when the payment provider is slow? What if the notification service is unavailable? Should an inventory failure prevent the application from accepting an otherwise valid order? How do we add a new analytics, fraud detection, or shipping service without modifying the original order service?

These are some of the problems that event-driven microservices are designed to solve.

In this project, I built a serverless, event-driven order-processing system using:

Amazon API Gateway
AWS Lambda
Amazon Simple Notification Service
Amazon Simple Queue Service
Amazon DynamoDB
Amazon CloudWatch
AWS Serverless Application Model

The central architectural decision was to use Amazon SNS for event distribution and Amazon SQS for durable, service-specific message buffering.

The result is an architecture in which payment, inventory, and notification services can operate, fail, recover, and scale independently.

Project repository: https://github.com/Donhadley22/aws-sns-sqs-microservices

Note: If the repository remains private, readers will not be able to access the source code until it is made public.

The problem with tightly coupled microservices

Consider the following synchronous workflow:

Client
  |
  v
Order Service
  |
  v
Payment Service
  |
  v
Inventory Service
  |
  v
Notification Service

Although the application has been divided into services, the services are still operationally dependent on one another.

The Order Service cannot finish until Payment responds. Payment might depend on Inventory, and Notification might become part of the same request chain.

This creates several problems.

Increased response time

The client must wait for every downstream operation to complete.

If payment takes two seconds, inventory takes one second, and notification takes three seconds, the request can take six seconds or longer.

Failure propagation

A notification failure could cause the entire order request to fail, even though the order and payment were successful.

Difficult scaling

Payment traffic, notification traffic, and inventory traffic may have different scaling patterns. A synchronous architecture makes it harder to scale them independently.

Difficult service expansion

Adding fraud detection, shipping, analytics, or audit services usually requires modifying the producing service or inserting more synchronous calls.

Tight runtime dependency

Every downstream service must be available at the same time as the upstream service.

This is precisely where messaging services become valuable.

Solution overview

The project uses the following architecture:

Client
   |
   v
Amazon API Gateway
   |
   v
Order Service Lambda
   |
   +------> Orders DynamoDB Table
   |
   v
Amazon SNS Order Events Topic
   |
   +------> Payment SQS Queue ------> Payment Lambda
   |                                     |
   |                                     +------> Payments Table
   |                                     |
   |                                     +------> SNS outcome event
   |
   +------> Inventory SQS Queue ----> Inventory Lambda
   |                                     |
   |                                     +------> Inventory Table
   |                                     |
   |                                     +------> SNS outcome event
   |
   +------> Notification SQS Queue -> Notification Lambda
                                             |
                                             +------> Notifications Table

Each SQS queue also has a dedicated dead-letter queue.

Amazon SNS and Amazon SQS are often used together because SNS can distribute one published event to multiple subscribers, while SQS stores each service's copy until that service is ready to process it. This means the producer and consumers do not need to be available at the same time.

Understanding the responsibility of each service

1. Order Service

The Order Service is exposed through Amazon API Gateway.

It is responsible for:

Receiving the HTTP request
Validating the order payload
Generating an order ID
Calculating the order total
Writing the order to DynamoDB
Publishing an OrderCreated event to Amazon SNS
Returning a 202 Accepted response

The service does not directly invoke Payment, Inventory, or Notification.

That is a critical design decision.

The Order Service knows that an order has been created, but it does not need to know every service interested in that event.

2. Payment Service

The Payment Service receives messages from its dedicated Payment SQS queue.

It:

Reads the OrderCreated event
Processes the payment rule
Stores the payment result
Publishes either PaymentCompleted or PaymentFailed

Because Payment owns its own queue, a payment-processing slowdown does not block Inventory or prevent the Order Service from accepting new requests.

3. Inventory Service

The Inventory Service independently receives its own copy of the OrderCreated event.

It:

Checks the requested items
Simulates stock reservation
Stores the inventory outcome
Publishes either InventoryReserved or InventoryFailed

Payment and Inventory can therefore process the same order in parallel.

Neither service needs to call the other.

4. Notification Service

The Notification Service is interested only in the results generated by Payment and Inventory.

It subscribes to:

PaymentCompleted
PaymentFailed
InventoryReserved
InventoryFailed

The service records a notification for the customer based on the event it receives.

In a production environment, this could be extended to deliver:

Email through Amazon SES
SMS through Amazon SNS
Mobile push notifications
Slack or Microsoft Teams notifications
Webhook callbacks
In-application notifications

Why use both SNS and SQS?

A frequent question from students is:

Why not publish directly to SQS, or trigger every service directly from SNS?

The answer lies in the different responsibilities of the two services.

Amazon SNS provides fan-out

Amazon SNS follows a publish-subscribe model.

The publisher sends one event to a topic. The topic distributes copies to all matching subscriptions.

The Order Service therefore publishes only once:

Order Service -> Order Events SNS Topic

SNS then distributes the event to:

Payment Queue
Inventory Queue

Additional consumers can be added later without changing the Order Service.

For example:

Fraud Detection Queue
Analytics Queue
Shipping Queue
Audit Queue

Amazon SQS provides durability and back-pressure

Amazon SQS stores messages until a consumer successfully processes them or their retention period expires.

This is important because consumers do not always process traffic at the same speed.

Suppose the system receives 10,000 orders during a promotional campaign. The Order Service can continue accepting requests while the Payment queue temporarily accumulates messages.

Payment workers can process the backlog according to their available concurrency.

The queue absorbs the traffic spike instead of allowing the spike to overload the Payment Service.

This is known as back-pressure handling.

Step-by-step event flow

Step 1: The client submits an order

The client sends:

POST /orders
Content-Type: application/json

Example request:

{
  "customerEmail": "student@example.com",
  "currency": "USD",
  "items": [
    {
      "sku": "LAPTOP-001",
      "quantity": 1,
      "price": 850
    },
    {
      "sku": "MOUSE-002",
      "quantity": 2,
      "price": 25
    }
  ],
  "paymentOutcome": "approved",
  "inventoryOutcome": "reserved"
}

API Gateway forwards the request to the Order Service Lambda function.

Step 2: The order is stored

The Order Service generates an order ID and stores the order in the Orders DynamoDB table.

Its initial state changes from:

ORDER_RECEIVED

to:

EVENT_PUBLISHED

after the event is successfully published.

Step 3: The Order Service publishes an event

The event follows a consistent envelope:

{
  "eventId": "unique-event-id",
  "eventType": "OrderCreated",
  "eventVersion": "1.0",
  "source": "order-service",
  "occurredAt": "2026-07-02T10:00:00.000Z",
  "data": {
    "orderId": "unique-order-id",
    "customerEmail": "student@example.com",
    "items": [],
    "totalAmount": 900,
    "currency": "USD"
  }
}

This envelope provides several important fields:

eventId supports tracing and deduplication.
eventType identifies the business event.
eventVersion supports schema evolution.
source identifies the producer.
occurredAt records when the event happened.
data contains the business payload.

A well-defined event contract is essential because events become APIs between services.

Changing an event structure without version control can break several consumers simultaneously.

Step 4: SNS applies subscription filters

The Payment subscription receives only OrderCreated events:

FilterPolicy:
  eventType:
    - OrderCreated

The Inventory subscription uses the same filter:

FilterPolicy:
  eventType:
    - OrderCreated

The Notification subscription receives only outcome events:

FilterPolicy:
  eventType:
    - PaymentCompleted
    - PaymentFailed
    - InventoryReserved
    - InventoryFailed

By default, an SNS subscription receives every message published to its topic. A filter policy limits delivery to messages whose attributes or body match the subscription's conditions.

Filtering prevents irrelevant events from reaching queues and reduces unnecessary Lambda invocations.

Step 5: Lambda processes messages from SQS

Each queue is configured as an event source for its corresponding Lambda function.

For example:

PaymentQueueEvent:
  Type: SQS
  Properties:
    Queue: !GetAtt PaymentQueue.Arn
    BatchSize: 5
    MaximumBatchingWindowInSeconds: 1
    FunctionResponseTypes:
      - ReportBatchItemFailures

The Lambda event source mapping polls SQS and sends messages to the function in batches.

The Payment and Inventory services use a batch size of five, while Notification uses a larger batch because notification records are lightweight.

Handling failures correctly

Building an event-driven system is not only about delivering messages. It is about designing what happens when processing fails.

This project separates failures into two categories.

Business failures

A business failure is a valid result that the application understands.

Examples include:

Payment declined
Product out of stock
Address validation failed
Customer account suspended

These should not normally be sent to a dead-letter queue.

The service should process the message successfully and publish a business outcome such as:

PaymentFailed
InventoryFailed

This allows other services to react to the failure.

For example, Notification can inform the customer that payment was declined.

Technical failures

A technical failure means the message could not be processed because of an unexpected problem.

Examples include:

Database unavailable
Malformed event
Dependency timeout
Permission failure
Uncaught application exception

These failures should be retried.

If processing continues to fail, the message should eventually move to a dead-letter queue.

Dead-letter queues

Every service queue in this project has its own DLQ:

Payment Queue ------> Payment DLQ
Inventory Queue ----> Inventory DLQ
Notification Queue -> Notification DLQ

The redrive policy is configured as follows:

RedrivePolicy:
  deadLetterTargetArn: !GetAtt PaymentDeadLetterQueue.Arn
  maxReceiveCount: 3

After three unsuccessful processing attempts, SQS moves the message to the relevant DLQ.

Dedicated DLQs provide failure isolation. A poison message for Payment does not pollute Inventory or Notification processing.

The DLQ retention period in this project is longer than the source queue retention period, which follows AWS guidance for preserving failed messages long enough for investigation.

CloudWatch alarms monitor the approximate number of visible messages in each DLQ.

In production, those alarms should notify the operations team through:

An SNS alert topic
Email
Slack
Microsoft Teams
PagerDuty
An incident-management platform

A DLQ should never become a graveyard where failed messages are ignored. Every DLQ needs an operational process for investigation, correction, replay, or deletion.

Partial batch failure handling

Assume Lambda receives five SQS messages and only one fails.

Without partial batch handling, the entire batch may be retried, including the four messages that were processed successfully.

This creates:

Duplicate work
Increased cost
Reduced throughput
More complicated data handling

The project enables:

FunctionResponseTypes:
  - ReportBatchItemFailures

Each consumer returns only the failed message IDs:

return {
  batchItemFailures: [
    {
      itemIdentifier: record.messageId
    }
  ]
};

Lambda then makes only those failed messages available for retry.

AWS recommends partial batch responses to avoid reprocessing successful messages when one record in an SQS batch fails.

Idempotency is not optional

Amazon SQS standard queues provide at-least-once processing semantics. This means the same message may occasionally be delivered more than once.

Therefore, a consumer must be able to process duplicate messages safely.

Imagine a Payment Service that charges a customer every time it receives a message. If the message is delivered twice, the customer could be charged twice.

That is unacceptable.

In this project, the Payment Service uses a conditional DynamoDB write:

await documentClient.send(
  new PutCommand({
    TableName: PAYMENTS_TABLE,
    Item: paymentResult,
    ConditionExpression: "attribute_not_exists(orderId)"
  })
);

The first message stores the payment.

A duplicate message encounters the existing orderId, preventing a second payment record from being created.

The project also generates deterministic outcome event IDs:

eventId: `${orderCreatedEvent.eventId}:payment-outcome`

If Payment stores its result but temporarily fails while publishing the outcome event, the original SQS message is retried.

On retry:

The duplicate database write is recognized.
The service safely republishes the outcome.
The deterministic event ID allows downstream consumers to detect the duplicate outcome.

Idempotency should be considered at every side-effect boundary:

Database writes
Payment requests
Inventory reservations
Emails
Webhooks
External API calls
Event publication

Visibility timeout configuration

When SQS delivers a message to a consumer, the message becomes temporarily invisible to other consumers.

This period is called the visibility timeout.

If the consumer successfully processes and deletes the message, processing is complete.

If the message is not deleted before the timeout expires, it becomes visible and can be processed again.

This project configures:

VisibilityTimeout: 120

The Lambda functions have:

Timeout: 15

The visibility timeout is therefore significantly longer than the function timeout.

AWS recommends setting the queue visibility timeout to at least six times the Lambda function timeout to allow time for retries if processing is throttled.

A visibility timeout that is too short can cause the same message to become visible while the first consumer is still processing it.

A timeout that is unnecessarily long can delay retries after genuine failures.

It should be selected based on actual processing duration, batching configuration, throttling risk, and retry requirements.

Long polling

The queues use:

ReceiveMessageWaitTimeSeconds: 20

This enables long polling.

Long polling allows SQS to wait briefly for a message instead of immediately returning an empty response. AWS documents that long polling can reduce empty and false-empty responses, with a maximum wait time of 20 seconds.

Although Lambda manages polling when SQS is configured as an event source, setting appropriate queue parameters remains useful for queue behavior and for any additional consumers that may poll the queue directly.

Infrastructure as Code with AWS SAM

The entire solution is defined in one AWS SAM template.

The template creates:

API Gateway routes
Four Lambda functions
One SNS topic
Three primary SQS queues
Three dead-letter queues
Four DynamoDB tables
SNS subscriptions
Queue policies
IAM permissions
Lambda event source mappings
CloudWatch alarms
CloudFormation outputs

This makes the architecture repeatable.

Instead of manually creating resources in the AWS Management Console, the environment can be deployed using:

npm run install:all
sam validate --lint
sam build
sam deploy --guided

The SAM build packages each Lambda service and creates the deployment artifacts.

The guided deployment captures the stack name, AWS Region, environment parameter, IAM capability confirmation, and local SAM configuration file.

After CloudFormation completes, the stack outputs include the topic, queue, table, and API details created by the template.

The API URL can then be read from the CloudFormation stack outputs and used for test requests.

Subsequent deployments use:

sam build
sam deploy

The stack can be removed with:

sam delete --stack-name sns-sqs-microservices

Infrastructure as Code provides several practical advantages:

Consistent environments
Version-controlled infrastructure
Easier peer review
Repeatable deployment
Reduced configuration drift
Easier disaster recovery
A foundation for CI/CD automation

The deployed resources are visible across the AWS console. The Lambda functions represent the four services in the workflow.

The DynamoDB tables give each service its own persistence boundary.

CloudFormation keeps the whole environment grouped as one repeatable stack.

Testing the architecture

A serious microservices project should test more than the successful path.

This project contains three test scenarios.

Test 1: Successful order

{
  "paymentOutcome": "approved",
  "inventoryOutcome": "reserved"
}

Expected results:

Order stored
Payment approved
Inventory reserved
Payment notification recorded
Inventory notification recorded
No DLQ messages

The successful request returns a 202 Accepted response with the generated orderId, eventId, and EVENT_PUBLISHED status.

Test 2: Business failure

{
  "paymentOutcome": "declined",
  "inventoryOutcome": "out_of_stock"
}

Expected results:

PaymentFailed published
InventoryFailed published
Failure notifications recorded
No DLQ messages

The services operated correctly. The business simply produced negative outcomes.

Test 3: Technical failure

{
  "simulateTechnicalFailure": "payment"
}

Expected behavior:

Payment processing throws an exception
The message becomes visible again
Lambda retries it
The receive count increases
The message moves to the Payment DLQ after three attempts
The Payment DLQ CloudWatch alarm enters ALARM state

Testing failure behavior is just as important as testing successful processing.

A system is not resilient merely because it has a queue. Resilience depends on retry policy, idempotency, timeout selection, DLQ handling, monitoring, and recovery procedures.

The transactional outbox consideration

The Order Service currently performs two operations:

1. Write the order to DynamoDB
2. Publish OrderCreated to SNS

These operations are not part of one atomic transaction.

Consider this sequence:

Order write succeeds
Application crashes
SNS publication never happens

The order exists, but Payment and Inventory never receive the event.

This is known as the dual-write problem.

For a learning project, the current implementation records an EVENT_PUBLISH_FAILED state so the failure can be identified.

For a production system, I would implement the transactional outbox pattern.

A DynamoDB-based version could work as follows:

Store the order and an outbox event in one DynamoDB transaction.
Enable DynamoDB Streams.
Invoke an event publisher from the stream.
Publish the outbox event to SNS.
Record successful publication or allow the outbox item to expire.

The transactional outbox pattern addresses inconsistent outcomes when an application must update a database and publish an event as part of the same business operation.

Business benefits

1. Better customer response times

The API does not need to wait for every downstream service to finish.

It can acknowledge that the order was accepted while processing continues asynchronously.

This provides a more responsive customer experience.

2. Reduced blast radius

If Notification fails, Payment and Inventory can continue operating.

If Inventory experiences a backlog, Payment can still process messages.

Failures are isolated to individual queues and consumers.

3. Independent scalability

Each microservice scales according to its own workload.

For example:

Payment Service: 100 concurrent workers
Inventory Service: 40 concurrent workers
Notification Service: 500 concurrent workers

The business does not need to scale the complete application just because one function is under heavy load.

4. Easier business expansion

New consumers can subscribe to the event topic without changing the original publisher.

A business can add:

Fraud detection
Customer loyalty points
Data analytics
Shipping fulfilment
Audit logging
Recommendation engines

This reduces the cost and risk of adding new capabilities.

5. Traffic-spike protection

Queues absorb sudden increases in traffic.

This is particularly valuable for:

Flash sales
Ticket releases
Registration deadlines
Payroll processing
Month-end billing
Marketing campaigns

The producer can accept work while consumers process the backlog at a controlled rate.

6. Improved operational visibility

Each queue exposes useful CloudWatch metrics, including:

Visible message count
In-flight message count
Age of the oldest message
Number of messages received
Number of messages deleted
DLQ depth

These metrics help teams identify which part of a business workflow is slowing down.

7. Cost-efficient serverless operation

The architecture uses managed services and does not require teams to operate message brokers or application servers.

The business pays based largely on usage while AWS manages the underlying infrastructure.

8. Stronger team autonomy

Different teams can own different services.

For example:

Checkout Team -> Order Service
Finance Team -> Payment Service
Operations Team -> Inventory Service
Customer Experience Team -> Notification Service

Teams can deploy and scale their services independently, provided they continue to respect the agreed event contracts.

Where this architecture can be applied

E-commerce order processing

OrderCreated
PaymentProcessed
InventoryReserved
ShipmentCreated
CustomerNotified

This is the most direct application of the project.

Financial transaction processing

TransactionSubmitted
FraudCheckRequested
ComplianceCheckCompleted
LedgerUpdated
CustomerAlerted

Additional controls are required for financial workloads, especially around exactly-once business effects, security, audit trails, and reconciliation.

User registration and onboarding

UserRegistered
IdentityVerificationRequested
WelcomeEmailRequested
CRMProfileCreated
AnalyticsEventRecorded

The registration API can remain responsive while secondary activities happen asynchronously.

Media-processing pipelines

FileUploaded
VirusScanRequested
VideoTranscodingRequested
ThumbnailGenerationRequested
MetadataExtractionRequested

Each processing stage can have its own queue and worker capacity.

Internet of Things workloads

DeviceReadingReceived
AnomalyDetectionRequested
DataArchived
AlertGenerated
DashboardUpdated

Queues help absorb bursts from thousands of devices.

Logistics and fulfilment

PackageCreated
WarehouseAssigned
DriverRequested
TrackingUpdated
CustomerNotified

Different services can react to fulfilment events without direct dependencies.

Insurance claims processing

ClaimSubmitted
DocumentValidationRequested
FraudAssessmentRequested
AdjusterAssigned
CustomerUpdated

Long-running workflows can be divided into independent processing stages.

Healthcare administrative workflows

AppointmentBooked
ReminderRequested
BillingRecordCreated
InsuranceCheckRequested
AuditRecordCreated

Sensitive workloads require appropriate security, privacy, encryption, logging, and regulatory controls.

CI/CD and DevOps automation

BuildCompleted
SecurityScanRequested
ArtifactPublished
DeploymentRequested
NotificationRequested

SNS and SQS can decouple build, scanning, deployment, audit, and notification tasks.

Where this architecture may not be appropriate

Event-driven architecture should not be adopted simply because it is modern.

It introduces:

Eventual consistency
More infrastructure components
More complex debugging
Schema-management requirements
Duplicate-delivery considerations
Distributed tracing challenges
More demanding operational procedures

A simple synchronous request may be better when:

The client requires an immediate final result.
The workflow contains only one fast and reliable dependency.
The application is small and unlikely to expand.
Eventual consistency is unacceptable.
The team lacks the operational maturity to manage distributed systems.

Architecture should solve a genuine business or engineering problem, not merely increase the number of AWS services in a diagram.

Key lessons from the project

Lesson 1: Microservices are not automatically decoupled

Breaking a monolith into multiple HTTP services does not remove coupling.

If every service synchronously depends on the next service, the system remains operationally coupled.

True decoupling requires careful control of dependencies, contracts, data ownership, and failure behavior.

Lesson 2: Give every consumer its own queue

Payment and Inventory should not compete for messages from one shared queue.

With one queue per microservice, each interested service receives its own copy and controls its own retries, backlog, scaling, and DLQ.

Lesson 3: Events should describe facts

Good event names describe something that has happened:

OrderCreated
PaymentCompleted
PaymentFailed
InventoryReserved

Avoid vague names such as:

ProcessData
HandleRequest
RunService

Business-oriented event names make architectures easier to understand.

Lesson 4: Business failures are not technical failures

A declined payment is not a processing exception.

It is a valid business outcome and should be published as an event.

Sending every negative outcome to a DLQ makes operations noisy and removes valuable business meaning.

Lesson 5: At-least-once delivery requires idempotency

Duplicate delivery is not an unusual edge case that can be ignored.

Every consumer that performs a side effect must define how duplicates are detected and handled.

Lesson 6: DLQs need ownership

Creating a DLQ is not enough.

Teams need:

Alerts
Runbooks
Dashboards
Investigation procedures
Redrive procedures
Message-retention policies
Service ownership

Lesson 7: Event schemas need governance

An event is a contract.

Production systems should define:

Schema ownership
Event versioning
Required and optional fields
Compatibility rules
Validation
Deprecation procedures
Consumer contract tests

Lesson 8: Observability must cross service boundaries

A single request can generate several events across several services.

Use fields such as:

eventId
correlationId
orderId
source
eventType
occurredAt

These values should appear consistently in structured logs so teams can trace a workflow from beginning to end.

Lesson 9: Design failure paths before production

Successful requests are usually the easiest part of the system.

The difficult questions are:

What happens when publication fails?
What happens when a consumer times out?
What happens when the same event arrives twice?
What happens when a DLQ begins growing?
How is a failed event replayed safely?

These questions should be answered during architecture design, not after an incident.

Lesson 10: Start simple, then harden deliberately

This project is intentionally understandable.

A production version can later add:

Amazon Cognito or another identity provider
AWS WAF
AWS KMS customer-managed keys
AWS Secrets Manager
Amazon SES
AWS X-Ray tracing
CloudWatch dashboards
AWS Lambda Powertools
Event schema validation
Transactional outbox
CI/CD with GitHub Actions and AWS OIDC
Multi-account environments
Reserved or maximum concurrency
Replay and redrive automation

Starting with a clear foundation is better than introducing every possible service before the application has demonstrated the need.

Final thoughts

The primary value of this project is not simply that it uses Amazon SNS and Amazon SQS.

Its value is that it demonstrates several fundamental distributed-system principles:

Producers should not need to know every consumer.
Consumers should own their queues.
Failures should be isolated.
Messages should be retried safely.
Duplicate delivery should not create duplicate business effects.
Business outcomes should be represented as events.
Operations teams should be able to detect and recover failed work.

Amazon SNS handles event distribution.

Amazon SQS provides durability, buffering, retry isolation, and independent consumer control.

AWS Lambda provides serverless processing.

DynamoDB provides service-owned persistence.

AWS SAM makes the complete environment repeatable through Infrastructure as Code.

Together, these services create an architecture that can respond quickly to customers, absorb traffic spikes, isolate failures, scale individual workloads, and support new business capabilities without repeatedly rewriting the original Order Service.

That is the real purpose of decoupling: not to make the architecture diagram more impressive, but to make the business more adaptable and the system more resilient.

Deploying Infrastructure to Azure Using HCP Terraform and GitHub VCS

Bernard Chika Uwaezuoke — Thu, 04 Jun 2026 16:42:00 +0000

INTRODUCTION

Every Cloud Engineer eventually reaches the same point.
At first, running Terraform from your laptop feels exciting. You write your .tf files, run:

terraform init terraform plan terraform apply
and Azure resources appear like magic.

But as projects grow, that approach becomes risky.
What happens when another engineer needs to review your changes?
What happens when the state file is sitting on someone’s laptop?
What happens when a junior engineer accidentally applies changes to production without approval?

That is where HCP Terraform and GitHub VCS integration come in.
In this project, I will walk you through how to deploy Azure infrastructure using Terraform code stored in GitHub, with HCP Terraform handling remote runs, state management, plan review, and controlled apply operations.

As a Cloud Engineer and technical instructor, this is the kind of workflow I encourage students and teams to learn early because it reflects how real infrastructure should be managed: version-controlled, reviewable, auditable, and repeatable.

HCP Terraform supports a VCS-driven workflow where it fetches Terraform configuration from a connected repository and automatically starts plan and apply operations when changes are pushed. This allows GitHub to act as the source of truth for infrastructure code.

Project Goal

We are going to deploy basic Azure infrastructure using:

Azure as the cloud provider
Terraform as the Infrastructure as Code tool
GitHub as the version control system
HCP Terraform as the remote execution and state management platform

The deployment will create:

Azure Resource Group
Virtual Network
Subnet
Network Security Group
Public IP
Network Interface
Linux Virtual Machine

Architecture Overview

In simple terms:

We write Terraform code locally.
We push the code to GitHub.
HCP Terraform detects the change.
HCP Terraform runs terraform plan.
After approval, HCP Terraform runs terraform apply.
Azure resources are deployed.

This is a professional DevOps workflow because no one is manually creating resources in the Azure portal, and no one is running production infrastructure from their laptop (Minimum DevOps requirements).

Prerequisites

Before starting, make sure you have:

An Azure subscription
A GitHub account
An HCP Terraform account
Terraform installed locally
Azure CLI installed locally
Basic knowledge of Git and Terraform

You also need permission to create resources in Azure and permission to create or use a service principal click here to know more.

Here We Go!

Step 1: Create a GitHub Repository

Create a new GitHub repository.
azure-hcp-terraform-vcs
Clone the repository locally
git clone https://github.com/YOUR_USERNAME/azure-hcp-terraform-vcs.git
Switch to the cloned repository
cd azure-hcp-terraform-vcs

Creating a Repo in GitHub

For details on how to create a GitHub repository, click here

Create the Terraform project structure azure-hcp-terraform-vcs/

│
├── main.tf
├── variables.tf
├── outputs.tf
├── providers.tf
├── terraform.tfvars
└── README.md

The above structure is simple enough for beginners to understand, but also professional enough to build on later.

Step 2: Configure the Azure Provider

Create providers.tf.

terraform {
  required_version = ">= 1.6.0"

  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~> 4.0"
    }
  }
}

provider "azurerm" {
  features {}
}

This tells Terraform to use the AzureRM provider.
The features {} block is required by the AzureRM provider.

Step 3: Create Terraform Variables

Create variables.tf.

variable "resource_group_name" {
  description = "Name of the Azure Resource Group"
  type        = string
}

variable "location" {
  description = "Azure region where resources will be deployed"
  type        = string
}

variable "vnet_name" {
  description = "Name of the Virtual Network"
  type        = string
}

variable "vnet_address_space" {
  description = "Address space for the Virtual Network"
  type        = list(string)
}

variable "subnet_name" {
  description = "Name of the subnet"
  type        = string
}

variable "subnet_address_prefixes" {
  description = "Address prefixes for the subnet"
  type        = list(string)
}

variable "vm_name" {
  description = "Name of the Linux Virtual Machine"
  type        = string
}

variable "admin_username" {
  description = "Admin username for the virtual machine"
  type        = string
}

variable "public_key" {
  description = "SSH public key content for the virtual machine"
  type        = string
}

Important Note

In HCP Terraform, avoid using local file paths like this:
file("C:/Users/USER/.ssh/azure_rsa.pub")
That path exists on your laptop, not inside the HCP Terraform remote execution environment.
Instead, paste the actual public key content into an HCP Terraform variable and check the sensitive atribute box.

Step 4: Create the Azure Infrastructure Code

Create main.tf.

resource "azurerm_resource_group" "rg" {
  name     = var.resource_group_name
  location = var.location
}

resource "azurerm_virtual_network" "vnet" {
  name                = var.vnet_name
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name
  address_space       = var.vnet_address_space
}

resource "azurerm_subnet" "subnet" {
  name                 = var.subnet_name
  resource_group_name  = azurerm_resource_group.rg.name
  virtual_network_name = azurerm_virtual_network.vnet.name
  address_prefixes     = var.subnet_address_prefixes
}

resource "azurerm_network_security_group" "nsg" {
  name                = "${var.vm_name}-nsg"
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name

  security_rule {
    name                       = "Allow-SSH"
    priority                   = 1001
    direction                  = "Inbound"
    access                     = "Allow"
    protocol                   = "Tcp"
    source_port_range          = "*"
    destination_port_range     = "22"
    source_address_prefix      = "*"
    destination_address_prefix = "*"
  }
}

resource "azurerm_public_ip" "public_ip" {
  name                = "${var.vm_name}-public-ip"
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name
  allocation_method   = "Static"
  sku                 = "Standard"
}

resource "azurerm_network_interface" "nic" {
  name                = "${var.vm_name}-nic"
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name

  ip_configuration {
    name                          = "internal"
    subnet_id                     = azurerm_subnet.subnet.id
    private_ip_address_allocation = "Dynamic"
    public_ip_address_id          = azurerm_public_ip.public_ip.id
  }
}

resource "azurerm_network_interface_security_group_association" "nsg_association" {
  network_interface_id      = azurerm_network_interface.nic.id
  network_security_group_id = azurerm_network_security_group.nsg.id
}

resource "azurerm_linux_virtual_machine" "vm" {
  name                = var.vm_name
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name
  size                = "Standard_D2s_v3"
  admin_username      = var.admin_username

  network_interface_ids = [
    azurerm_network_interface.nic.id
  ]

  admin_ssh_key {
    username   = var.admin_username
    public_key = var.public_key
  }

  os_disk {
    name                 = "${var.vm_name}-osdisk"
    caching              = "ReadWrite"
    storage_account_type = "Standard_LRS"
  }

  source_image_reference {
    publisher = "Canonical"
    offer     = "ubuntu-24_04-lts"
    sku       = "server"
    version   = "latest"
  }
}

This code builds a simple Azure VM environment with networking, security, and SSH access.
This is a good starter project because it touches the core Azure infrastructure components: compute, networking, identity, and security.

Step 5: Create Outputs

Create outputs.tf.

output "resource_group_name" {
  description = "The name of the created resource group"
  value       = azurerm_resource_group.rg.name
}

output "public_ip_address" {
  description = "The public IP address of the virtual machine"
  value       = azurerm_public_ip.public_ip.ip_address
}

output "vm_name" {
  description = "The name of the virtual machine"
  value       = azurerm_linux_virtual_machine.vm.name
}

Outputs are useful because after deployment, HCP Terraform will show important values such as the public IP address of the VM.

Step 6: Create an Example tfvars File

Create terraform.tfvars

resource_group_name      = "rg-hcp-terraform-demo"
location                 = "westeurope"
vnet_name                = "vnet-hcp-demo"
vnet_address_space       = ["10.0.0.0/16"]
subnet_name              = "subnet-hcp-demo"
subnet_address_prefixes  = ["10.0.1.0/24"]
vm_name                  = "vm-hcp-demo"
admin_username           = "azureuser"
public_key               = "ssh-rsa AAAA..."

Do not commit real secrets or sensitive values into GitHub.
For public_key, use the content of your .pub file.
For example:
cat ~/.ssh/azure_rsa.pub
On Windows PowerShell:
Get-Content C:\Users\USER\.ssh\azure_rsa.pub
Copy the full public key content.

Step 7: Create an Azure Service Principal

HCP Terraform needs permission to deploy resources into Azure.
The common approach is to create a service principal and store its values as environment variables in HCP Terraform. Microsoft recommends using service principals for automated tools such as Terraform instead of using a fully privileged personal user account.

Login to Azure:
az login
Confirm your subscription:
az account show
Set the subscription:
az account set --subscription "<SUBSCRIPTION_ID>"
Create a service principal:

az ad sp create-for-rbac \
  --name "sp-hcp-terraform-azure" \
  --role "Contributor" \
  --scopes "/subscriptions/<SUBSCRIPTION_ID>"

The output will look similar to this:

{
  "appId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
  "displayName": "sp-hcp-terraform-azure",
  "password": "xxxxxxxxxxxxxxxxxxxx",
  "tenant": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
}

Save these values securely.
You will need them in HCP Terraform.

Step 8: Push Your Code to GitHub

In your working director, run:

git status
git add .
git commit -m "Initial Azure infrastructure with Terraform"
git push origin main

At this point, GitHub becomes the source of truth for your infrastructure code.
This is a major shift from manual cloud administration to real Infrastructure as Code practice.

The files in the repository in GitHub

Step 9: Create an HCP Terraform Organization

Go to HCP Terraform and create an organization. Click here
Click on Create Organization
Select Personal
A good organization naming pattern could be:
peabsmart-cloud-labs

On the left pane inside the organization click on Projects, then click on New project and enter the project detail and Click on Create. For this project, we will use Azure-Training-Projects

Projects in HCP Terraform help organize related workspaces, especially when you are managing multiple environments or cloud providers. HashiCorp’s HCP Terraform learning path includes project and variable set organization as part of the recommended collaboration workflow.

Step 10: Connect HCP Terraform to GitHub and Create a HCP Terraform Workspace

In the project you created, Click on Create a workspace HCP Terraform:

Select Version Control Workflow
Choose GitHub.

Authorize HCP Terraform to access your GitHub account or organization.
Select the repository: azure-hcp-terraform-vcs
Click on Create

Terraform working directory: /

Use / if your .tf files are in the root of the repository.
If your files are inside a folder liketerraform/, then set the working directory to: terraform

This is a common mistake. If the working directory is wrong, HCP Terraform will not find your Terraform configuration.

The Terraform variable page will popup
Enter the required variable values from terraform.tfvars file we created earlier and click Save variables.

Step 11: Add Azure Credentials to HCP Terraform

Go to the workspace:
Workspace > Variables
Add the following as Environment Variables:
ARM_CLIENT_ID ARM_CLIENT_SECRET ARM_SUBSCRIPTION_ID ARM_TENANT_ID

Use these values:

ARM_CLIENT_ID        = appId from the service principal output
ARM_CLIENT_SECRET    = password from the service principal output
ARM_SUBSCRIPTION_ID  = your Azure subscription ID
ARM_TENANT_ID        = tenant from the service principal output
Mark ARM_CLIENT_SECRET as Sensitive.

Microsoft documents these Azure Terraform authentication variables as the standard service-principal environment variables for Terraform automation.

Step 12: Queue the First Terraform Plan

Once the workspace is connected to GitHub and variables are added, HCP Terraform will usually trigger a run automatically.

You can also manually queue a run:
Workspace > Runs > Queue plan
HCP Terraform will run:
terraform init
terraform plan
remotely.
This is where the value becomes clear.
The plan is no longer hidden on one person’s machine. It is visible, reviewable, and part of the deployment workflow.

Step 13: Review the Plan

A good Cloud Engineer does not blindly apply infrastructure changes.
Review the plan carefully.
Look for:

Resources to be created
Resources to be modified
Resources to be destroyed
Unexpected region changes
Unexpected VM sizes
Public exposure
Security group rules
Naming mistakes

For example, if the plan says:
Plan: 8 to add, 0 to change, 0 to destroy.
that means Terraform intends to create 8 resources.
A depicted in the image below

Step 14: Apply the Infrastructure

If the plan looks correct, click: Confirm & Apply

HCP Terraform will run the apply operation remotely.
After a successful apply, you should see your outputs:

resource_group_name = rg-hcp-terraform-demo
vm_name             = vm-hcp-demo
public_ip_address   = x.x.x.x
You can now test SSH access:
ssh azureuser@<PUBLIC_IP_ADDRESS>

Apply completed

Prove of authorization after review (Reviewer's Comment)

Step 15: Verify Resources in Azure

Go to the Azure portal and check:
Resource Groups > rg-hcp-terraform-demo
You should see:
Virtual Machine
Virtual Network
Subnet
Network Security Group
Public IP
Network Interface
OS Disk

As Depicted below

You can also verify using Azure CLI:

az resource list \
  --resource-group rg-hcp-terraform-demo \
  --output table

Step 16: Make a Change Through GitHub

Imagine your team wants to change the VM size.

Update this line in main.tf:
size = "Standard_D2s_v3"
Change it to:
size = "Standard_B1s"
Commit and push:

git add .
git commit -m "Update VM size"
git push origin main

HCP Terraform detects the GitHub change and starts a new run.
This is the power of VCS-driven infrastructure.
You are no longer clicking around in Azure. You are managing infrastructure through code, Git history, and controlled deployment runs.

Step 17: Destroy the Infrastructure Safely

When you are done with the demo, destroy the resources to avoid unnecessary Azure cost.

In HCP Terraform: Workspace > Settings > Destruction and Deletion > Queue destroy plan

Confirm by typing delete
Review the destroy plan, then confirm.

Deletion Completed

HCP Terraform supports destroy operations from the workspace, and HashiCorp warns that deleting a workspace does not automatically destroy the infrastructure it manages. You must queue a destroy plan if you want the actual cloud resources removed.

Common Errors and How to Fix Them

Error 1: Unsupported Operator for public_key_path

Example error:
invalid HCL for variable "public_key_path" at 1,1: Unsupported operator
Cause:
You are probably passing a Windows path like this directly into HCP Terraform:
C:/Users/USER/.ssh/azure_rsa.pub
HCP Terraform runs remotely and cannot access files on your local machine.
Fix:
Use the actual public key content instead:
public_key = "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQ..."

Error 2: List of String Required, But Have String

Bad value:
vnet_address_space = "[\"10.0.0.0/16\"]"
Correct value:
vnet_address_space = ["10.0.0.0/16"]
Cause:
The first example is a string. The second example is a real HCL list.

Error 3: Azure Authentication Failed

Check that these environment variables exist in HCP Terraform:
ARM_CLIENT_ID ARM_CLIENT_SECRET ARM_SUBSCRIPTION_ID ARM_TENANT_ID
Also confirm that the service principal has enough permissions on the subscription or resource group.

Error 4: Terraform Cannot Find Configuration

Cause:
Your HCP Terraform workspace working directory is wrong.
If your files are here:
repo/main.tf
use:
/
If your files are here:
repo/terraform/main.tf
use:
terraform

Best Practices I Teach My Students

1. Do Not Store Secrets in GitHub

Never commit:

client secrets
passwords
private keys
.tfstate files Use HCP Terraform sensitive variables for secrets.

2. Use Git Branches and Pull Requests

For production teams, avoid pushing directly to main.
Use:
feature branch > pull request > review > merge > HCP Terraform run
This improves collaboration and prevents careless infrastructure changes.

3. Keep State Remote

One of the strongest benefits of HCP Terraform is remote state management.
Your state is not scattered across laptops.
It is centrally managed, protected, and connected to the workspace.

4. Separate Environments

For serious projects, create separate workspaces:

azure-dev
azure-staging
azure-prod Each workspace should have its own variables and approval controls.

5. Use Clear Naming Standards

Bad names:

test-rg
vm1
network-demo

Better names:

rg-dev-hcp-demo-weu
vnet-dev-hcp-demo-weu
vm-dev-hcp-demo-01 Good naming helps with operations, cost tracking, troubleshooting, and governance.

Final Reflection

This project is more than just deploying a virtual machine to Azure.
It demonstrates a professional Infrastructure as Code workflow.
As a Cloud Engineer, I see Terraform as more than a provisioning tool. It is a discipline. It teaches structure, repeatability, version control, collaboration, and accountability.

As a technical instructor, I also see this workflow as one of the best ways to help students understand how modern cloud teams work. They learn that cloud engineering is not about clicking buttons in the portal. It is about building systems that are predictable, secure, reviewable, and easy to reproduce.

A Real-World Serverless Appointment Booking Backend on AWS

Bernard Chika Uwaezuoke — Mon, 12 Jan 2026 23:02:56 +0000

From “Hello World” to a Real-World, Production-Ready System

Introduction: From Demo to Impact.

Most tutorials stop at “Hello from Serverless API!”
That is useful as a starting point, but it doesn’t solve a real problem.

In the real world, businesses do not need “Hello World.”
They need customers booked, schedules organized, and systems that scale without breaking.
In this blog, we’ll take a simple AWS Lambda + API Gateway setup and evolve it into a Smart Appointment Booking & Management API. A serverless backend that could realistically power a clinic, salon, or consulting business.

No servers.
No infrastructure headaches.
Just clean, scalable cloud architecture.

Problem Definition

Most small service-based businesses often face:
• Manual appointment booking
• Double bookings
• No automated reminders
• Poor visibility into schedules
Hiring engineers to manage servers is expensive.
This is where serverless architecture shines.

The Serverless Solution Architecture

Prerequisites

Active AWS account
Basic understanding of serverless architecture
Knowledge of Python or Node.js
Familiarity with REST APIs and HTTP methods
Basic understanding of authentication (JWT)
Fundamental AWS IAM concepts
Ability to use API testing tools (Postman or cURL)

Now let's dive in!

Step 1: Creating the Core Lambda Function

1. Create the Lambda Function

Sign into AWS Console here
Search for and select Lambda in the global search bar.

Click on the Create a function button.

Select Author from scratch
Configure:
o Function Name: AppointmentAPI
o Runtime: Python 3.x
o Execution Role: Create new role with basic permissions
Click Create Function as shown in the images below.

2. Basic Lambda Handler (Foundation)

Go to the Code section of the function you just created and replace the body of code there with the code below.

import json

def lambda_handler(event, context):
    return {
        "statusCode": 200,
        "body": json.dumps({
            "message": "Appointment API is running"
        })
    }

Click on the Deploy button to deploy the function.

Click on the Test button and Create an Event to test your function, then click the Test button as shown below.

The function ran successfully!

Step 2: Exposing the API with API Gateway

1. Create REST API

Search for and select API Gateway in the global search bar.

Click Create API
Choose REST API
Select Regional
Select Security Policy (as shown in the image below)
Click on Create API button

2. Create Resource

Resource name: appointments
Path: /

3. Create Methods

Inside the created API Resource, click on the Create method button.

Add the following methods:

Method	Purpose
POST	Create appointment
GET	Fetch appointments
PUT	Update appointment
DELETE	Cancel appointment

Each method integrates with the same Lambda function.

Steps to Create a Method

For a POST /appointments method:

Select the /appointments resource
Click Actions → Create Method
Choose POST
Toggle the switch for Lambda proxy integration
Set Integration type to Lambda Function
Enable Lambda Proxy Integration

Choose your region and select your Lambda (AppointmentAPI)

Click Create method.

What it does: Creates a new appointment entry (writes to DynamoDB).

*For a GET /appointments method *

Repeat the same steps, but choose GET:

/appointments → Actions → Create Method → GET
Integration type: Lambda Function
Enable Lambda Proxy Integration
Select the same Lambda function
Click Create method

What it does: Returns appointments (for a user or provider).

For a PUT /appointments method.

Repeat again for PUT:

/appointments → Actions → Create Method → PUT
Integrate with the same Lambda
Enable Lambda Proxy Integration
Click Create method

What it does: Updates an appointment (reschedule, status update, etc.)

For a DELETE /appointments method.

Repeat for DELETE:

/appointments → Actions → Create Method → DELETE
Integrate with same Lambda
Enable Lambda Proxy Integration
Click Create method

What it does: Cancels an appointment (soft delete recommended, by setting status = canceled).

The four CRUD Methods.

Step 3: Designing the Data Model (DynamoDB)

Before we write a single line of booking logic, we need to get the data model right. In serverless systems, your database design is everything, because your API performance, cost, and scalability depend on it.

For this Smart Appointment Booking API, we’ll use Amazon DynamoDB: a fully managed NoSQL database designed for fast reads/writes at massive scale.

Setting up the DynamoDB Table

1. Open DynamoDB

Search for and select Lambda in the global search bar.

Click DynamoDB → Tables → Create table

2. Create the Appointments Table

On Create table pane, enter:

Table details

Table name: Appointments

Partition key (PK): userId (Type: String)

Sort key (SK): appointmentId (Type: String)

3. Choose Table Settings

Under Table settings, choose On-demand (Pay per request)

This is best for MVPs and unpredictable traffic

Click Create table button.

4. Add a Global Secondary Index (GSI) for Fast Lookups

This is optional, but it’s what makes your API feel “real” when you need queries like:

“Show all appointments on a date”

“Show all booked appointments”

“Show all appointments by service type”

Create a GSI for date

Open the Appointments table
Go to Indexes tab
Click Create index

Index details

Partition key: date (String)

Sort key (optional): time (String)

Index name: date-time-index

Leave other configurations as default
Click Create index

Now you can query: “All appointments on 2026-02-20, sorted by time.”

5. Give Lambda Permission to Use DynamoDB

Lambda won’t be able to read/write until IAM allows it.

Go to Lambda → our function (AppointmentAPI)
Open Configuration → Permissions
Click the Execution role (opens IAM)
Click Add permissions → Attach policies

Attach one of these:

Quick (broad, good for learning) but goes against the Least Privilege Principle:

AmazonDynamoDBFullAccess

Better (recommended for real projects):

Create a custom policy allowing only:

dynamodb:PutItem

dynamodb:GetItem

dynamodb:UpdateItem

dynamodb:DeleteItem

dynamodb:Query

dynamodb:Scan (optional)

However, for purpose of this demo, we will use the first option (AmazonDynamoDBFullAccess).

Lambda Permission (IAM Role)

Attach Policy (AmazonDynamoDBFullAccess)

Click on Add permission button.

Step 4: Booking an Appointment (Business Logic)

Now that DynamoDB is ready and API Gateway routes requests to Lambda, we are ready to build the real booking engine.

Here is what we will implement:

POST /appointments to creates a new booking
Validates required fields
Generates a unique appointmentId
Writes the appointment into DynamoDB
Returns a clean JSON response

1. Lambda: Read the Incoming Request

When API Gateway triggers Lambda (proxy mode), the body arrives as a string under event["body"].

What Lambda Receives
event["httpMethod"] → "POST"
event["path"] → "/appointments"
event["body"] → JSON string

2. Implement Appointment Booking Logic (Python)

Create appointment + store in DynamoDB

Since our DynamoDB table is named Appointments and has keys:
userId (PK), appointmentId (SK)

Go to the Lambda function and add the body of codes below:

import json
import os
import uuid
import boto3
from datetime import datetime

dynamodb = boto3.resource("dynamodb")
TABLE_NAME = os.environ.get("APPOINTMENTS_TABLE", "Appointments")
table = dynamodb.Table(TABLE_NAME)

def lambda_handler(event, context):
    method = event.get("httpMethod")
    path = event.get("path")

    # Route: POST /appointments
    if method == "POST" and path == "/appointments":
        return handle_create_appointment(event)

    return response(404, {"error": "Route not found"})

def handle_create_appointment(event):
    # In a real system, userId comes from Cognito JWT claims.
    # For now, we’ll hardcode a sample user.
    user_id = "user_123"

    # Parse JSON body
    try:
        body = json.loads(event.get("body") or "{}")
    except json.JSONDecodeError:
        return response(400, {"error": "Invalid JSON body"})

    service = body.get("service")
    date = body.get("date")
    time = body.get("time")

    # Validate required fields
    missing = [k for k in ["service", "date", "time"] if not body.get(k)]
    if missing:
        return response(400, {"error": f"Missing fields: {', '.join(missing)}"})

    # Basic input validation (light but helpful)
    if not is_valid_date(date):
        return response(400, {"error": "Invalid date format. Use YYYY-MM-DD"})
    if not is_valid_time(time):
        return response(400, {"error": "Invalid time format. Use HH:MM (24h)"})

    appointment_id = f"apt_{uuid.uuid4().hex[:10]}"
    item = {
        "userId": user_id,
        "appointmentId": appointment_id,
        "service": service,
        "date": date,
        "time": time,
        "status": "BOOKED",
        "createdAt": datetime.utcnow().isoformat() + "Z"
    }

    # Write to DynamoDB
    table.put_item(Item=item)

    # Return response
    return response(201, {
        "appointmentId": appointment_id,
        "status": "BOOKED"
    })

def response(status_code, body):
    return {
        "statusCode": status_code,
        "headers": {
            "Content-Type": "application/json"
        },
        "body": json.dumps(body)
    }

def is_valid_date(date_str):
    try:
        datetime.strptime(date_str, "%Y-%m-%d")
        return True
    except Exception:
        return False

def is_valid_time(time_str):
    try:
        datetime.strptime(time_str, "%H:%M")
        return True
    except Exception:
        return False

Click on Deploy

3. Configure Lambda Environment Variable

Instead of hardcoding table name.

Navigate to Lambda → our function (AppointmentAPI) → Configuration

On the left pane, click on Environment variables → Edit and Add environment variable:

Key: APPOINTMENTS_TABLE

Value: Appointments

Click Save

4. Create Stage in API Gateway

Go to API Gateway, click on our API (AppointmentRestAPI), on the left pane, under API: AppointmentRestAPI
click on Stages, click on Create stage. Enter the following details:
Stage name: prod
Deployment: select date and time from the draw-down.
Leave others as default and click the Create stage button.

Then click on deploy using the created stage (prod)

Lambda Triggers API Routes

Go to our Lambda function, click on API Gateway under the AppointmentAPI and click on configuration. To see the triggers (The four routes of POST, GET, PUT and DELETE)

5. Test the Endpoint

Using cURL
Replace the URL with your API Gateway Invoke URL:

(In Windows PowerShell)

Invoke-RestMethod `
  -Method POST `
  -Uri "https://8cjba7n9wb.execute-api.us-east-1.amazonaws.com/prod/appointments" `
  -ContentType "application/json" `
  -Body '{"service":"haircut","date":"2026-02-20","time":"14:00"}'

Response in PowerShell

Table item in DynamoDB table
Go to DynamoDB, click on our table and find the new item posted there by the Lambda function, triggered by the API.

At this point, we’ve built a real booking engine.

Step 5: Set Notifications & Reminders (SNS + SES)

Bookings are pointless if people forget them. Real systems reduce no-shows with instant confirmations and scheduled reminders.

So, let's add:

SNS for SMS alerts
SES for email confirmations

Triggers for:

Booking confirmation (immediate)
24-hour reminder (scheduled)
Cancellation alert (immediate)

SMS Notifications with Amazon SNS

1. Create (or choose) an SNS setup for SMS

Go to Amazon SNS in AWS Console
In the left menu, click Text messaging (SMS) (if shown)
Scroll down to SMS preferences (or click Edit SMS preferences if shown) as shown in the images below.

Set:

Default message type → Transactional

Monthly spending limit → set a small amount (e.g. 1 or 5 USD)

Click Save changes

In many regions you can send SMS directly without creating a topic (you publish to a phone number).

2. Test SMS directly from the SNS console (optional but recommended)

On the same screen, click Publish text message (top right)

Enter:

Phone number → your phone number (with country code, e.g. +1...)
Message → Test SMS from Appointment API
Click Publish message

If you receive the SMS, SNS is working correctly.

3. Allow your Lambda to send SMS

Go to AWS Lambda
Open your function (e.g. AppointmentAPI)
Go to Configuration → Permissions
Click the Execution role (opens IAM)

Attach this policy:

AmazonSNSFullAccess

Scroll down and click Add permissions button.

Lambda can now send SMS.

4. Send SMS from Lambda (this is all you need)
Add the body of code below to our Lambda code:

import boto3

sns = boto3.client("sns")

def send_sms(phone_number, message):
    sns.publish(
        PhoneNumber=phone_number,
        Message=message
    )

Response (Booking confirmation)

send_sms(
    "+15551234567",
    "Booking confirmed: Haircut on 2026-02-20 at 14:00"
)

5. When SMS is triggered

Booking confirmation

After successful POST /appointments

Cancellation alert

After DELETE /appointments or status update to canceled

24-hour reminder

Sent from a scheduled Lambda (step no includede here)

Epilogue: Completing the Journey to a Full SaaS Backend

What we’ve built throughout this blog is a solid foundation, a functional, real-world serverless backend capable of handling bookings, persistence, and user-facing workflows. But modern SaaS systems don’t stop at functionality alone. They mature through security, reliability, deployment discipline, and cost awareness.

To evolve this solution into a fully production-grade SaaS backend, the following layers naturally complement what we have already covered:

Securing the API with Authentication

Adding authentication ensures that every request is trusted and every action is tied to an authenticated identity. This is where user isolation, role-based access, and data protection come into play, which is critical for any real application handling customer data.

Monitoring & Reliability

Visibility is what turns a working system into a reliable one. By introducing monitoring, logs, and alerts, the system becomes observable, debuggable, and resilient under real traffic and failure scenarios.

Deployment Strategy

A proper deployment strategy, separating development, staging, and production environments, that allows changes to be released safely and confidently. This transforms the project from a single deployment into a system that can evolve continuously.

Cost Breakdown: Why Serverless Wins

Understanding the cost model completes the architectural picture. With serverless, you pay only for usage, scale automatically with demand, and eliminate idle infrastructure—making this approach especially powerful for startups and growing businesses.

The Final Result

When these layers are added on top of the system we have built, the result is a complete SaaS-ready backend:

Secure
Scalable
Production-ready
Business-focused

This architecture is not theoretical. It is directly applicable to real industries and real customers.

This system could power:

Clinics managing patient appointments
Salons handling bookings and reminders
Consultants scheduling client sessions
Repair services coordinating visits and follow-ups

Closing Thought

The true value of serverless architecture isn’t just removing servers, it is in enabling teams to focus on business logic, user experience, and growth instead of infrastructure. What we have built here is not an endpoint, but a platform, one that can be extended, secured, observed, and scaled into a real product.

That is the difference between a tutorial and a real SaaS system.

AWS Lightsail in Action: Launching and Scaling Your Web Server Efficiently.

Bernard Chika Uwaezuoke — Sun, 20 Apr 2025 17:03:23 +0000

Description

Deploy a Java web app to AWS Lightsail using Jenkins for CI/CD. Learn how to launch, automate, and scale your app with this hands-on tutorial.

Are you looking for a fast, affordable, and beginner-friendly way to deploy your Java app to the cloud?

In this guide, you'll learn how to:

Launch an AWS Lightsail instance
Use Jenkins to build and deploy a containerized Java web application
Scale your application smartly

Introduction

AWS Lightsail

AWS Lightsail is a lightweight Virtual Private Server (VPS) service by Amazon Web Services, perfect for developers and small teams looking to deploy applications without diving into complex AWS configurations.

Key Features of AWS Lightsail

Straightforward Pricing with Predictable Costs

One of Lightsail’s biggest advantages is its flat-rate pricing model. Unlike EC2’s pay-as-you-go model, Lightsail bundles compute resources, SSD storage, and data transfer into fixed monthly plans, starting as low as $5/month. This makes budgeting easier, especially for developers working on side projects or small teams managing costs without sacrificing performance.

Quick Instance Setup and Pre-configured Applications

Starting with AWS Lightsail is incredibly easy. Users can launch a virtual machine (instance) in just a few clicks. You can choose your preferred operating system (Linux or Windows) or select from a range of pre-configured application stacks such as WordPress, Node.js, LAMP, and more. This allows developers to skip lengthy server setup processes and get their applications online faster.

Built-in Networking and Management Tools

Lightsail includes several built-in tools to simplify server management. Users can easily assign static IPs, configure DNS settings, manage firewalls, and even integrate load balancers for high availability setups. These features are accessible via a user-friendly web interface, which removes the need for complex networking configurations typically associated with EC2 or other AWS infrastructure.

Snapshots, Backups, and High Availability

Data protection and recovery are made simple through Lightsail’s snapshot feature. You can create snapshots of your instances and storage volumes either manually or automatically. These snapshots can be used to restore data quickly or clone environments for testing or scaling. Lightsail also supports high availability by allowing load balancer integration and deploying across different availability zones.

Seamless Migration to EC2

When your project outgrows Lightsail’s capabilities, migrating to EC2 is straightforward. Lightsail offers a built-in export tool that converts your instance into an EC2-compatible AMI. This makes it easy to transition to a more flexible and powerful AWS environment while maintaining your application setup and configurations.

Ideal Use Cases

AWS Lightsail is best suited for lightweight workloads and projects that require quick deployment and minimal maintenance. It is ideal for:

Hosting blogs, portfolios, and e-commerce websites
Running small-scale web applications and APIs
Creating development or staging environments
Deploying MVPs (Minimum Viable Products)
Students or beginners learning server deployment

Prerequisites

Please ensure you have the following before we begin:

An AWS account
Jenkins installed (locally or remotely)
Apache Maven installed (locally or remotely)
A Java project (using Maven or Gradle)
Docker installed (locally or remotely)
Basic familiarity with terminal commands

Step 1: Create a Lightsail Instance

Visit AWS Lightsail

Click Create instance
Select your instance location. For this project, I chose Virginia us-east-1 region.

Choose your instance image; this is the operating system.
- Platform: Linux/Unix
- Blueprint: OS-only (Ubuntu) For this demo, I chose Ubuntu distribution of Linux operating system without any Apps added.

Pick an instance plan from (as low as $5/month). This is one of the major benefits of the AWS Lightsail it removes the unpredictability of cloud costs and allows small businesses to operate within a tight budget.

Give your instance a unique name; this will enable you to identify it, and then click on the Create Instance button to launch it.

The instance starting

Click on the Instance icon to see the details

Download the .pem private SSH key. You need this to connect to the instance from an SSH agent or terminal. For this project, we connected directly from the Lightsail management console.

Click on the Networking tab and scroll down to IPv4 Firewall under it click on Add rule. This will open a pane to configure inbound rule. Under Application, select Custom, for Protocol select TCP and Port or range enter 8080.

Step 2: Connect to Your Instance

Use the command below if you are connecting from a command line.
ssh -i ~/Downloads/lightsail.pem ubuntu@your-lightsail-ip
For this project, I will be connecting from the Lightsail console.

The Ubuntu Lightsail Instance Terminal

Once connected, run the following commands to update the server and install Java and Docker:

sudo apt update && sudo apt upgrade -y

sudo apt install -y default-jdk

java -version  

sudo apt install -y apt-transport-https ca-certificates curl software-properties-common gnupg lsb-release

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] \
  https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

sudo apt update

sudo apt install -y docker-ce docker-ce-cli containerd.io

docker --version
sudo systemctl status docker

sudo usermod -aG docker $USER
newgrp docker

Checking Java Version

Step 3: Deploy the Application

Clone the Project Repository: Clone the sample Java application repository from GitHub.

git clone https://github.com/Donhadley22/practice-java-app.git

This repository contains a basic Java project, Dockerfile and Jenkinsfile. These will be used to build a Docker image, test, and deploy the application to AWS Lightsail.

Here is the Jenkinsfile:

pipeline {
  agent any
     tools {
    maven 'maven-3.9.9'
  }

  environment {
    DOCKER_IMAGE          = 'donhadley/peabsmartapp'
    DOCKERFILE            = 'Dockerfile'
    DOCKER_REGISTRY       = 'index.docker.io/v1'
    DOCKER_CREDENTIALS_ID = 'dockerhub-creds'
    DOCKER_TAG            = "${env.BUILD_NUMBER}"
    LS_INSTANCE_ID       = 'i-05d37916155efbc80'
    LS_REGION            = 'us-east-1'
    SSH_CREDENTIALS_ID    = 'ec2-ssh-key'
    LS_USER              = 'ubuntu'
    LS_HOST              = '52.87.109.141'
    CONTAINER_NAME        = 'peabsmart-app'
    HOST_PORT             = 80
    CONTAINER_PORT        = 8080
  }

  stages {

    stage('Checkout') {
      steps {
        echo 'Checking out code...'
        checkout scm
      }
    }

    stage('Package') {
      steps {
        echo 'Building the Java project and packaging WAR...'
          sh 'mvn clean package'
  }
}

stage('Test') {
      steps {
        echo 'Running tests...'
         sh 'mvn test'
      }
    }
    stage('Build') {
      steps {
        echo 'Building Docker image...'
        script {
          docker.build("${DOCKER_IMAGE}:${DOCKER_TAG}", "-f ${DOCKERFILE} .")
        }
      }
    }

    stage('Push') {
      steps {
        echo 'Pushing Docker image to Docker Hub...'
        script {
          docker.withRegistry('https://index.docker.io/v1/', "${DOCKER_CREDENTIALS_ID}") {
          docker.image("${DOCKER_IMAGE}:${DOCKER_TAG}").push()
      }
    }
  }
}


    stage('Deploy') {
      steps {
        echo 'Deploying to AWS EC2...'
        script {
          sshagent([SSH_CREDENTIALS_ID]) {
            sh """
              ssh -o StrictHostKeyChecking=no ${EC2_USER}@${EC2_HOST} \\
              'docker pull ${DOCKER_IMAGE}:${DOCKER_TAG} && \
              docker stop ${CONTAINER_NAME} || true && \
              docker rm ${CONTAINER_NAME} || true && \
              docker run -d --name ${CONTAINER_NAME} -p ${HOST_PORT}:${CONTAINER_PORT} ${DOCKER_IMAGE}:${DOCKER_TAG}'
             """

          }
        }
      }
    }
  }

  post {
    success {
      echo "✅ Deployed successfully to EC2 🎉"
    }
    failure {
      echo "❌ Deployment failed. Please check the logs for details."
    }
  }
}
// This Jenkinsfile is a declarative pipeline that automates the process of building, testing, and deploying a Dockerized Java application to an AWS instance. It includes stages for checking out the code, building the Docker image, running tests, pushing the image to Docker Hub, and deploying it to the Lightsail instance. The pipeline uses environment variables for configuration and SSH credentials for secure access to the instance.

Make sure the Jenkins server has:

Access to your .pem key

Maven installed

Correct permissions on the Lightsail instance

Setting-up the Jenkins Pipeline job:
Login to Jenkins with your credentials, and on the Dashboard click on New item.

On your New item pane, enter the pipeline job name, select Multibranch Pipeline and click OK

On the pipeline Configuration pane under General, enter the Display Name and Description of your pipeline job.

Scroll down to the Branch Sources and select GitHub, enter your GitHub credentials (Username and access token), and select it. Also paste the GitHub HTTP URL under Repository HTTPS URL for this project.

Click on the Validate button to ensure the credential is okay and Jenkins can access the repository in GitHub.

In Build Configuration, under Mode, enter by Jenkinsfile and Script Path Jenkinsfile. In most cases Jenkins does these automatically.

Click Save and Jenkins will scan the Repository to find the Jenkinsfile.

Go to the Jenkins Dashboard, Click on the pipeline you just created and select Build now. This will trigger the job to run the various stages declared in the Jenkinsfile.

Step 4: Access Your Java App

Once deployed, visit:

http://<your-lightsail-ip>:8080
Your Java app should now be live!
Make use of the Public IP Address of your Lightsail instance.

Scale Smartly with AWS Lightsail

Scale with Confidence!
These are Lightsail features that grow with you and your business,
Add Snapshots for Backup and Disaster Recovery.
Snapshots in Amazon Lightsail allow you to create point-in-time backups of your instances, databases, and block storage disks. These snapshots can be used to restore your resources in case of data loss or to create new resources based on the snapshot.

How to create a snapshot

Navigate to the Lightsail console.

Select your** Instance**.

Go to the Snapshots tab.

Click Create snapshot, provide a name, and confirm.

You can also enable automatic snapshots to have Lightsail create daily backups during a specified time window.

Use a Load Balancer for High Availability

Lightsail Load Balancers distribute incoming traffic across multiple instances, enhancing the availability and fault tolerance of your application. They automatically perform health checks and route traffic only to healthy instances.

Steps to set up a load balancer

In the Lightsail console, go to the Networking tab.

Click Create Load Balancer.

Assign a name and select the region.

Attach your instances to the load balancer.

This setup ensures that your application remains accessible even if one or more instances fail. This is only applicable when you have at least two or more Lightsail instances running same application.

Attach Additional Storage Blocks

As your data grows, you might need more storage. Lightsail Block Storage allows you to attach additional SSD-backed storage disks to your instances.

To add and attach a storage disk

In the Lightsail console, select your instance.

Go to the Storage tab and click Create disk.

Specify the disk size and name.

After creation, attach the disk to your instance.

Connect to your instance via SSH to format and mount the new disk.

Use Lightsail Containers for Docker Apps

Lightsail Container Services enable you to deploy and manage containerized applications without managing the underlying infrastructure. You can push your Docker images directly or use images from public registries.

Deploying a containerized application

In the Lightsail console, navigate to Containers on the left pane.

Click Create container service.

Define the service name, power, and scale.

Add your container by specifying the image and deployment settings.

Deploy the service and access it via the provided endpoint.

Seamless Migration to EC2 or ECS

When your application outgrows Lightsail's capabilities, you can migrate to Amazon EC2 or Amazon ECS for more advanced features.

Migrating to EC2:

Create a snapshot of your Lightsail instance.

In the Snapshots tab, select the snapshot and click Export to Amazon EC2.

Follow the prompts to create an Amazon Machine Image (AMI) in EC2.

Launch a new EC2 instance using the exported AMI.

This process allows you to retain your application's configuration and data while transitioning to a more scalable environment.

Summary

In this guide, you learned how to deploy and manage a Java application using Amazon Lightsail, integrating essential DevOps practices for scalability and reliability.

You began by launching a Lightsail instance, selecting the appropriate operating system and instance plan tailored to your application's needs. This process provided you with a virtual private server equipped with a static IP, enabling consistent access and management.

Next, you configured a Jenkins pipeline job on your existing Jenkins server, integrating it with your GitHub repository containing the application code and Jenkinsfile. This setup enabled automated builds and deployments, ensuring that code changes are efficiently tested and deployed.

You then deployed your Java application as a Docker image to the server. By containerizing your application, you ensured consistency across development and production environments, simplified dependency management, and facilitated scalability.
Jenkins

To prepare for future growth and ensure high availability, you explored scaling strategies. This included adding snapshots for backup and disaster recovery, using load balancers to distribute traffic across multiple instances, attaching additional storage blocks to accommodate growing data needs, and leveraging Lightsail Containers for deploying Dockerized applications. These features empower you to scale your application seamlessly and maintain uptime during peak usage.

Finally, you learned about the seamless migration path from Lightsail to more advanced AWS services like EC2 or ECS. This transition allows you to leverage a broader set of tools and configurations as your application's requirements evolve.

By following this guide, you've established a robust foundation for deploying, managing, and scaling your Java application in the cloud, ensuring resilience and adaptability in a dynamic digital landscape.

Ready to Elevate Your DevOps Journey?

I hope this guide has provided you with valuable insights into deploying and managing your Java application using Amazon Lightsail. If you have any questions, need further clarification, or have suggestions for future topics, feel free to reach out.

Stay connected by following my blog for more tutorials and updates. Your feedback and engagement are always welcome!

Configure a project and repository structure to support secure pipelines in Azure DevOps

Bernard Chika Uwaezuoke — Mon, 31 Mar 2025 18:00:19 +0000

Let’s Hack the Cloud: Cost-Reduction Strategies Across the Cloud Space. (AWS Lambda as a case study)

Bernard Chika Uwaezuoke — Sun, 26 Jan 2025 14:30:47 +0000

Introduction

Cloud computing has revolutionized how businesses manage and scale their operations. However, with great flexibility comes the challenge of managing costs effectively. For companies navigating the cloud space, cost optimization is not just an operational necessity but a competitive advantage.
In this blog, we’ll discuss some proven strategies for reducing cloud costs and dive deep into the transformative role of Lambda Functions in cutting infrastructure expenses.

The Importance of Cloud Cost Optimization

The cloud offers scalable, pay-as-you-go pricing, but companies often face ballooning costs due to:
• Over-provisioned resources.
• Idle or underutilized infrastructure.
• Lack of visibility into resource consumption.

Cost optimization in the cloud is critical for:

Improving ROI on cloud investments.
Ensuring scalability without excessive expenses.
Allowing startups and enterprises alike to focus budgets on innovation rather than overhead.

Role of Serverless Computing in Cost Optimization

What is Serverless Computing?
Serverless computing allows developers to build and deploy applications without managing underlying infrastructure. In serverless, the cloud provider automatically provisions and scales resources as needed, and users are billed only for the execution time of their code.
One of the most popular serverless offerings is AWS Lambda, but other cloud providers offer equivalents:
• Azure Functions (Microsoft Azure)
• Google Cloud Functions (Google Cloud Platform)
• Cloud Run (Google Cloud for containerized workloads)

Lambda Functions: A Game-Changer for Cost Reduction

What Are Lambda Functions?
Lambda functions are small units of compute that execute code in response to triggers or events. They run only when needed, making them highly efficient for a wide range of use cases.

How Lambda Functions Reduce Cloud Costs

1. Pay-Per-Use Pricing Model:
Unlike traditional infrastructure where you pay for allocated resources regardless of usage, Lambda charges only for the actual time your code runs (measured in milliseconds). For example, running a virtual machine 24/7 costs significantly more than triggering a Lambda function 1,000 times per day for a few milliseconds each time.
2. Elimination of Idle Resources:
In traditional setups, servers are often idle during off-peak hours, wasting money. Lambda scales automatically with workload demands, ensuring you never pay for unused capacity.
3. Simplified Infrastructure:
Lambda reduces the need for complex infrastructure. By abstracting server management, it saves costs associated with maintaining and monitoring VMs or container clusters.
4. Integration with Other Cloud Services:
Lambda seamlessly integrates with cloud services like S3, DynamoDB, and API Gateway, creating cost-efficient workflows without additional orchestration tools.
5. Reduced Maintenance Costs:
No patching, updates, or hardware replacement. Lambda eliminates the operational overhead and associated costs of maintaining traditional servers.

Use Cases Where Lambda Excels in Cost Reduction

1. Data Processing:
Transforming or analyzing logs stored in S3 buckets.
Cost-Saving Aspect: Instead of provisioning an EC2 instance to continuously run processing jobs, Lambda triggers only when new data arrives.
2. Event-Driven Applications:
Auto-scaling customer notification services.
Cost-Saving Aspect: Scale dynamically based on the number of notifications, reducing costs during low activity periods.
3. Microservices:
Building microservices architecture with APIs.
Cost-Saving Aspect: Each service runs independently, scaling and billing separately, preventing over-provisioning.
4. Scheduled Tasks:
Running periodic jobs like backups or system checks.
Cost-Saving Aspect: Replace always-on cron jobs with Lambda functions triggered at specified intervals.
5. Application Prototypes:
Building and testing proof-of-concept applications.
Cost-Saving Aspect: Start small with minimal costs, scaling up only when the application reaches production.

Best Practices to Maximize Cost Savings with Lambda

1. Right-Sizing Memory Allocation:
Choose the appropriate memory size for your Lambda functions to avoid overpaying for unnecessary capacity.
2. Optimize Function Duration:
Streamline your code to reduce execution time, minimizing the cost per invocation.
3. Use Reserved Concurrency:
For predictable workloads, reserve concurrency limits to control costs and ensure availability.
4. Monitor Usage with AWS Cost Explorer:
Continuously track Lambda usage patterns and costs to identify optimization opportunities.
5. Leverage AWS Free Tier:
Take advantage of Lambda’s free tier, which provides 1 million requests and 400,000 GB-seconds of compute time per month.

Beyond Lambda: General Cloud Cost Optimization Tips

1. Auto-Scaling:
Enable auto-scaling for resources like EC2 instances and Kubernetes clusters to handle varying workloads efficiently.
2. Use Spot and Reserved Instances:
Leverage spot instances for non-critical workloads and reserved instances for predictable workloads.
3. Monitor and Optimize Storage:
Regularly audit and remove unused snapshots and use lifecycle policies for S3 storage.
4. Rightsize Resources:
Continuously evaluate and resize over-provisioned resources using tools like Azure Advisor or AWS Compute Optimizer.
5. Implement Budgets and Alerts:
Set cost budgets and configure alerts to prevent unexpected overspending.

Conclusion

Reducing cloud costs doesn’t mean sacrificing performance or scalability, it’s about leveraging the right tools and strategies. Lambda functions, with their pay-per-use model and automatic scaling, exemplify how serverless computing can dramatically cut infrastructure expenses.
By adopting a thoughtful approach to cloud resource management, companies can strike the perfect balance between cost and efficiency, ensuring sustainable growth in the cloud era.

Let’s hack the cloud together!

Configuring Agent Pools and Understanding Pipeline Styles in Azure DevOps

Bernard Chika Uwaezuoke — Mon, 05 Feb 2024 09:48:54 +0000

This blog is inspired by a Microsoft Lab with similar tittle

INTRODUCTION
Welcome to our latest exploration into the dynamic realm of Azure DevOps! In today's blog post, we delve into the crucial aspects of configuring agent pools and unravel the intricacies of pipeline styles. Navigating the ever-evolving landscape of DevOps requires a deep understanding of these fundamental components, and we're here to guide you through the process.

Whether you're a seasoned DevOps engineer or just starting your journey, mastering the configuration of agent pools and comprehending various pipeline styles is essential for streamlining your development workflows. Join us as we unravel the mysteries behind these integral concepts, providing insights, best practices, and actionable tips to empower your Azure DevOps journey. Let's embark on this knowledge-packed expedition together!

The post will cover the creation and management of agent pools, as well as an understanding of different pipeline styles. It will also discuss the significance of these concepts in the context of DevOps practices and their impact on the efficiency of software delivery.

Let start by defining the relevant concepts.

Azure DevOps
Azure DevOps is an end-to-end software development platform that supports Agile, DevOps, and other continuous development practices. It provides a range of tools and services for requirements and project management, version control, automated builds, testing, release management, and reporting. Azure DevOps aims to help teams create and improve products at a faster pace than traditional software development approaches, fostering a collaborative culture and integrating with a variety of other tools and services. In simple terms, it's a comprehensive solution that enables teams to efficiently manage the entire software development lifecycle, from planning and coding to testing and deployment, all in one place.

What is agent pool and its use in Azure DevOps?
An agent pool is a collection of one or more agents that can be used to run builds and deployments in Azure DevOps. Agents are the machines that execute the tasks defined in pipelines. Agent pools are used to organize agents based on their capabilities, such as operating system, software, or hardware configurations. This allows teams to manage and allocate resources more efficiently and to ensure that the right agents are used for the right jobs. Agent pools can be created and managed from the Azure DevOps organization or project settings. They can be used in both build and release pipelines and can be either self-hosted or provided by Microsoft as hosted agents.

Azure DevOps Pipelines
Azure DevOps Pipeline is a service that allows you to automatically build, test, and deploy your code to various platforms and cloud environments. It supports continuous integration (CI) and continuous delivery (CD) practices, enabling you to automate the entire software delivery process. Azure Pipelines can be used with any language or platform, and it provides cloud-hosted agents for building and deploying applications. It offers features such as parallel jobs, native container support, and flexible deployments to Kubernetes, serverless, or virtual machines.

Pipelines Configurations Supported by Azure DevOps
Azure DevOps supports YAML-based pipelines and classic UI-based pipelines configurations. The classic UI allows users to define and manage pipelines using a graphical interface, which can be a preferred option for those who are not familiar with or prefer not to use YAML syntax. However, Microsoft has been encouraging the use of YAML pipelines due to their portability, version control integration, and ease of sharing.

In this exercise, we will be using the YAML-based pipeline editor, which is a tool provided by Azure DevOps to author and edit pipelines using YAML syntax.

Prerequisites:

Azure Account with subscription
Microsoft Edge or an Azure DevOps supported browser.
Set up an Azure DevOps organization
Git for Windows
Visual Studio Code.

Objectives
By the time you conclude this learning journey with us, you will be able to do the following in Azure DevOps:

Implement YAML-based pipelines.
Implement self-hosted agents.

Here we go!

Step 1: Setting up an Azure DevOps Organization

Login to Azure portal at www.portal.azure.com.
On the global search bar, type in Azure DevOps select Azure DevOps Organizations.

One the Azure DevOps Home page, click on My Azure DevOps Organizations.

On the Azure DevOps Organizations page, click on Create new organization.

This will take you to a new Signup page, click on the Continue button if you do not wish to switch directory.

On the next page, enter a globally unique organization name, the location you intend to host your projects and the authentication character displayed.

We can see the new organization we created and can now start creating our projects.

Step 2: Create a New Project

Here we will be creating a project named eShopOnWeb.
Enter the project name and click on Create project

Our project landing page!

Step 3: Import our project files from a Git repository to Azure DevOps

On the project landing page, on the left pane under your project name, click on Repos>Files, Import a Repository. Select Import.

On the Import a Git Repository window, paste the URL of the Git repository where your application is located. For this exercise, we will be using a project provided by Microsoft. Here is the URL https://github.com/MicrosoftLearning/eShopOnWeb.git and click Import

The project in Azure DevOps Repos Environment

CONFIGURE A YAML-BASED PIPELINE

Step 4: Create an Azure DevOps YAML pipeline

Create a template-based Azure DevOps YAML pipeline.
On the EShopOnWeb project page, in the vertical navigational pane on the left side, click Pipelines.
Click the Create Pipeline button - if you don’t have any other pipelines created yet or click New pipeline to create an additional new one.

On the Where is your code? pane, click Azure Repos Git. Since we have imported the code there. _You can select other locations based on where you have your code. _

On the Select a repository pane, click EShopOnWeb. Which is our project.

On the Configure your pipeline pane, click Existing Azure Pipelines YAML File.

On the Select an existing YAML file, select main for the Branch, and /.ado/eshoponweb-ci-pr.yml for the Path and click Continue

This sample pipeline is a basic .NET application Build pipeline, which does the following:
A single Stage: Build
A single Job: Build
3 tasks within the Build Job:
Dotnet Restore
Dotnet Build
Dotnet Publish



resources:
  repositories:
    - repository: self
      trigger: none

stages:
- stage: Build
  displayName: Build .Net Core Solution
  jobs:
  - job: Build
    pool:
      vmImage: ubuntu-latest
    steps:
    - task: DotNetCoreCLI@2
      displayName: Restore
      inputs:
        command: 'restore'
        projects: '**/*.sln'
        feedsToUse: 'select'

    - task: DotNetCoreCLI@2
      displayName: Build
      inputs:
        command: 'build'
        projects: '**/*.sln'

    - task: DotNetCoreCLI@2
      displayName: Test
      inputs:
        command: 'test'
        projects: 'tests/UnitTests/*.csproj'

    - task: DotNetCoreCLI@2
      displayName: Publish
      inputs:
        command: 'publish'
        publishWebProjects: true
        arguments: '-o $(Build.ArtifactStagingDirectory)'

On the Review your pipeline YAML pane, click the down-facing caret symbol next to the Run button, click Save. This action creates the pipeline.

The created pipeline

MANANGE AZURE DEVOPS AGENT POOLS

Step 5: Configure an Azure DevOps self-hosting agent

In the Azure DevOps portal, in the upper right corner of the Azure DevOps page, click the User settings icon, depending on whether or not you have preview features turned on, you should either see a Security or Personal access tokens item in the menu, if you see Security, click on that, then select Personal access tokens. On the Personal Access Tokens pane, and click + New Token.

On the Create a new personal access token pane, click the Show all scopes link and, specify the following settings and click Create (leave all others with their default values):

Name = EShopOnWeb
Scope (custom defined) = **Agent Pools** (show more scopes option below if needed)
Permissions = **Read and manage**

On the Success pane, copy the value of the personal access token to Clipboard and click the Close button.
On the Personal Access Token pane of the Azure DevOps portal, click Azure DevOps symbol in the upper left corner and then click Organization settings label in the lower left corner.

To the left side of the Overview pane, in the vertical menu, in the Pipelines section, click Agent pools.

On the Agent pools pane, in the upper right corner, click Add pool.

On the Add agent pool pane, in the Pool type dropdown list, select **Self-hosted, in the Name text box, type **proj-2420-pool and then click Create.

Back on the Agent pools pane, click the entry representing the newly created proj-2420-pool.
On the Jobs tab of the proj-2420-pool pane, click the New agent button.

On the Get the agent pane, ensure that the Windows and x64 tabs are selected, and click Download to download the zip archive containing the agent binaries to download it into the local **Downloads*8 folder within your user profile.

To Register the Agent

Start Windows PowerShell as administrator and in the Administrator: Windows PowerShell console run the following lines to create the C:\agent directory and extract the content of the downloaded archive into it. By running these lines of code:



cd \
mkdir agent ; cd agent
$TARGET = Get-ChildItem "$Home\Downloads\vsts-agent-win-x64-*.zip"
Add-Type -AssemblyName System.IO.Compression.FileSystem

In the same Administrator: Windows PowerShell console, run the following to configure the agent:

.\config.cmd

-Follow the prompt, specify the values of the following settings:

Enter server URL = https://dev.azure.com/Project-24201

That's the URL of your Azure DevOps organization, in the format https://dev.azure.com/**organization_name**, where organization_name represents the name of your Azure DevOps organization.

Enter authentication type (press enter for PAT) = **Enter**

Enter personal access token = **The access token you recorded earlier in this task**

Enter agent pool (press enter for default) = **proj-2420-pool**

Enter agent name = **az400m03-vm0**

Enter work folder (press enter for _work) = **Enter**

(Only if shown) Enter Perform an unzip for tasks for each step. (press enter for N) WARNING: only press **Enter** if the message is shown

If the above prompt is not displayed, then continue with your configuration.

Enter run agent as service? (Y/N) (press enter for N) = **Y**

enter enable SERVICE_SID_TYPE_UNRESTRICTED (Y/N) (press enter for N) = **Y**

Enter User account to use for the service (press enter for NT AUTHORITY\NETWORK SERVICE) = **Enter**

Enter whether to prevent service starting immediately after configuration is finished? (Y/N) (press enter for N) = **Enter**

At the end of registering the agent, you will get successful notification as shown below.

Back to the Azure DevOps portal

In the web browser window displaying the Azure DevOps portal, in the upper left corner, click the Azure DevOps label.

From the list of projects, click the tile representing your EShopOnWeb project.

On the EShopOnWeb pane, in the vertical navigational pane on the left side, in the Pipelines section, click Pipelines.

On the Recent tab of the Pipelines pane, select EShopOnWeb and, on the EShopOnWeb pane, select Edit.

Here we provide the detail of the new agent we registered to the YAML-based CI pipeline.
On the EShopOnWeb edit pane, in the existing YAML-based pipeline, replace line 13 which says vmImage: ubuntu-latest designating the target agent pool with the newly created self-hosted agent pool.

On the EShopOnWeb edit pane, in the upper right corner of the pane, click Save and Run. This will automatically trigger the build based on this pipeline.

You can check the option to Enable system diagnostic

In the Azure DevOps portal, in the vertical navigational pane on the left side, in the Pipelines section, click Pipelines. Depending on your system setup, the pipeline might prompt you for permissions. Click Permit to allow the pipeline to run.

On the Recent tab of the Pipelines pane, click the EShopOnWeb entry, on the Runs tab of the EShopOnWeb pane, select the most recent run, on the Summary pane of the run, scroll down to the bottom, in the Jobs section, click Phase 1 and monitor the job until its successful completion.

When the process concludes, you will be notified of the status of your pipeline as successfully built.

Thank you for your time, kindly subscribe and follow our page for more insightful articles.

S3crets Unveiled: A Painless Guide to Hosting Your First Static Website on AWS.

Bernard Chika Uwaezuoke — Fri, 02 Feb 2024 21:22:36 +0000

INTRODUCTION

Welcome to the enchanting realm of Cloud Hosting! If the mere thought of launching a website feels like deciphering an alien language, fear not. In this step-by-step journey, we'll unravel the mystery of hosting a static website on AWS S3, transforming the seemingly complex into a seamless symphony of digital creation. Picture it as your backstage pass to the CloudCraft Chronicles, where we'll guide you with clarity, unraveling the S3crets to effortlessly elevate your web presence. Embrace the magic of simplicity as we embark on this adventure together, turning novices into cloud virtuosos!

PREREQUISITES

AWS Account: Ensure you have an active AWS account. If you don't have one yet, fear not! You can do that for free on
www.aws.amazon.com
Basic Web Files: Have your static website files ready. Whether it's a portfolio, blog, or a simple landing page, make sure your HTML, CSS, and other assets are set to go.
Curiosity and Eagerness: The most important prerequisites! Bring along your curiosity and eagerness to learn. We're here to make this journey enjoyable and enlightening, even for the absolute beginner.

Now that our toolkit is prepped, let's set the stage for our CloudCraft Chronicles and unlock the door to AWS S3 hosting magic!

STEP 1: login to your AWS account

Go to your web browser and enter this URL: https://aws.amazon.com/console/
Click on sign in to the console button.

Select to sign in as a Root user if you own the account, otherwise, if given access by an organization or a third party, the select IAM user
Provide your accout email and click the Next button.

Provide your password and click on sign in button.

STEP 2: Creating an S3 Bucket

On the global search bar at the top of your home page, type s3 and select s3 from the auto-dropdown.

On the S3 page, click on Create bucket button.

On the box AWS Region box, dropdown and select the region you want to host your static website.
Leave it in General purpose
Then enter a globally unique bucket name in the box Bucket name.

Scroll down on same page.
Leave the default ACL disabled option.
Uncheck the Block all public access option.

A warning massage will be disblayed intimating you of the implication of making the bucket public, acknowledge it, leave other configurations as default and continue to the base of the page to click on the Create bucket button.

The created project-2420 S3 bucket and message confirming its successful creation.

STEP 3: Uploading our Static web files to the S3 Bucket

Click on the created bucket project-2420
On the bucket page, click on Upload button. This will open a pane that you can use to navigate to your resources in your local environment.

Select add files or add folder depending on how your web resources are saved. For this demonstration, we will be using add folder.

We will be uploading our resources from our local desktop.

The uploaded files

The index.html and the error.html or in this instance 404.html. They are the primary pages that display on the website.

Scroll to the bottom of the page and click the Upload button.

Files uploaded successfully.

STEP 4: Configure the Static Website

We go back to the project-2420 S3 bucket page by clicking on Close button and click on the permission

Scroll down on the opened page and click on Edit button by Bucket policy.

Input the policy below in the editor with your bucket name replacing the place holder. (We got the policy from AWS documentation on S3 static website hosting)

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "PublicReadGetObject",
            "Effect": "Allow",
            "Principal": "*",
            "Action": [
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::Bucket-Name/*"
            ]
        }
    ]
}

Click on Save changes button at the base of the page.
You will get a notification that the policy change was successful.
The click on property on the same project-2420 bucket page.

Scroll down to the base of the page and on Static website hosting, click Edit

Select Enable option for Static website

This will extend the pane, exposing boxes for you to input your index.html and error.html configurations.

Leave other configurations as default and click Save Changes button at the base of the page.
You will get this notification of success.

At the base of the page, we can see our static website bucket endpoint which we can use to access it over a browser.

Copy the endpoint and use it to view the static website.

The webpage displayed

Cleanup to avoid careless Charges

Click on the Action button on the project-2420 bucket page.
Select empty bucket, enter permanently delete in the provided box and click the Empty button.
You will get a notification indicating that the content of the bucket has been deleted.
Then click on the delete button on the project-2420 bucket page.
You will be prompted to enter the bucket name in the provided box. This action will activate the Delete bucket button. Click on the Delete bucket button and the bucket will be deleted.

CONCLUSION
As our CloudCraft Chronicles come to a close, take a moment to marvel at the digital masterpiece you've crafted in the AWS S3 cloudscape. From a novice navigating the unknown, you've emerged as a virtual virtuoso, orchestrating your static website with finesse.

Remember, the magic of hosting extends beyond technology; it's about empowering your ideas to soar in the vast expanse of the internet. Whether you're a hobbyist, entrepreneur, or dream-weaver, your corner of the web is now a reality.

As you embark on your digital journey, may the CloudCraft Chronicles be a guiding star, illuminating the path to endless possibilities. The curtain may fall on this blog, but your story in the cloud has just begun.

Until our next adventure, happy hosting, and may your digital dreams continue to unfold!

Please subscribe and follow our blog

Building a Custom VPC Infrastructure in AWS with Terraform.

Bernard Chika Uwaezuoke — Tue, 16 Jan 2024 04:19:55 +0000

INTRODUCTION
Amazon Web Services (AWS) provides a robust set of tools to manage cloud infrastructure, and Terraform enhances this capability by enabling Infrastructure as Code (IaC). In this article, we will explore how to use Terraform to create a custom VPC (Virtual Private Cloud) in AWS, complete with an internet gateway, a route table, public and private subnets, security groups, network interfaces, and EC2 instances.

PREREQUISITES
Before embarking on the journey of creating a custom VPC infrastructure in AWS using Terraform, ensure you have the following in place:

1. AWS Account:

Create an AWS account if you don't have one already. Navigate to the AWS Console at www.aws.amazon.com to set up your account.

2. AWS CLI and Credentials:

Install the AWS Command Line Interface (CLI) on your local machine. You can download it here.
Configure your AWS credentials using the aws configure command.
Ensure your credentials have the necessary permissions to create and manage resources.

3.Terraform Installation:

Download and install the latest version of Terraform on your local machine.

4. Text Editor or Integrated Development Environment (IDE):

Choose a text editor or IDE of your preference for editing Terraform configuration files. Popular choices include VSCode, Sublime Text, or Atom.
(For this demonstration, we will be using VSCode.)

5. Basic Understanding of AWS Services:

Familiarize yourself with basic AWS concepts, including VPC, subnetting, internet gateways, security groups, and EC2 instances. This foundational knowledge will help you design an infrastructure that meets your specific requirements.

6. SSH Key Pair:

Generate an SSH key pair if you plan to access your EC2 instances. This key pair will be used when creating EC2 instances in your Terraform configuration.

7. Knowledge of Terraform Basics:

Understand the fundamental concepts of Terraform, such as providers, resources, variables, and modules. Refer to the official Terraform documentation at https://developer.hashicorp.com/terraform/docs for guidance.

8. Customization:

Modify the provided Terraform configuration according to your specific needs. Update variables like cidr_block, availability_zone, key_name, and ami with values suitable for your project.

9. Security Considerations:

Be mindful of security best practices. Restrict inbound and outbound traffic in your security groups to only the necessary ports and IP ranges.

Let dive right in!

STEP 1: CREATE YOUR WORKING DIRECTORY AND CONFIGURATION FILES

Go to your Git Bash interface and type this command cd Desktop to navigate to the location you want to create the folder. In this very instance, we are creating it at the Desktop.
Create a Directory (folder). Type the following command mkdir <folder name>to create the directory.

Navigate to the directory you just created using the cd command.
While in the directory, use the command code . to open the VScode IDE.

Create'main.tf' and provider.tf files.

STEP 2: CONFIGURE AWS ACCESS AND PROVIDER

Run the following commands: aws --version to confirm your version of aws-cli and aws configure to provide your AWS credentials in your VScode terminal.
The credentials:

AWS Access Key ID [****************ZYAS]:
AWS Secret Access Key [****************6H9/]: 
Default region name [us-east-1]: 
Default output format [json]:

Go to the official Terraform site https://registry.terraform.io/providers/hashicorp/aws/latest/docs to get your Terraform codes to customize for your deployment.
In the provider.tf file, paste the AWS Provider configuration code, then save.

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

# Configure the AWS Provider
provider "aws" {
  region = "us-east-1"
}

Then run the terraform init command to initialize Terraform. This command will create a folder containing terraform plugins and a .terraform.lock.hcl file, indicating that your environment is ready to apply your deployments.

STEP 3 DEFINE THE VPC, INTERNET GATEWAY, AND ROUTE TABLE

Go to the main.tf file and start declaring the resources you want provisioned using the HashiCorp Configuration Language (HCL).

# Create a VPC
resource "aws_vpc" "main-vpc" {
  cidr_block       = "10.0.0.0/16"
  instance_tenancy = "default"

  tags = {
    Name = "main-vpc"
  }
}

# Create an Internet Gateway
resource "aws_internet_gateway" "classgw" {
  vpc_id = aws_vpc.main-vpc.id

  tags = {
    Name = "classgw"
  }
}

# Create a Route Table
resource "aws_route_table" "classRT" {
  vpc_id = aws_vpc.main-vpc.id

  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.classgw.id
  }
}

Run the commands terraform init, terraform validate, terraform plan and terraform apply in sequence to provision the resources. You can choose to do this in batches, as we are doing here, or run the commands after declaring all your resources.

Output of terraform plan command

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # aws_internet_gateway.classgw will be created        
  + resource "aws_internet_gateway" "classgw" {
      + arn      = (known after apply)
      + id       = (known after apply)
      + owner_id = (known after apply)
      + tags     = {
          + "Name" = "classgw"
        }
      + tags_all = {
          + "Name" = "classgw"
        }
      + vpc_id   = (known after apply)
    }

  # aws_route_table.classRT will be created
  + resource "aws_route_table" "classRT" {
      + arn              = (known after apply)
      + id               = (known after apply)
      + owner_id         = (known after apply)
      + propagating_vgws = (known after apply)
      + route            = [
          + {
              + carrier_gateway_id         = ""
              + cidr_block                 = "0.0.0.0/0"
              + core_network_arn           = ""
              + destination_prefix_list_id = ""
              + egress_only_gateway_id     = ""
              + gateway_id                 = (known after apply)
              + ipv6_cidr_block            = ""
              + local_gateway_id           = ""
              + nat_gateway_id             = ""
              + network_interface_id       = ""
              + transit_gateway_id         = ""
              + vpc_endpoint_id            = ""
              + vpc_peering_connection_id  = ""
            },
        ]
      + tags_all         = (known after apply)
      + vpc_id           = (known after apply)
    }

  # aws_vpc.main-vpc will be created
  + resource "aws_vpc" "main-vpc" {
      + arn                                  = (known after apply)
      + cidr_block                           = "10.0.0.0/16"
      + default_network_acl_id               = (known after apply)
      + default_route_table_id               = (known after apply)
      + default_security_group_id            = (known after apply)
      + dhcp_options_id                      = (known after apply)
      + enable_dns_hostnames                 = (known after apply)
      + enable_dns_support                   = true
      + enable_network_address_usage_metrics = (known after apply)
      + id                                   = (known after apply)
      + instance_tenancy                     = "default"
      + ipv6_association_id                  = (known after apply)
      + ipv6_cidr_block                      = (known after apply)
      + ipv6_cidr_block_network_border_group = (known after apply)
      + main_route_table_id                  = (known after apply)
      + owner_id                             = (known after apply)
      + tags                                 = {
          + "Name" = "main-vpc"
        }
      + tags_all                             = {
          + "Name" = "main-vpc"
        }
    }

Plan: 3 to add, 0 to change, 0 to destroy.

──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── 

Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run "terraform apply" now.

Output of terraform plan command

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # aws_internet_gateway.classgw will be created
  + resource "aws_internet_gateway" "classgw" {
      + arn      = (known after apply)
      + id       = (known after apply)
      + owner_id = (known after apply)
      + tags     = {
          + "Name" = "classgw"
        }
      + tags_all = {
          + "Name" = "classgw"
        }
      + vpc_id   = (known after apply)
    }

  # aws_route_table.classRT will be created
  + resource "aws_route_table" "classRT" {
      + arn              = (known after apply)
      + id               = (known after apply)
      + owner_id         = (known after apply)
      + propagating_vgws = (known after apply)
      + route            = [
          + {
              + carrier_gateway_id         = ""
              + cidr_block                 = "0.0.0.0/0"
              + core_network_arn           = ""
              + destination_prefix_list_id = ""
              + egress_only_gateway_id     = ""
              + gateway_id                 = (known after apply)
              + ipv6_cidr_block            = ""
              + local_gateway_id           = ""
              + nat_gateway_id             = ""
              + network_interface_id       = ""
              + transit_gateway_id         = ""
              + vpc_endpoint_id            = ""
              + vpc_peering_connection_id  = ""
            },
        ]
      + tags_all         = (known after apply)
      + vpc_id           = (known after apply)
    }

  # aws_vpc.main-vpc will be created
  + resource "aws_vpc" "main-vpc" {
      + arn                                  = (known after apply)
      + cidr_block                           = "10.0.0.0/16"
      + default_network_acl_id               = (known after apply)
      + default_route_table_id               = (known after apply)
      + default_security_group_id            = (known after apply)
      + dhcp_options_id                      = (known after apply)
      + enable_dns_hostnames                 = (known after apply)
      + enable_dns_support                   = true
      + enable_network_address_usage_metrics = (known after apply)
      + id                                   = (known after apply)
      + instance_tenancy                     = "default"
      + ipv6_association_id                  = (known after apply)
      + ipv6_cidr_block                      = (known after apply)
      + ipv6_cidr_block_network_border_group = (known after apply)
      + main_route_table_id                  = (known after apply)
      + owner_id                             = (known after apply)
      + tags                                 = {
          + "Name" = "main-vpc"
        }
      + tags_all                             = {
          + "Name" = "main-vpc"
        }
    }

Plan: 3 to add, 0 to change, 0 to destroy.
aws_vpc.main-vpc: Creating...
aws_vpc.main-vpc: Creation complete after 5s [id=vpc-07733c710c0a92868]
aws_internet_gateway.classgw: Creating...
aws_internet_gateway.classgw: Creation complete after 2s [id=igw-08480f88ccaf9b886]
aws_route_table.classRT: Creating...
aws_route_table.classRT: Creation complete after 2s [id=rtb-0bed4113ba3664e06]

Apply complete! Resources: 3 added, 0 changed, 0 destroyed.

We will repeat the above steps until we finish provisioning our resources.
The provisioned resources in AWS management console

STEP 4: CREATE PUBLIC AND PRIVATE SUBNETS

On the main.tf file, go ahead and declare the commands for the subnets to provision both public and private subnets

# Create public subnets
resource "aws_subnet" "main_pubs1" {
  vpc_id            = aws_vpc.main-vpc.id
  cidr_block        = "10.0.1.0/24"
  availability_zone = "us-east-1a"

  tags = {
    Name = "main_pubs1"
  }
}

resource "aws_subnet" "main_pubs2" {
  vpc_id            = aws_vpc.main-vpc.id
  cidr_block        = "10.0.2.0/24"
  availability_zone = "us-east-1b"

  tags = {
    Name = "main_pubs2"
  }
}

# Create private subnets
resource "aws_subnet" "main_privs1" {
  vpc_id            = aws_vpc.main-vpc.id
  cidr_block        = "10.0.3.0/24"
  availability_zone = "us-east-1c"

  tags = {
    Name = "main_privs1"
  }
}

resource "aws_subnet" "main_privs2" {
  vpc_id            = aws_vpc.main-vpc.id
  cidr_block        = "10.0.4.0/24"
  availability_zone = "us-east-1d"

  tags = {
    Name = "main_privs2"
  }
}

In this demonstration, we have two public and two private subnets.

STEP 5: ASSOCIATE SUBNETS WITH THE ROUTE TABLE

# Associate public subnets with the route table
resource "aws_route_table_association" "a" {
  subnet_id      = aws_subnet.main_pubs1.id
  route_table_id = aws_route_table.classRT.id
}

resource "aws_route_table_association" "b" {
  subnet_id      = aws_subnet.main_pubs2.id
  route_table_id = aws_route_table.classRT.id
}

STEP 6: CREATE A SECURITY GROUP

# Create a security group
resource "aws_security_group" "class_SG" {
  name        = "class_SG"
  description = "Allow SSH, HTTP, HTTPS inbound traffic"
  vpc_id      = aws_vpc.main-vpc.id

  ingress {
    description = "SSH from VPC"
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  ingress {
    description = "HTTPS from VPC"
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  ingress {
    description = "HTTP from VPC"
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
     from_port   = 0
     to_port     = 0
     protocol    = "-1"
     cidr_blocks = ["0.0.0.0/0"]
   }

  tags = {
    Name = "allow_traffic"
  }
}

STEP 7: CREATE NETWORK INTERFACES

Create network interfaces for the two instances.

# Create network interfaces
resource "aws_network_interface" "class_NI" {
  subnet_id       = aws_subnet.main_pubs1.id
  private_ips     = ["10.0.1.50"]
  security_groups = [aws_security_group.class_SG.id]
}

resource "aws_network_interface" "main-NIC" {
  subnet_id       = aws_subnet.main_pubs2.id
  private_ips     = ["10.0.2.20"]
  security_groups = [aws_security_group.class_SG.id]
}

STEP 8: ALLOCATE ELASTIC IPS AND ASSOCIATE WITH NETWORK INTERFACES

# Create Elastic IPs
resource "aws_eip" "class_EIP" {
  vpc                       = true
  network_interface         = aws_network_interface.class_NI.id
  associate_with_private_ip = "10.0.1.50"
}

resource "aws_eip" "class_EIP1" {
  vpc                       = true
  network_interface         = aws_network_interface.main-NIC.id
  associate_with_private_ip = "10.0.2.20"
}

STEP 9: LAUNCH TWO EC2 INSTANCES

One in a Public Subnet and the other in a Private Subnet.

# Create EC2 instances
resource "aws_instance" "class_instance" {
  ami           = "ami-0005e0cfe09cc9050"
  instance_type = "t2.micro"
  key_name      = "Don-KP"
  network_interface {
    network_interface_id = aws_network_interface.class_NI.id
    device_index         = 0
  }

  tags = {
    Name = "hello1"
  }
}

resource "aws_instance" "class_instance222" {
  ami           = "ami-0005e0cfe09cc9050"
  instance_type = "t2.micro"
  key_name      = "Don-KP"
  network_interface {
    network_interface_id = aws_network_interface.main-NIC.id
    device_index         = 0
  }

  tags = {
    Name = "hello2"
  }
}

Resources Created

STEP 10: DESTROY

Use the command terraform destroy to clean-up the resources provisioned to have a clean slate and avoid paying for resources not used for production.

CONCLUSION
In this tutorial, we've walked through the process of creating a custom VPC infrastructure in AWS using Terraform. This modular approach allows for easy maintenance, scalability, and reproducibility of your cloud environment. By leveraging Terraform, you gain the benefits of Infrastructure as Code, making it simpler to manage and collaborate on complex AWS architectures.

Thank you for your time!

Connecting the Dots: A Step-by-Step Guide to Azure VNet Peering with Azure CLI.

Bernard Chika Uwaezuoke — Thu, 19 Oct 2023 23:23:29 +0000

INTRODUCTION

In today's interconnected digital landscape, the ability to create secure and seamless connections between different components of your cloud infrastructure is paramount. Azure Virtual Network (VNet) peering, one of the many powerful features offered by Microsoft Azure, enables you to achieve just that. With VNet peering, you can establish network communication between Virtual Networks in Azure, creating a unified and efficient network environment for your applications and services.

But, as with any sophisticated technology, getting started with Azure VNet peering may seem daunting at first. That's where we come in! In this comprehensive guide, we'll walk you through the step-by-step process of implementing Azure VNet peering using Azure CLI, the command-line interface that offers precise control and automation capabilities for Azure services.

Whether you're an Azure newcomer looking to understand the fundamentals or an experienced cloud architect seeking a refresher, this blog will provide you with the insights and practical instructions you need to create, configure, and manage Azure VNet peering like a pro.

Join us as we demystify the world of Virtual Network peering, exploring the key concepts, prerequisites, and best practices. By the end of this guide, you'll be equipped with the knowledge and tools to effortlessly establish network connections across your Azure VNets, enabling your applications and services to communicate seamlessly.

So, without further ado, let's dive into the exciting world of Azure VNet peering with Azure CLI and take the first step toward creating an interconnected, efficient, and resilient cloud infrastructure.

PREREQUISITES

Azure Subscription
Azure CLI
Planning and Design

STEP 1: Sign into Azure Account
Login to your Azure account through your Command Line Interface (CLI) by typing this command az login as shown in the image below then tap the enter key to run it.

This opens a tab in your browser, where you will be required to provide your login details.

Select the particular account you wish to deploy the resources and sign in.
A notification indicating you have been granted access to the account displays.

Your account detail is displayed on your CLI.

STEP 2: Create a Resource Group

A resource group is a logical grouping of Azure resources that are related to a specific application, project, or environment. It serves as a logical container for our resources in Azure and helps us in organization, management, access control, deletion and cleanup of our account.

We create the resource group using this command:
az group create --name <resource group name> --location <azure region>

From the above image, it can be seen that for this exercise, we created a resource group called Proj-RG. Which is visible on the portal (see below).

STEP 3: Create the Virtual Networks (Vnet).

For this exercise, we will be creating two virtual networks (EastVNet and WestVNet).

We proceed to create the virtual networks that we intend to peer. The convention is to have distinct IP address range for each Vnet to ensure that they do not overlap. Virtual networks that are to be peered should not have overlapping IP addresses.

We use this command to create the Vnets one after the other.
az network vnet create --resource-group <resource group name> --name <Vnet name> --address-prefix 10.1.0.0/16 --subnet-name <subnet name> --subnet-prefix 10.1.0.0/24
For EastVNet

From the above image, the IP address range for EastVnet can be seen as "10.1.0.0/16" and the subnet IP address range is "10.1.0.0/16".

For WestVnet
We can use this command az network vnet list --output table to verify if indeed the virtual networks were created.

We can add an additional subnet to an existing virtual network through this command.

az network vnet create --resource-group Proj-RG --name EastVNet --address-prefix 10.2.0.0/16 --subnet-name EastSubnet1 --subnet-prefix 10.2.0.0/24
This will create an aditional subnet in EastVNet.

We use this command to verify the subnets in a particular Vnet. az network vnet subnet list --resource-group Proj-RG --vnet-name EastVNet --output table

We can also proceed to the Azure portal to view the virtual networks we created inside the resource group (Proj-RG).

STEP 4: Peering the Virtual Networks

The final step is to peer the virtual networks.
We use the following commands to achieve that.

We begin by creating West to East peering. That is peering WestVNet to EastVNet.

We use this command to get the Resource ID of the EastVNet in order to perform the peering.

az network vnet show --resource-group Proj-RG --name EastVNet --query id --out tsv

We now run this command for the peering using the EastVNet resource ID.



az network vnet peering create --name WesttoEastPeering --resource-group Proj-RG --vnet-name WestVNet --remote-vnet "/subscriptions/21b8f625-edc8-4cf1-99ec-861cfa81ecc3/resourceGroups/Proj-RG/providers/Microsoft.Network/virtualNetworks/EastVNet" --allow-vnet-access

Output of the command



{
  "allowForwardedTraffic": false,
  "allowGatewayTransit": false,
  "allowVirtualNetworkAccess": true,
  "doNotVerifyRemoteGateways": false,
  "etag": "W/\"2a34dc33-8176-4ad6-ab9f-db94281f712f\"",
  "id": "/subscriptions/21b8f625-edc8-4cf1-99ec-861cfa81ecc3/resourceGroups/Proj-RG/providers/Microsoft.Network/virtualNetworks/WestVNet/virtualNetworkPeerings/WesttoEastPeering",
  "name": "WesttoEastPeering",
  "peeringState": "Initiated",
  "peeringSyncLevel": "RemoteNotInSync",
  "provisioningState": "Succeeded",
  "remoteAddressSpace": {
    "addressPrefixes": [
      "10.1.0.0/16"
    ]
  },
  "remoteVirtualNetwork": {
    "id": "/subscriptions/21b8f625-edc8-4cf1-99ec-861cfa81ecc3/resourceGroups/Proj-RG/providers/Microsoft.Network/virtualNetworks/EastVNet",
    "resourceGroup": "Proj-RG"
  },
  "remoteVirtualNetworkAddressSpace": {
    "addressPrefixes": [
      "10.1.0.0/16"
    ]
  },
  "resourceGroup": "Proj-RG",
  "resourceGuid": "ed95c55f-94b2-0ecd-2f52-dc24ce394fb7",
  "type": "Microsoft.Network/virtualNetworks/virtualNetworkPeerings",
  "useRemoteGateways": false
}

To peer East to West. That is peering EastVNet to WestVNet, in order to complete the peering circuit.
We run this command again for the WestVNet to get the resource ID

az network vnet show --resource-group Proj-RG --name WestVNet --query id --out tsv

We can now go ahead and run the peering command to peer EastVnet to WestVnet.



az network vnet peering create --name EasttoWestPeering --resource-group Proj-RG --vnet-name EastVNet --remote-vnet "/subscriptions/21b8f625-edc8-4cf1-99ec-861cfa81ecc3/resourceGroups/Proj-RG/providers/Microsoft.Network/virtualNetworks/WestVNet" --allow-vnet-access

Output of the command



{
  "allowForwardedTraffic": false,
  "allowGatewayTransit": false,
  "allowVirtualNetworkAccess": true,
  "doNotVerifyRemoteGateways": false,
  "etag": "W/\"32bb25f8-1dc4-4b74-af18-4a183d67f138\"",
  "id": "/subscriptions/21b8f625-edc8-4cf1-99ec-861cfa81ecc3/resourceGroups/Proj-RG/providers/Microsoft.Network/virtualNetworks/EastVNet/virtualNetworkPeerings/EasttoWestPeering",
  "name": "EasttoWestPeering",
  "peeringState": "Connected",
  "peeringSyncLevel": "FullyInSync",
  "provisioningState": "Succeeded",
  "remoteAddressSpace": {
    "addressPrefixes": [
      "10.2.0.0/16"
    ]
  },
  "remoteVirtualNetwork": {
    "id": "/subscriptions/21b8f625-edc8-4cf1-99ec-861cfa81ecc3/resourceGroups/Proj-RG/providers/Microsoft.Network/virtualNetworks/WestVNet",
    "resourceGroup": "Proj-RG"
  },
  "remoteVirtualNetworkAddressSpace": {
    "addressPrefixes": [
      "10.2.0.0/16"
    ]
  },
  "resourceGroup": "Proj-RG",
  "resourceGuid": "ed95c55f-94b2-0ecd-2f52-dc24ce394fb7",
  "type": "Microsoft.Network/virtualNetworks/virtualNetworkPeerings",
  "useRemoteGateways": false
}

We can use this command to verify the state of peering.

az network vnet peering list --resource-group Proj-RG --vnet-name EastVNet --output table

SOME BENEFITS OF VIRTUAL NETWORK PEERING

Private Network Communication: VNet peering enables private, low-latency communication between VNets, creating a virtual private network within Azure. This is especially useful when you want to keep network traffic within the Azure backbone, avoiding exposure to the public internet.

Simplified Network Architecture: VNets that are peered together can communicate as if they are part of a single network. This simplifies network architecture and eliminates the need for complex and costly VPN gateways or dedicated network appliances.

Cost-Efficiency: VNet peering is a cost-effective solution for interconnecting VNets. You don't incur data transfer charges within the same Azure region when using VNet peering, which can lead to significant cost savings compared to data transfer over a VPN or ExpressRoute.

Low Latency: Since VNet peering traffic stays within Azure's high-speed backbone network, it results in low-latency communication, making it suitable for applications that require quick response times.

Transitive Connectivity: VNet peering can be transitive, allowing traffic to flow from one VNet to another through a peered VNet. This can simplify network topologies and routing.

Redundancy and High Availability: By peering VNets in different regions, you can create redundant network architectures for disaster recovery and high availability scenarios.

Isolation: Even when VNets are peered, they maintain their isolation. You can apply network security group (NSG) rules and route traffic as needed, providing granular control over communication.

Hybrid Scenarios: VNet peering can be used in hybrid scenarios where you connect on-premises networks to Azure VNets, making it easier to extend your network infrastructure to the cloud.

In conclusion, mastering the art of Azure VNet peering with Azure CLI opens up a world of possibilities for creating efficient, secure, and interconnected cloud architectures. By following the step-by-step instructions in this guide, you'll be well on your way to harnessing the full potential of Azure's networking capabilities. So, don't hesitate to dive in, explore, and embrace the power of Azure VNet peering to enhance your cloud infrastructure and take your applications to the next level.

Thank you for your time and happy peering!

A Step-by-Step Guide on Creating a Resource Group, Virtual Network and Subnet in Azure with Terraform.

Bernard Chika Uwaezuoke — Wed, 23 Aug 2023 01:36:42 +0000

Introduction

Welcome to this in-depth blog series where we unravel the art of provisioning Azure resources using Terraform, a powerful Infrastructure as Code (IaC) tool. In this digital era, orchestrating cloud infrastructure efficiently is paramount, and Terraform provides the brush strokes you need to paint your Azure canvas with precision. This article will guide you through the process of creating a resource group, a virtual network, and a subnet in Azure using Terraform, equipping you with the skills to architect and manage cloud resources seamlessly.

What is Terraform?

Terraform is an open-source infrastructure as code (IaC) tool developed by HashiCorp. It allows you to define, manage, and provision infrastructure resources in a declarative way using configuration files. These configuration files, written in HashiCorp Configuration Language (HCL), describe the desired state of your infrastructure components such as virtual machines, networks, storage, and other cloud services.

Terraform provides a way to manage infrastructure across various cloud providers such as AWS, Azure, Google Cloud, Oracle etc. and even on-premises environments. It helps automate the process of creating, updating, and destroying resources by treating infrastructure as code, enabling you to version-control your infrastructure configurations and collaborate more effectively with your team.

Some Terraform key concepts

1. Providers: These are plugins that interface with various cloud providers' APIs. Each provider offers resource types that Terraform can manage, such as virtual machines, networks, databases, and more.

2. Resources: These are the fundamental building blocks of your infrastructure, representing the individual components you want to manage. For example, an AWS instance or an Azure virtual machine might be represented as a resource.

3. Modules: Modules allow you to encapsulate and reuse configurations. They can be thought of as reusable blueprints for creating infrastructure components.

4. State: Terraform maintains a state file that keeps track of the current state of your infrastructure as defined in your configuration files. This state file is used to determine the difference between the desired state and the actual state and to perform updates accordingly.

5. Execution Plan: Terraform generates an execution plan before making any changes to your infrastructure. This plan outlines what actions Terraform will take to reach the desired state, allowing you to review and approve changes before they are applied.

6.Apply: The "terraform apply" command is used to apply the changes defined in your configuration files to your infrastructure. It provisions or updates resources based on the execution plan.

Prerequisites

Install VScode (IDE)
Install Terraform
Azure Account
Install WSL Ubuntu
Azure CLI

Let's dive in now!

Download and install Visual Studio Code
Use the link bellow to download and install Microsoft VScode.

https://code.visualstudio.com/download
Download MS Visual Studio Code – VSCode

- Select the suitable one for your operating system. We are selecting Windows for this article.

Click Install/Launch/
Sign in to MS Visual Studio with your Microsoft Account To create a password for the default user.

Configure Visual Studio to run Ubuntu terminal

Open the VScode and click on the extension icon as shown below.

Type remote wsl in the search bar select and then install.

Install WSL Ubuntu from Microsoft Store

Search for Microsoft Store on your system.
In Microsoft Store, Search for, download, and INSTALL Ubuntu
Unpack the App.

Install Terraform in your VScode

Use the documentation in the link below to Install Terraform
https://cloudlinuxtech.com/install-terraform-on-ubuntu-uninstall-terraform/

Copy and paste these commands, one line at a time:

curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -

sudo apt-add-repository "deb [arch=$(dpkg --print-architecture)] https://apt.releases.hashicorp.com
$(lsb_release -cs) main"

$sudo apt update
$sudo apt install terraform
$ terraform --version
$ which terraform

Click on the extension icon by the left of your VScode search for and install these extensions indicated in the image below:

Configure Provider (Azure)

Now download and install azure-cli for ubuntu with these commands one after the other:

sudo apt update
sudo curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash 
az --version

Now that we are done setting-up our working environment, let's now commence the main business of deploying our resources using terraform.

Deploying our resources

Create a folder on our desktop named it AZURE TERRAFORM1.
Open the folder using VScode
Create two files named provider.tf and main.tf respectively.

Click on the terminal button on the top of the VScode to open your interface below and select the wsl ubuntu to display your shell promptin order to be able to run your Linux commands.

Go to the Terraform registry for Azure on the link below to get your templates for deploying Azure resources.

https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs

In the provider.tf file, copy the provider configuration from the terraform registry and paste. Then go to the terminal and initialize terraform by running this command.

terraform init

You will get this notification indicating that the initialization was successful.

This will open a browser tab for you to provide your login credentials.

Select the appropriate account and sign-in.

The notification below will be displayed on the browser indicating that your login was successful.

The account and subscription details will be displayed in your terminal.

Go to the main.tf file and start deploying your resources.

Provisioning a Resource Group

Go to the Terraform Azure registry and copy a template for Resource group by typing the resource group in the search bar.
Customize to your preference by editing the values of the parameters resource, name and location to suite your specification. The unique id is the element in the quotation mark after the resource.
In this instance, the resource is "azurerm_resource_group" we specified the unique id as Proj-rg. The name is project1-RG and the location is east us

When we are done defining our resource in HashiCorp Configuration Language, we run the terraform init command again to start the process of provisioning.

Then terraform plan command, to ascertain that what we intend doing is indeed what terraform will do.

We can now run the terraform apply command to deploy the resource.

We will be prompted by terraform if we want to go ahead with the provisioning, and we type yes to approve.

Terraform will deploy our resource for us as specified.

View the provisioned resource in *Azure portal *.

To Provision a Virtual Network

Now we want to create a Virtual Network within the resource group we just created.

Go to the Terraform Registry for Azure and copy the template for provisioning a virtual network and paste it right below the resource group code in the main.tf file in our code editor.
Edit it to our specification as shown in the image below.

When we are done editing to our specification, by providing the unique id vnet1, name project1-network, resource group Proj-rg, location specified with the resource group unique id and the private IPv4 range 10.0.0.0/16. We conslude the process by running these same three commands one after the other to deploy the virtual network to Azure cloud.

terraform init
terraform plan
terraform apply

View our Virtual Network in Azure Portal.

Provisioning a subnet in our Virtual network

We begin by going to the terraform library again to search for subnet and copy the desired template and place it right below the virtual network code on our editor.
We continue by providing our customized value for the parameters in the template as shown in the image below.
In this instance, the unique id for our subnet is example and the name is example-subnet.
We provide the resource group and virtual network parameters values using there unique Proj-rg and vnet1 respectively.
We also provide the IPv4 address range for the subnet 10.0.1.0/24.

We conclude the process by running the terraform commands one after the other again.

terraform init
terraform plan
terraform apply

We view our subnet in Azure portal.

We can also see the address range of the virtual network the resource group in the Azure portal.

Below is our module, which we can push to Github for future us and to be used by others.

Destroying our resources
As convention demands, we clean-up our cloud account by deleting resources that are no longer in use.
To destroy our resources, using terraform, we just run the command terraform destroy on out terminal and type yes when prompted or add the argument --auto-approve to approve the removal automatically.

From the above images, we can see that the resources we provisioned have been destroyed.

With this, we come to this guide on using terraform for Azure.

Conclusion
In this step-by-step guide, we've embarked on a journey into the world of cloud infrastructure provisioning with Terraform on the Microsoft Azure platform. We've covered the fundamentals, walked through the essential setup, and explored how to create and manage resources in a structured and efficient manner.

As you venture forward, consider exploring advanced Terraform topics such as managing state, handling secrets, and **integrating **with other tools in the DevOps ecosystem. Remember, learning is a continuous journey, and each step you take deepens your mastery.

Thank you for joining us on this exploration of Terraform and Azure. Please, feel free to share your thoughts and questions in the comments section, and remember, to subscribe and follow us for more articles like this.