DEV Community: Hassan Aftab

AI-Powered Penetration Testing: How I Used Claude + Kali Linux MCP to Automate Security Assessments

Hassan Aftab — Thu, 22 Jan 2026 08:37:25 +0000

Introduction: The Future of Offensive Security is Conversational

Picture this: Instead of juggling multiple terminal windows, memorizing command syntax, and manually piecing together scan results, you simply have a conversation with an AI that executes security tools, analyzes findings, and generates comprehensive reports—all in real-time.

Sounds like science fiction? It's not. I just completed a full penetration test using Claude Desktop connected to a Kali Linux MCP (Model Context Protocol) server, and the experience has fundamentally changed how I think about security assessments.

In this article, I'll walk you through exactly how I set this up, what I discovered, and why this approach is a game-changer for DevSecOps professionals.

The Problem with Traditional Pen Testing

As security professionals, we've all been there:

Terminal juggling: Multiple SSH sessions, tmux panes, and terminal tabs
Command syntax hell: Was it nmap -sV -sC or -sC -sV? Do I need sudo?
Context switching: Running a scan, analyzing output, documenting findings, then moving to the next tool
Report fatigue: Hours spent formatting findings into readable reports
Knowledge gaps: Junior analysts missing critical steps in methodology

Don't get me wrong—traditional pen testing works. But it's slow, error-prone, and doesn't scale well for modern DevSecOps teams conducting continuous security assessments.

Enter AI-Assisted Security Testing

The idea is simple but powerful: What if we could have a conversational interface to our security tools?

Instead of this traditional workflow:

# Terminal 1: Port scanning
nmap -sV -sC -p 80,443 target.example.com -oN nmap_results.txt

# Terminal 2: Directory enumeration  
ffuf -u https://target.example.com/FUZZ -w wordlist.txt -mc 200,403

# Terminal 3: Header analysis
curl -I https://target.example.com

# Terminal 4: Take notes, start writing report...
vim findings.md

We could do this:

Me: "Run nmap on ports 80 and 443, then check for common vulnerabilities with other tools"

AI: *Executes scans, analyzes results, identifies issues*
    "I've completed the assessment. Found strong security headers but 
    discovered CSP using 'unsafe-inline'. Here's the full report..."

This isn't just about convenience—it's about fundamentally rethinking how we approach security testing.

The Technology Stack

Here's what I used to make this work:

1. Claude Desktop

The AI interface that understands security context and can reason about findings. Claude can:

Understand security terminology and concepts
Chain multiple tools together logically
Analyze output for actual vulnerabilities (not just running tools)
Generate professional reports with prioritized findings

2. Kali Linux MCP Server

A Model Context Protocol server that provides Claude with secure access to a Kali Linux environment. Think of it as a bridge between the AI and your security tools.

Key features:

Sandboxed execution environment
Support for all standard Kali tools
Background job management for long-running scans
Output parsing and structured results

3. Traditional Security Tools

The same tools we know and love:

nmap - Port scanning and service detection
ffuf - Directory and file enumeration
whatweb - Technology fingerprinting
curl - HTTP analysis
netcat - Network connectivity testing

The difference? Now they're orchestrated by AI.

The Assessment: A Real-World Example

Let me walk you through an actual security assessment I performed using this setup. (All identifying information has been anonymized for obvious reasons.)

Phase 1: Reconnaissance

Me: "Let's run a quick nmap scan on ports 80 and 443"

Claude immediately executed:

nmap -sV -sC -p 80,443 target.example.com

Within seconds, I had results showing:

Port 80: Returns 426 Upgrade Required (forcing HTTPS ✅)
Port 443: nginx server, valid SSL certificate
Technology: HTML5 Single Page Application

But here's where it gets interesting—Claude didn't just dump the nmap output. It analyzed it:

"The application properly enforces HTTPS. Port 80 returns 426 Upgrade Required,
which is better than a simple 301 redirect. The SSL certificate is valid and
the server is running nginx. Let's check the security headers next."

Phase 2: Security Header Analysis

Without me having to specify the exact curl command, Claude proactively ran:

curl -I https://target.example.com

And immediately identified the security posture:

✅ Strong Controls Found:

HSTS with preload (max-age=31536000)
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Secure cookies with HttpOnly and SameSite flags

⚠️ Concerns Identified:

Content-Security-Policy uses 'unsafe-inline'
Multiple trusted external domains in CSP

Claude didn't just list these—it explained the security implications of each finding.

Phase 3: Directory Enumeration

Me: "Let's try ffuf for directory enumeration"

Claude executed:

ffuf -u https://target.example.com/FUZZ \
     -w /usr/share/seclists/Discovery/Web-Content/common.txt \
     -mc 200,204,301,302,307,401,403,405 \
     -fc 404 -t 40

Results: 4,750 paths tested in ~25 seconds at 190 req/sec

The interesting part? Claude automatically analyzed the results:

4,674 paths returned 200 with identical content (typical SPA behavior)
19 paths returned 403 Forbidden (sensitive files properly blocked)
2 paths returned 301 redirects (potential misconfiguration!)

Phase 4: Deep Dive on Findings

Here's where AI really shines. Claude noticed something interesting in the redirects:

/assets → http://target.example.com:4200/assets/
/media  → http://target.example.com:4200/media/

Claude's analysis:

"These redirects reference port 4200, which is the default port for Angular
development servers. This suggests either:

A development server is running in production

Nginx is misconfigured

Assets aren't being served properly

Let me test if port 4200 is accessible..."

Claude then automatically ran netcat to test:

nc -zv -w 5 target.example.com 4200

Result: Port 4200 is NOT accessible externally (good!), but the configuration issue remains.

This kind of contextual reasoning—connecting dots between different findings—is where AI really excels.

Phase 5: Comprehensive Reporting

At the end, I simply said: "Generate a comprehensive security report"

Claude produced:

✅ Executive summary with risk rating
✅ Detailed findings with CVSS-style severity ratings
✅ Technical details for each vulnerability
✅ Prioritized remediation recommendations
✅ Compliance mapping (OWASP Top 10)
✅ Evidence with exact commands and output

Total time from start to final report: ~15 minutes

Traditional approach: Would have taken 2-3 hours

Real Findings (Anonymized)

Here's what the assessment uncovered:

🟢 Strong Security Controls (Good News)

HTTPS Enforcement: Perfect implementation with 426 status code
Security Headers: Comprehensive set of modern security headers
File Access Controls: All sensitive files (.git, .env, .svn) properly blocked with 403
Cookie Security: HttpOnly, Secure, and SameSite flags properly set
SSL/TLS: Valid certificate, HTTP/2 enabled

🟡 Medium Priority Issues

CSP 'unsafe-inline'
- Both script-src and style-src allow inline scripts
- Reduces XSS protection effectiveness
- Recommendation: Remove 'unsafe-inline', use nonce-only approach
Port 4200 References
- Redirects expose internal development port
- Suggests nginx misconfiguration
- Recommendation: Fix asset serving configuration
Development Environment Exposure
- Domain clearly marked as "dev"
- robots.txt confirms staging environment
- Recommendation: Implement IP whitelisting or VPN access

🟢 Low Priority Observations

Broad CSP Domain Trust: Multiple Azure services in allow-list
Server Header Exposure: nginx version visible
Certificate Expiration: Valid for 2 more months

Overall Risk Rating: LOW-MEDIUM

The application has solid security fundamentals with room for CSP hardening and access control improvements.

The Real Value: Beyond Tool Execution

Here's what makes this approach truly powerful—it's not just about running tools faster. It's about:

1. Intelligent Analysis

Claude doesn't just execute commands; it understands security concepts:

Recognizes what 'unsafe-inline' means for CSP
Knows that port 4200 is an Angular dev server
Understands the relationship between findings
Prioritizes issues based on actual risk

2. Contextual Reasoning

When Claude found the port 4200 reference, it didn't stop there:

Tested if the port was accessible
Explained what port 4200 typically indicates
Suggested multiple potential causes
Recommended specific fixes

3. Adaptive Methodology

The assessment flow was dynamic:

Started with broad reconnaissance
Dove deeper based on findings
Connected related issues
Adjusted scan parameters based on results

4. Knowledge Transfer

Every step was explained:

Why each tool was chosen
What the output means
How findings relate to security principles
What the business impact is

This makes it perfect for training junior security analysts.

Practical Applications

This approach works incredibly well for:

1. Continuous Security Testing

Integrate AI-assisted scanning into CI/CD pipelines:

# In your CI/CD pipeline
- name: Security Scan
  run: |
    claude-security-scan --target $STAGING_URL \
                         --output security-report.md

2. Compliance Audits

"Check this application against OWASP Top 10 and generate a compliance report"

3. Security Training

Junior analysts can learn by watching Claude's methodology:

Which tools to use when
How to interpret results
What findings matter most
How to communicate risk

4. Bug Bounty Hunting

Accelerate reconnaissance phase:

Quick subdomain enumeration
Technology fingerprinting
Common vulnerability checks
Automated documentation

5. Red Team Exercises

Chain complex attack scenarios:
"Enumerate subdomains, identify web applications, scan for vulnerabilities,
and generate target priority list"

The Limitations (Let's Be Honest)

This approach isn't perfect. Here's what it doesn't do:

❌ Complex Exploitation

Claude can identify vulnerabilities but won't automatically exploit them. SQLi, XSS, and RCE still require human expertise.

❌ Social Engineering

No AI assistance for phishing, pretexting, or physical security testing.

❌ Zero-Day Discovery

This accelerates known vulnerability scanning, not novel vulnerability research.

❌ Replace Critical Thinking

AI amplifies human skills; it doesn't replace security expertise and judgment.

❌ Handle Authentication

Complex authenticated scanning still requires manual session management.

Ethical Considerations

Let me be crystal clear: Always get explicit written authorization before security testing.

This technology makes scanning easier, which also means it's easier to accidentally (or intentionally) test unauthorized systems.

Golden rules:

✅ Get written permission before ANY security testing
✅ Stay within authorized scope
✅ Document everything
✅ Report findings responsibly
✅ Anonymize data when sharing publicly
❌ Never test production systems without approval
❌ Never share sensitive findings publicly

Unauthorized security testing is illegal in most jurisdictions. Don't be that person.

Setting It Up Yourself

Want to try this? Here's how to get started:

Prerequisites

Claude Desktop (or API access)
Docker (for Kali Linux container)
Basic understanding of security tools
Authorization for a test environment

Quick Start: Follow this Guide


# 1. Clone the Kali MCP server
git clone https://github.com/hassanaftab93/kali-docker-mcp

# 2. Build the Docker container
cd kali-mcp-server
docker build -t kali-mcp .

# 3. Run the server
docker run -d -p 3000:3000 kali-mcp

# 4. Configure Claude Desktop
# Add MCP server configuration to settings

# 5. Start testing!
# Open Claude Desktop and start conversing

(Note: URLs anonymized for security. Search for "Kali MCP Server" or "MCP penetration testing" for actual repositories)

The Future of Security Testing

This is just the beginning. Here's where I see this going:

Short Term (Now - 6 months)

Integration with more specialized tools (Burp Suite, Metasploit)
Automated exploit validation
Real-time vulnerability database lookups
Custom security workflow automation

Medium Term (6-18 months)

AI-assisted exploit development
Automated threat modeling
Intelligent false positive filtering
Natural language security policies

Long Term (18+ months)

Autonomous security testing agents
AI-powered red team exercises
Predictive vulnerability analysis
Self-healing security systems

My Take: Augmentation, Not Replacement

Here's the bottom line: AI won't replace security professionals.

But security professionals who use AI will replace those who don't.

This technology handles the tedious parts:

Tool execution
Output parsing
Report generation
Documentation

While we focus on the parts that require human expertise:

Critical thinking
Exploit development
Business context
Strategic recommendations
Client communication

The future of offensive security is collaborative—humans and AI working together.

Conclusion: The Paradigm Shift

Going from traditional pen testing to AI-assisted security assessment feels like going from punch cards to a modern IDE. The fundamental skills are the same, but the experience is night and day.

What took 3 hours now takes 15 minutes.

What required deep tool knowledge now requires clear communication.

What was tedious documentation is now automatic.

This isn't about making security testing easier (though it does). It's about making it better, faster, and more consistent.

If you're in DevSecOps, offensive security, or security research, I highly recommend exploring AI-assisted workflows. Start small—automate one part of your process—and expand from there.

The tools are ready. The technology works. The only question is: Are you ready to adapt?

Resources & Further Reading

Tools Mentioned:

Claude Desktop / Claude API
Kali Linux
nmap, ffuf, whatweb, curl
Model Context Protocol (MCP)

Learning Resources:

OWASP Testing Guide
Model Context Protocol Documentation
Kali Linux Documentation
AI Safety in Security Testing

Communities:

r/netsec
HackerOne Community
AI Security Research Groups

About This Article

This assessment was performed on an authorized test environment with explicit permission. All identifying information has been anonymized. The findings and methodology shared here are for educational purposes.

Questions? Comments? Drop them below or connect with me on LinkedIn.

Found this useful? Share it with your security team and help spread knowledge about AI-assisted security testing.

#cybersecurity #ai #security #devops #testing #automation #tutorial #linux #webdev #cloudcomputing

App of Apps Pattern: Efficient Kubernetes Deployment Strategies with Terraform and ArgoCD

Hassan Aftab — Thu, 23 Jan 2025 19:21:50 +0000

1. Introduction

Cloud-Native Technologies Overview
Terraform, Kubernetes, ArgoCD Integration

2. Traditional Infrastructure Challenges

Manual Management Pitfalls
Deployment Complexities

3. Modern Infrastructure Approach

Infrastructure as Code
GitOps Principles

4. Terraform: Infrastructure Provisioning

Cluster Configuration
Network Setup
Resource Management

5. ArgoCD Configuration

Initial Setup
RBAC Configuration
Login Procedures

6. GitOps Workflow

Repository Structures

Application Repository
GitOps Repository
Manifest Management

Implementation Strategies

Repository Configuration
ArgoCD CLI Commands
"App of Apps" Pattern

7. GitOps Philosophical Principles

Core Concepts

Declarative Configuration
Version Control
Automated Synchronization

Organizational Benefits

Infrastructure Team Advantages
Development Team Improvements
Operational Efficiency

8. Practical Implementation Strategy

Infrastructure Definition
ArgoCD Deployment
Continuous Integration Workflow

9. Conclusion

Technology Synergy
Scalability and Agility

10. Recommended Tools And Resources

Infrastructure Tools
Monitoring Solutions
Version Control Platforms

Introduction

In the rapidly evolving landscape of cloud-native technologies, organizations are continuously seeking strategies to streamline infrastructure provisioning, enhance deployment reliability, and improve team collaboration.

The combination of Terraform, Kubernetes, ArgoCD, and GitOps principles offers a powerful solution to these challenges.

The Traditional Infrastructure Challenge

Traditionally, infrastructure management and application deployments were manual, error-prone processes:

Inconsistent environments
Configuration drift
Limited visibility into changes
Complex rollback procedures
Slow and risky deployments

Enter the Modern Approach Infrastructure as Code and GitOps

Terraform Infrastructure Provisioning Reimagined

Terraform transforms infrastructure management by:

Defining infrastructure using declarative configuration files
Supporting multi-cloud and hybrid cloud deployments
Enabling consistent, reproducible infrastructure creation
Providing a clear, version-controlled view of infrastructure state

Terraform Kubernetes Cluster Provisioning Example

In a project, where my project structure is as follows:

├── Makefile
├── README.md
├── plans
│   └── plan.tfplan
├── resources.helm.tf
├── resources.main.tf
├── resources.outputs.tf
├── resources.variables.tf
├── terraform.backend.tf
├── terraform.data.tf
├── terraform.locals.tf
└── terraform.provider.tf

In resources.main.tf, I define the resources required to spin the kubernetes cluster as seen below:

module "network_security_group" {
  source              = "git::ssh://git@ssh.dev.azure.com/v3/<organizationName>/<projectName>/<repoName>//modules/network_security_group"
  name                = "rnd-nsg"
  location            = data.azurerm_resource_group.existing.location
  resource_group_name = data.azurerm_resource_group.existing.name

  rules = [
    {
      name                       = "nsg-rule-1"
      priority                   = 100
      direction                  = "Inbound"
      access                     = "Allow"
      protocol                   = "*"
      source_port_range          = "*"
      destination_port_range     = "*"
      source_address_prefix      = "*"
      destination_address_prefix = "*"
    },
    {
      name                       = "nsg-rule-2"
      priority                   = 101
      direction                  = "Outbound"
      access                     = "Allow"
      protocol                   = "*"
      source_port_range          = "*"
      destination_port_range     = "*"
      source_address_prefix      = "*"
      destination_address_prefix = "*"
    }
  ]
  depends_on = [data.azurerm_resource_group.existing]
  tags       = merge(var.tags)
}

module "virtual_network" {
  source              = "git::ssh://git@ssh.dev.azure.com/v3/<organizationName>/<projectName>/<repoName>//modules/virtual_network"
  name                = "rndvnet"
  location            = data.azurerm_resource_group.existing.location
  resource_group_name = data.azurerm_resource_group.existing.name
  address_space       = ["10.0.0.0/16"]
  depends_on          = [data.azurerm_resource_group.existing, module.network_security_group.this]
  tags                = merge(var.tags)
}

module "k8ssubnet" {
  source                     = "git::ssh://git@ssh.dev.azure.com/v3/<organizationName>/<projectName>/<repoName>//modules/subnet"
  name                       = "rndsubnetk8s"
  resource_group_name        = data.azurerm_resource_group.existing.name
  virtual_network_name       = module.virtual_network.virtual_network_name
  subnet_address_prefix      = ["10.0.1.0/24"]
  service_endpoints          = ["Microsoft.Storage", "Microsoft.Web"]
  delegation_name            = null
  service_delegation_name    = null
  service_delegation_actions = null
  depends_on                 = [data.azurerm_resource_group.existing, module.virtual_network.this, module.network_security_group.this]
}

module "podsubnet" {
  source                     = "git::ssh://git@ssh.dev.azure.com/v3/<organizationName>/<projectName>/<repoName>//modules/subnet"
  name                       = "rndsubnetpods"
  resource_group_name        = data.azurerm_resource_group.existing.name
  virtual_network_name       = module.virtual_network.virtual_network_name
  subnet_address_prefix      = ["10.0.2.0/24"]
  service_endpoints          = ["Microsoft.Storage"]
  delegation_name            = "aks-delegation"
  service_delegation_name    = "Microsoft.ContainerService/managedClusters"
  service_delegation_actions = ["Microsoft.Network/virtualNetworks/subnets/join/action"]
  depends_on                 = [data.azurerm_resource_group.existing, module.virtual_network.this, module.network_security_group.this]
}

module "subnet_nsg_association" {
  source                    = "git::ssh://git@ssh.dev.azure.com/v3/<organizationName>/<projectName>/<repoName>//modules/subnet_network_security_group_association"
  subnet_id                 = module.k8ssubnet.subnet_id
  network_security_group_id = module.network_security_group.id
  depends_on                = [data.azurerm_resource_group.existing, module.k8ssubnet.this, module.network_security_group.this]
}

module "container_registry" {
  source                           = "git::ssh://git@ssh.dev.azure.com/v3/<organizationName>/<projectName>/<repoName>//modules/container_registry"
  resource_group_name              = data.azurerm_resource_group.existing.name
  location                         = data.azurerm_resource_group.existing.location
  name                             = "rndacrteo"
  sku                              = "Standard"
  is_admin_enabled                 = true
  is_public_network_access_enabled = true
  depends_on                       = [data.azurerm_resource_group.existing]
  tags                             = merge(var.tags)
}

module "kubernetes" {
  source                               = "git::ssh://git@ssh.dev.azure.com/v3/<organizationName>/<projectName>/<repoName>//modules/kubernetes/kubernetes_cluster"
  name                                 = "k8s-cluster"
  resource_group_name                  = data.azurerm_resource_group.existing.name
  location                             = data.azurerm_resource_group.existing.location
  node_count                           = 1
  dns_prefix                           = "rndaks"
  vnet_subnet_id                       = module.k8ssubnet.subnet_id
  vm_size                              = "Standard_A2_v2"
  pod_subnet_id                        = module.podsubnet.subnet_id
  kubernetes_version                   = "1.30.0"
  is_role_based_access_control_enabled = true
  sku_tier                             = "Free"
  default_node_pool_name               = "k8spool"
  service_cidr                         = "10.1.0.0/16"
  dns_service_ip                       = "10.1.0.10"
  depends_on                           = [module.k8ssubnet.this, module.podsubnet.this]
}

module "k8s_role_assignment" {
  source                           = "git::ssh://git@ssh.dev.azure.com/v3/<organizationName>/<projectName>/<repoName>//modules/role_assignment"
  principal_id                     = module.kubernetes.kubelet_identity_object_id
  role_definition_name             = "AcrPull"
  scope                            = module.container_registry.id
  skip_service_principal_aad_check = true
}

Then in resources.helm.tf, I define the resources required to configure the helm charts into the above provisioned cluster

module "ingress_nginx" {
  source = "git::ssh://git@ssh.dev.azure.com/v3/<organizationName>/<projectName>/<repoName>//modules/helm/helm_release"

  chart            = "ingress-nginx"
  name             = "ingress-nginx"
  create_namespace = true
  namespace        = "ingress-nginx"
  values = [
    <<-EOF
    controller:
      service:
        type: LoadBalancer
    EOF
  ]
  repository = "https://kubernetes.github.io/ingress-nginx"
  providers = {
    helm = helm.helmk8s
  }
  depends_on = [data.azurerm_kubernetes_cluster.cluster, module.kubernetes]
}

module "argocd" {
  source = "git::ssh://git@ssh.dev.azure.com/v3/<organizationName>/<projectName>/<repoName>//modules/helm/helm_release"

  chart            = "argo-cd"
  name             = "argocd"
  create_namespace = true
  namespace        = "argocd"
  values = [
    <<-EOF
    server:
      service:
        type: ClusterIP
      configs:
        params:
          "server.insecure": "true"
    admin:
        username: admin
    EOF
  ]
  repository = "https://argoproj.github.io/argo-helm"
  providers = {
    helm = helm.helmk8s
  }
  depends_on = [data.azurerm_kubernetes_cluster.cluster, module.kubernetes]
}

Next, we need to run a few commands to be able to access argoCD's server / CLI

ArgoCD Initial Setup and Configuration Guide

1. Password Reset and Retrieval

# Command Breakdown

# Reset the admin secret by clearing existing password data
kubectl patch secret argocd-secret -n argocd -p '{"data": {"admin.password": null, "admin.passwordMtime": null}}'

##### Restart ArgoCD server to generate new password
kubectl delete pods -n argocd -l app.kubernetes.io/name=argocd-server

##### Retrieve the initial admin password
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d

We will use this password later

# What This Does

Clears existing admin password
Triggers ArgoCD to generate a new password
Extracts the new password for login

2. RBAC (Role-Based Access Control) Configuration

# Edit RBAC ConfigMap

kubectl -n argocd edit configmap argocd-rbac-cm

# RBAC Policy Configuration

policy.csv: |
# Grant full admin access to all applications
p, role:admin, applications, *, *, *

# Set default role to admin
policy.default: role:admin

# Policy Explanation

p, role:admin, applications, *, *, *: Provides full access to all applications
policy.default: role:admin: Sets admin as the default role for all users

NOTE: In our case, we set role:admin, since there will only be one cluster admin that will have access to the kubernetes and/or argoCD server

3. Restart ArgoCD Server

# Restart ArgoCD to apply RBAC changes
kubectl -n argocd rollout restart deployment argocd-server

4. Login to ArgoCD

Since we don't expose our argocd server as a LoadBalancer, or Ingress, we need to access it using port-forwarding, this ensures further security

# Forward port to access ArgoCD server locally
kubectl port-forward service/argocd-server -n argocd 8888:443

##### Login using admin credentials
argocd login 127.0.0.1:8888 --username admin --password <retrieved-password>

Now you can use argocd CLI as well as, open localhost:8888 in the browser and ... Voila!

Important Notes

Security Warning: The default admin role provides full cluster access
Regularly rotate credentials
Use strong, unique passwords

GitOps Workflow

ArgoCD bridges the gap between Git repositories and Kubernetes clusters:

Declarative continuous delivery
Automatic synchronization of application states
Easy rollbacks and versioning
Support for multiple sync strategies

ArgoCD Application Configuration

Application Repo

This repo contains the application code for the microservices, it can be one repo per microservice or it can be a monorepo that holds all microservice code.

It also contains the Pipelines that build and push the images to the Container Registry

./project
├── Azure-Pipelines
│   ├── azure-pipelines.yml
│   ├── azure-pipelines2.yml
│   └── azure-pipelines3.yml
├── Docker
│   ├── Dockerfile.1
│   ├── Dockerfile.2
│   ├── Dockerfile.3
│   └── nginx.conf
├── demo-app
│   ├── README.md
│   ├── angular.json
│   ├── package.json
│   ├── public
│   │   └── favicon.ico
│   ├── src
│   │   ├── app
│   │   │   ├── app.component.css
│   │   │   ├── app.component.html
│   │   │   ├── app.component.spec.ts
│   │   │   ├── app.component.ts
│   │   │   ├── app.config.server.ts
│   │   │   ├── app.config.ts
│   │   │   └── app.routes.ts
│   │   ├── index.html
│   │   ├── main.server.ts
│   │   ├── main.ts
│   │   ├── server.ts
│   │   └── styles.css
│   ├── tsconfig.app.json
│   ├── tsconfig.json
│   └── tsconfig.spec.json
├── demo-app2
│   ├── README.md
│   ├── angular.json
│   ├── package.json
│   ├── public
│   │   └── favicon.ico
│   ├── src
│   │   ├── app
│   │   │   ├── app.component.css
│   │   │   ├── app.component.html
│   │   │   ├── app.component.spec.ts
│   │   │   ├── app.component.ts
│   │   │   ├── app.config.server.ts
│   │   │   ├── app.config.ts
│   │   │   └── app.routes.ts
│   │   ├── index.html
│   │   ├── main.server.ts
│   │   ├── main.ts
│   │   ├── server.ts
│   │   └── styles.css
│   ├── tsconfig.app.json
│   ├── tsconfig.json
│   └── tsconfig.spec.json
└── demo-app3
    ├── README.md
    ├── angular.json
    ├── package.json
    ├── public
    │   └── favicon.ico
    ├── src
    │   ├── app
    │   │   ├── app.component.css
    │   │   ├── app.component.html
    │   │   ├── app.component.spec.ts
    │   │   ├── app.component.ts
    │   │   ├── app.config.server.ts
    │   │   ├── app.config.ts
    │   │   └── app.routes.ts
    │   ├── index.html
    │   ├── main.server.ts
    │   ├── main.ts
    │   ├── server.ts
    │   └── styles.css
    ├── tsconfig.app.json
    ├── tsconfig.json
    └── tsconfig.spec.json

Let's setup the Azure Pipelines for all 3 demo apps, so images can be pushed into ACR for later deployment

GitOps Repo

Next, we have the GitOps Repo that will be configured with ArgoCD to be monitored

./demo-app-manifests
├── manifests
│   ├── argocd
│   │   ├── argocd-application-master
│   │   │   └── masterapp.yaml
│   │   └── argocd-applications
│   │       ├── demo-app-argoapp.yaml
│   │       ├── demo-app2-argoapp.yaml
│   │       └── demo-app3-argoapp.yaml
│   └── main
│       ├── demo-app
│       │   ├── deployment.yaml
│       │   └── service-ingress.yaml
│       ├── demo-app2
│       │   ├── deployment.yaml
│       │   └── service-ingress.yaml
│       └── demo-app3
│           ├── deployment.yaml
│           └── service-ingress.yaml
└── readmes
    ├── README.md
    └── argoCD.md

The manifests folder is crucial for managing your Kubernetes deployments and configurations using ArgoCD. Here's a detailed look at its contents:

argocd:

argocd-application-master/masterapp.yaml:

The main ArgoCD application configuration.

argocd-applications:

Contains multiple ArgoCD application configurations (demo-app-argoapp.yaml, demo-app2-argoapp.yaml, demo-app3-argoapp.yaml), following the "App of Apps" pattern.

This pattern allows you to manage multiple applications under a single ArgoCD application.

main:

demo-app, demo-app2, demo-app3:

Each folder contains Kubernetes manifests for deploying and exposing the respective frontend applications.

deployment.yaml:

Defines the deployment configuration for the application.

service-ingress.yaml:

Configures the service and ingress for the application.

Run Important ArgoCLI Commands

Add the Azure DevOps repository to ArgoCD using the SSH URL:

    argocd repo add git@ssh.dev.azure.com:v3/myteo/Cloud/demo-app-manifests \
    --ssh-private-key-path ~/.ssh/id_azure \
    --upsert

Create a new ArgoCD project named demo-app-project:

    argocd proj create demo-app-project \
    --dest https://kubernetes.default.svc,* \
    --description "demo app project" \
    --src git@ssh.dev.azure.com:v3/myteo/Cloud/demo-app-manifests \
    --upsert

Now we simply run:

    kubectl apply -f ./manifests/argocd/argocd-application-master/masterapp.yaml

This deploys the master app to the argocd instance, which will now manage all the argoapps referenced under ./manifests/argocd/argocd-applications

This design is called the "App of Apps" pattern, similar to the master-slave node concept

The master app now handles deployment of any app yamls configured under the argocd-applications directory

GitOps The Philosophical Shift

GitOps represents a paradigm shift in infrastructure and application management:

Core Principles

Declarative Configuration: Everything is defined as code
Version Control: Git becomes the single source of truth
Automated Synchronization: Continuous reconciliation of desired and actual states
Immutable Infrastructure: Predictable, reproducible environments

Benefits of This Approach

For Infrastructure Teams

Reduced manual intervention
Improved consistency
Enhanced security through controlled changes
Faster recovery from failures

For Development Teams

Self-service deployment model
Increased visibility
Simplified collaboration
Faster time-to-market

For Organizations

Lower operational complexity
Reduced human error
Better compliance and auditing
Scalable infrastructure management

Practical Implementation Strategy

Define Infrastructure with Terraform
- Create Kubernetes cluster configurations
- Set up networking and security groups
- Configure node pools and cluster settings
Install ArgoCD on Kubernetes
- Deploy ArgoCD using Terraform or kubectl
- Configure repository connections
- Set up application deployment configurations
Implement GitOps Workflow
- Store all configurations in Git repositories
- Use pull requests for change management
- Leverage ArgoCD for continuous deployment

Conclusion

The synergy of Terraform, Kubernetes, ArgoCD, and GitOps principles offers a robust, scalable approach to modern infrastructure and application management. By embracing these technologies, organizations can achieve unprecedented levels of efficiency, reliability, and agility.

Recommended Tools and Resources

Terraform
Kubernetes
ArgoCD
Helm
GitHub/GitLab/Azure DevOps
Prometheus
Grafana

A Step-by-Step Guide to CI/CD Pipeline for Angular App with Azure Container Apps

Hassan Aftab — Tue, 10 Dec 2024 09:18:38 +0000

Prerequisites
Step 1: Dockerfile
Step 2: Azure Pipelines Configuration
Step 3: Additional Azure CLI Configuration
Best Practices
Troubleshooting
Conclusion
Resources
Dockerizing Other Technology Stacks

Prerequisites

Azure DevOps account
Azure Subscription
Angular project
Azure CLI installed
Docker installed
Azure Container Registry
Azure Container Apps environment

Step 1: Dockerfile

# Stage 1: Build Angular application
FROM node:16-alpine AS build
WORKDIR /usr/src/app
COPY package.json package-lock.json ./
RUN npm install
COPY . .
RUN npm run build --configuration=production

# Stage 2: Serve with Nginx
FROM nginx:alpine
COPY --from=build /usr/src/app/dist/your-angular-app /usr/share/nginx/html
COPY nginx.conf /etc/nginx/conf.d/default.conf
EXPOSE 80
CMD ["nginx", "-g", "daemon off;"]

Step 2: Azure Pipelines Configuration

trigger:
  - main

variables:
  # Azure Container Registry details
  acrName: 'yourcontainerregistry'
  imageRepository: 'angular-app'
  dockerfilePath: 'Dockerfile'
  tag: '$(Build.BuildId)'

  # Azure Container Apps details
  containerAppName: 'your-container-app-name'
  resourceGroup: 'your-resource-group'

stages:
- stage: Build
  displayName: Build and Test
  jobs:
  - job: BuildAndTest
    pool:
      vmImage: 'ubuntu-latest'
    steps:
    - task: NodeTool@0
      inputs:
        versionSpec: '16.x'
      displayName: 'Install Node.js'

    - script: |
        npm ci
        npm run lint
        npm run test -- --watch=false --browsers=ChromeHeadless
      displayName: 'npm install, lint, and test'

    - task: Docker@2
      displayName: 'Build Docker Image'
      inputs:
        command: build
        repository: $(imageRepository)
        dockerfile: $(dockerfilePath)
        tags: 
          - $(tag)
          - latest

    - task: Docker@2
      displayName: 'Push to Azure Container Registry'
      inputs:
        command: push
        repository: $(imageRepository)
        containerRegistry: $(acrServiceConnection)
        tags: 
          - $(tag)
          - latest

- stage: Approval
  displayName: Deployment Approval
  jobs:
  - job: ApprovalJob
    pool: server
    steps:
    - task: ManualValidation@0
      inputs:
        instructions: 'Please review and approve the deployment'
        onTimeout: 'reject'
        timeoutInMinutes: 120

- stage: Deploy
  displayName: Deploy to Azure Container Apps
  dependsOn: 
    - Build
    - Approval
  condition: succeeded()
  jobs:
  - deployment: DeployToContainerApp
    pool: 
      vmImage: 'ubuntu-latest'
    environment: 'production'
    strategy:
      runOnce:
        deploy:
          steps:
          - task: AzureCLI@2
            displayName: 'Deploy to Azure Container Apps'
            inputs:
              azureSubscription: 'Your-Azure-Subscription'
              scriptType: 'bash'
              scriptLocation: 'inlineScript'
              inlineScript: |
                # Login to Azure Container Registry
                az acr login --name $(acrName)

                # Update Container App with new image
                az containerapp update \
                  --name $(containerAppName) \
                  --resource-group $(resourceGroup) \
                  --image $(acrName).azurecr.io/$(imageRepository):$(tag)

Step 3: Additional Azure CLI Configuration

Before running the pipeline, set up your Azure Container Apps:

# Create Container Registry
az acr create --resource-group your-resource-group \
              --name yourcontainerregistry \
              --sku Basic

# Create Container Apps Environment
az containerapp env create \
  --name your-container-app-environment \
  --resource-group your-resource-group \
  --location eastus

# Create Container App
az containerapp create \
  --name your-container-app-name \
  --resource-group your-resource-group \
  --environment your-container-app-environment \
  --image yourcontainerregistry.azurecr.io/angular-app:latest \
  --target-port 80 \
  --ingress external

Best Practices

Use Azure Key Vault for sensitive configurations
Implement comprehensive testing before deployment
Use managed identities for secure access
Set up proper RBAC for deployment
Configure health checks in Container Apps
Use staging slots for zero-downtime deployments

Troubleshooting

Verify network configurations
Check container logs in Azure Portal
Ensure proper IAM roles are assigned
Validate ACR permissions
Check Container Apps network settings

Conclusion

This pipeline provides a robust CI/CD workflow for deploying an Angular application to Azure Container Apps with a manual approval stage.

Resources

so in summary, The pipeline includes:

Build and test stage
Manual approval stage
Deployment to Azure Container Apps
Azure Container Registry integration

Dockerizing Other Technology Stacks

The Dockerfile approach demonstrated for the Angular application can be easily adapted to other popular technology stacks as well

[Boost]

Hassan Aftab — Mon, 09 Dec 2024 06:08:04 +0000

Why Rootless Containers Matter: A Security Perspective

Hassan Aftab ・ Dec 9 '24

#devops #container #kubernetes #cloudcomputing

Why Rootless Containers Matter: A Security Perspective

Hassan Aftab — Mon, 09 Dec 2024 05:58:28 +0000

What Are Rootless Containers?
Security Risks in Root Containers
How Rootless Containers Mitigate Security Risks
Comparison: Root vs. Rootless Containers
Challenges and Limitations of Rootless Containers
Compliant Dockerfile for Rootless Containers
Conclusion

Containers have revolutionized the way software is built, shipped, and deployed, offering unparalleled scalability and portability. However, with great power comes great responsibility, particularly in ensuring the security of containerized applications.

One of the most significant advancements in this area is the advent of rootless containers.

In this article, we explore why rootless containers matter and compare the security vulnerabilities of traditional root containers with rootless alternatives.

What Are Rootless Containers?

Rootless containers are a form of containerization where the container runtime and processes inside the container do not require root (administrator) privileges on the host system.

This contrasts with traditional containers, which often run with elevated privileges and can pose significant security risks if compromised.

Rootless containers achieve isolation using technologies such as:

User Namespaces: Mapping container users to unprivileged host users.
Cgroups: Limiting resource usage without elevated permissions.
Seccomp and AppArmor: Enhancing security through system call filtering and mandatory access control.

Security Risks in Root Containers

Traditional root-based containers introduce several potential vulnerabilities:

Privilege Escalation:
If an attacker gains control of a root-based container, they can potentially escalate privileges to take control of the host system.
Namespace Escape:
Misconfigurations or vulnerabilities in the container runtime could allow attackers to break out of the container and access the host.
Insecure Defaults:
Many container environments are configured with defaults that prioritize ease of use over security, such as granting broad access to host resources.
Untrusted Images:
Pulling and running unverified container images increases the risk of running malicious code.

How Rootless Containers Mitigate Security Risks

Rootless containers address these issues in the following ways:

No Root Privileges:
Since rootless containers run without root privileges on the host, even if an attacker compromises the container, they cannot gain root access to the system.
Enhanced Isolation:
By leveraging user namespaces, rootless containers ensure that processes inside the container cannot interact with sensitive host resources.
Reduced Attack Surface:
Running without elevated permissions minimizes the pathways an attacker can exploit to compromise the host.
Stronger Defaults:
Rootless container runtimes are often designed with security-first principles, reducing the likelihood of insecure configurations.

Comparison: Root vs. Rootless Containers

Aspect	Root Containers	Rootless Containers
Host Privileges	Require root privileges on the host	Do not require root privileges on the host
Risk of Privilege Escalation	High	Minimal
Ease of Configuration	Easier to configure for complex setups	May require additional setup
Compatibility	Broad compatibility with existing tools	Limited compatibility with some legacy tools
Security	Higher risk of host compromise	Stronger isolation and lower risk

Challenges and Limitations of Rootless Containers

While rootless containers significantly enhance security, they are not without challenges:

Performance Overhead:
Rootless containers may introduce minor performance trade-offs due to additional isolation layers.
Compatibility Issues:
Some containerized applications or tools may not work seamlessly in a rootless environment.
Complexity:
Initial setup and troubleshooting can be more complex compared to root-based containers.

Compliant Dockerfile for Rootless Containers

Below is a Dockerfile designed to adhere to best practices for rootless containers. It avoids running processes as the root user and implements security-focused configurations.

# Use a minimal base image
FROM debian:bullseye-slim

# Set a non-root user and group
ARG USERNAME=appuser
ARG USER_UID=1001
ARG USER_GID=1001

# Create the non-root user
RUN groupadd --gid $USER_GID $USERNAME && \
    useradd --uid $USER_UID --gid $USER_GID --shell /bin/bash --create-home $USERNAME

# Switch to the non-root user
USER $USERNAME

# Set the working directory
WORKDIR /home/$USERNAME/app

# Copy application files
COPY --chown=$USER_UID:$USER_GID . .

# Install application dependencies
RUN mkdir -p /home/$USERNAME/app/data && \
    chmod -R 700 /home/$USERNAME/app

# Define the entry point and ensure it runs as the non-root user
ENTRYPOINT ["./entrypoint.sh"]

# Expose the necessary ports (non-root port numbers)
EXPOSE 8080

# Start the application
CMD ["./app"]

Explanation of the Dockerfile:

Minimal Base Image:
- The debian:bullseye-slim image is used to minimize the attack surface.
Non-Root User:
- A new user (appuser) is created with limited privileges to ensure processes do not run as root.
File Ownership:
- Application files are copied with ownership set to the non-root user for proper permissions.
Secure Ports:
- Ports above 1024 (e.g., 8080) are exposed, as binding to lower ports requires root privileges.
Strict Permissions:
- Application directories are created with strict permissions (700) to prevent unauthorized access.
Entrypoint and CMD:
- The ENTRYPOINT and CMD are set to ensure the application runs securely under the non-root user.

This Dockerfile follows best practices for rootless containers, ensuring enhanced security and compliance.

Conclusion

As container adoption continues to grow, so does the importance of addressing security risks. Rootless containers provide a robust solution for mitigating many of the vulnerabilities associated with traditional root-based containers.

By running without elevated privileges and offering enhanced isolation, they empower organizations to build more secure containerized environments.

However, adopting rootless containers requires careful consideration of their limitations and potential impacts on workflows.

By balancing security with usability, organizations can leverage rootless containers to create safer, more resilient infrastructures for their applications.

Using Docker for Penetration Testing: A Practical Guide

Hassan Aftab — Fri, 06 Dec 2024 08:09:39 +0000

Why Use Docker for Penetration Testing?
Popular Docker Images for Penetration Testing
Setting Up Docker for Penetration Testing
- Prerequisites
- Example: Running Kali Linux in a Container
- Example: Using OWASP ZAP
- Example: Running Nikto
- Example: Running Nmap
- Example: Reporting with Dradis
Best Practices
Conclusion

Penetration testing often requires a variety of tools, each with specific dependencies and configurations. Installing and maintaining these tools directly on your system can be cumbersome and messy. This is where Docker comes in, offering a clean, efficient, and portable way to set up your penetration testing environment.

In this article, we’ll explore how to use Docker for penetration testing, why it’s beneficial, and provide some practical examples to get you started.

Why Use Docker for Penetration Testing?

Here are a few reasons why Docker is an excellent choice for penetration testing:

Isolation: Each tool runs in its own container, preventing conflicts between dependencies.
Portability: Containers can run consistently across different environments.
Ease of Use: Pre-built Docker images are available for many popular penetration testing tools.
Version Control: You can easily manage different versions of tools by specifying the desired image tags.
Quick Setup: No need to install dependencies manually; just pull the required image and start using it.

Popular Docker Images for Penetration Testing

Here are some popular Docker images you can use to set up your penetration testing environment:

Kali Linux: The go-to Linux distribution for penetration testers.
OWASP ZAP: An open-source web application security scanner.
Metasploit Framework: A widely used penetration testing framework.
Nikto: A web server scanner for detecting vulnerabilities.
SQLMap: An automated tool for SQL injection.
Nmap: A powerful network discovery and security auditing tool.
Reporting Tools (Dradis): A collaboration and reporting platform for penetration testers.

Setting Up Docker for Penetration Testing

Prerequisites

Before starting, ensure you have Docker installed on your system. You can download and install Docker from the official website.

Example: Running Kali Linux in a Container

Pull the Kali Linux image:

   docker pull kalilinux/kali-linux-docker

Run a container:

   docker run -it kalilinux/kali-linux-docker /bin/bash

Install your required tools within the container:

   apt update && apt install -y nmap metasploit-framework nikto

Save the configured container as an image (optional):

   docker commit <container_id> kalilinux/with-tools

Now you can reuse this customized image without reconfiguring it.

Example: Using OWASP ZAP

Pull the OWASP ZAP image:

   docker pull owasp/zap2docker-stable

Run the container with GUI support (requires X11 forwarding):

   docker run -it -e DISPLAY=$DISPLAY -v /tmp/.X11-unix:/tmp/.X11-unix owasp/zap2docker-stable

Start scanning: Use the OWASP ZAP GUI to configure and initiate scans.

Example: Running Nikto

Run a container:

   docker run -it kalilinux/with-tools /bin/bash

Run a Nikto scan:

   nikto -h example.com

Example: Running Nmap

Run an Nmap scan:

   nmap -sV -p 80,443 example.com

Example: Reporting with Dradis

Pull the Dradis image:

   docker pull dradis/dradispro-ce

Run the Dradis container:

   docker run -d -p 3000:3000 dradis/dradispro-ce

Access the Dradis interface: Open your browser and navigate to http://localhost:3000 to start creating reports.

Best Practices

Use Bind Mounts: Mount directories from your host system into the container for easy data sharing.

   docker run -v $(pwd)/data:/data kalilinux/with-tools

Network Modes: Use Docker’s network modes (bridge, host, etc.) to simulate different network scenarios.
Custom Images: Create custom Dockerfiles for commonly used toolsets to save time.
Security: Be cautious with sensitive data in containers and clean up unused containers and images.

Conclusion

Docker simplifies the setup and management of penetration testing tools, making it a valuable asset for security professionals. By using pre-built images or creating custom containers, you can streamline your workflow, reduce setup time, and focus on testing.

Whether you’re a beginner or an experienced penetration tester, leveraging Docker can enhance your productivity and efficiency.

Happy testing! If you have any favorite Docker-based tools or tips, feel free to share them in the comments below.

Check out my latest article on Jupyter Notebooks and Docker

Hassan Aftab — Fri, 29 Nov 2024 09:26:54 +0000

Jupyter Notebooks in Docker

Hassan Aftab ・ Nov 29 '24

#datascience #docker #dataengineering #programming

Jupyter Notebooks in Docker

Hassan Aftab — Fri, 29 Nov 2024 09:16:31 +0000

Why use Docker for Jupyter Notebooks?

Docker provides an efficient and reproducible environment for running Jupyter Notebooks. With Docker, you can create isolated containers that ensure the dependencies and configurations for your Jupyter Notebooks are consistent across different systems. This is particularly useful for data science projects, where managing dependencies and ensuring reproducibility is crucial.

In this article, we will cover how to:

Set up Docker for Jupyter Notebooks.
Use pre-built Jupyter Docker images.
Customize your Jupyter Notebook environment with a custom Dockerfile.
Manage persistent data storage for notebooks.

1. Setting up Docker

Install Docker

Before using Docker, ensure it is installed on your system:

Linux: Follow the Docker Engine installation guide.
Windows/Mac: Install Docker Desktop from Docker’s official website.

Verify Docker installation:

docker --version

Verify Docker Service

Ensure the Docker service is running:

sudo systemctl start docker  # Linux

2. Using Pre-Built Jupyter Docker Images

The Jupyter Project provides pre-built Docker images with various configurations. These images come with pre-installed packages tailored for data science, machine learning, and scientific computing.

Available Images

Some popular Jupyter Docker images:

jupyter/base-notebook:
- Minimal Jupyter Notebook environment.
jupyter/scipy-notebook:
- Includes scientific computing libraries like NumPy, SciPy, and pandas.
jupyter/tensorflow-notebook:
- Includes TensorFlow and Keras for machine learning.
jupyter/r-notebook:
- Supports R in Jupyter.

Running a Jupyter Notebook Container

To start a Jupyter Notebook using the scipy-notebook image:

docker run -p 8888:8888 jupyter/scipy-notebook

Once the container starts, you will see a URL with a token in the logs, such as:

http://127.0.0.1:8888/?token=<token>

Copy this URL into your browser to access the Jupyter Notebook interface.

Running Jupyter with a Named Volume

To persist your work:

docker run -p 8888:8888 -v $(pwd):/home/jovyan/work jupyter/scipy-notebook

This command mounts your current directory to the container's work directory, ensuring changes are saved locally.

3. Customizing Your Jupyter Environment

If the pre-built images don’t meet your needs, you can create a custom Docker image with your own dependencies and configurations.

Creating a Custom Dockerfile

Create a Dockerfile:

   FROM jupyter/scipy-notebook

   # Install additional Python packages
   RUN pip install matplotlib seaborn

   # Set a custom working directory
   WORKDIR /home/jovyan/my-project

Build the Docker image:

   docker build -t my-custom-jupyter .

Run the container:

   docker run -p 8888:8888 -v $(pwd):/home/jovyan/work my-custom-jupyter

Adding Conda Environments

To include a Conda environment, modify the Dockerfile:

FROM jupyter/scipy-notebook

# Create and activate a Conda environment
RUN conda create -n myenv python=3.9 && \
    echo "source activate myenv" > ~/.bashrc

# Install packages in the new environment
RUN conda install -n myenv pandas matplotlib

4. Persistent Data Storage

By default, any data or notebooks created inside a Docker container are lost when the container stops. To avoid this, you can use Docker volumes or bind mounts.

Using Docker Volumes

Volumes are managed by Docker and provide a way to persist data independently of the container lifecycle:

docker volume create jupyter-data

docker run -p 8888:8888 -v jupyter-data:/home/jovyan/work jupyter/scipy-notebook

The jupyter-data volume will persist your notebooks and files.

Using Bind Mounts

Bind mounts map a local directory to a directory inside the container:

docker run -p 8888:8888 -v $(pwd):/home/jovyan/work jupyter/scipy-notebook

This maps the current directory ($(pwd)) to /home/jovyan/work in the container.

5. Advanced Usage

Networking

To allow multiple users to access your Jupyter Notebook over a network:

docker run -p 8888:8888 --ip=0.0.0.0 jupyter/scipy-notebook

Share the URL with the token for access.

Using Docker Compose

For more complex setups, use Docker Compose to manage multiple services (e.g., Jupyter + a database):

version: '3'
services:
  jupyter:
    image: jupyter/scipy-notebook
    ports:
      - "8888:8888"
    volumes:
      - ./notebooks:/home/jovyan/work
  postgres:
    image: postgres
    environment:
      POSTGRES_USER: user
      POSTGRES_PASSWORD: password

Start the services:

docker-compose up

Conclusion

Docker is an excellent tool for running and managing Jupyter Notebooks in a reproducible and isolated environment. Whether you use pre-built images or create custom ones, Docker simplifies dependency management and ensures consistent environments for your projects. By leveraging persistent storage and tools like Docker Compose, you can scale your Jupyter Notebook workflows to handle complex, multi-container setups.

Securing Your Web Applications (DAST): A Deep Dive into OWASP ZAP Scans with Docker

Hassan Aftab — Tue, 15 Oct 2024 15:35:03 +0000

Scanning Localhost Application with Docker ZAP

Scanning Localhost Application with Docker ZAP
- Table of Contents
- Introduction
- Key Features of ZAP
- Benefits of Using ZAP for DAST
- Getting Started with ZAP
- Setting up TIWAP
- Step-by-Step Instructions to Set Up the Web Application
- Docker Installation Instructions
- Execution Instructions
- Step 1: Pull the ZAP Docker Image
- Step 2: Start Your Localhost Application
- Step 3: Run the ZAP Scan
  - Explanation of the Command
  - Additional Flags to Pass to Image
- Command Output
- Step 4: Retrieve the HTML Report
- Additional Tips

Introduction

Dynamic Application Security Testing (DAST) is a type of security testing that involves testing an application in its running state. Unlike static analysis, which examines the source code, DAST scans an application from the outside in, simulating an attacker’s perspective. This approach helps identify vulnerabilities that might be exploited in a real-world attack scenario, such as SQL injection, cross-site scripting (XSS), and other web application vulnerabilities.

One of the most popular tools for performing DAST is the Zed Attack Proxy (ZAP), an open-source security scanner maintained by the Open Web Application Security Project (OWASP). ZAP is designed to be easy to use, even for those new to application security, while also providing powerful features for advanced users.

Key Features of ZAP

Automated Scanning: ZAP can automatically scan web applications for security vulnerabilities.
Passive Scanning: Monitors HTTP traffic and passively scans for vulnerabilities without altering the request-response cycle.
Active Scanning: Actively probes the web application for vulnerabilities by sending various types of malicious requests.
Spidering: Crawls the web application to discover all the pages and endpoints that need to be tested.
API Integration: Provides a robust API that allows for integration with CI/CD pipelines for continuous security testing.
Extensibility: Supports various add-ons and plugins to extend its functionality.

Benefits of Using ZAP for DAST

Comprehensive Coverage: ZAP can identify a wide range of vulnerabilities, including those listed in the OWASP Top Ten.
Ease of Use: Its intuitive interface and extensive documentation make it accessible for beginners.
Community Support: Being an OWASP project, ZAP benefits from a large community of contributors and users who help in improving and updating the tool.
Cost-Effective: As an open-source tool, ZAP is free to use, making it an attractive option for organizations of all sizes.

Getting Started with ZAP

To start using ZAP for DAST scanning, follow these steps:

Download and Install ZAP: ZAP is available for Windows, macOS, and Linux. You can download it from the official ZAP website.
Configure Your Browser: Set up your web browser to use ZAP as a proxy to capture and inspect the HTTP traffic.
Start Scanning: Use ZAP's spidering and scanning features to discover and test the web application's endpoints for vulnerabilities.
Review and Mitigate: Analyze the scan results to identify vulnerabilities and take appropriate actions to mitigate them.

Setting up TIWAP

Step-by-Step Instructions to Set Up the Web Application

Clone the GitHub Repository

First, clone the repository to your local machine. Open your terminal and run the following command:

   git clone https://github.com/hassanaftab93/TIWAP.git

This command downloads all the project files from the GitHub repository to a directory named TIWAP on your local machine.

Navigate to the Project Directory

After cloning the repository, navigate into the project directory:

   cd TIWAP

This changes your current directory to the TIWAP directory, where the project files are located.

Run Docker Compose to Spin Up the Web Application

Use Docker Compose to build and start the web application. Make sure Docker is installed and running on your system. Then run:

   docker-compose up -d

This command uses the docker-compose.yml file to build the Docker containers and start the application in detached mode (running in the background).

Verify the Application is Running

After running the Docker Compose command, verify that the application is running. Open your web browser and navigate to:

   https://localhost:8080

or you can

   curl -k https://localhost:8080

As seen in the images above, the web application up and running.

Docker Installation Instructions

In order for you to run any of these commands, you'll need to have Docker installed. You can view the instructions here: Get Docker: Installation Instructions.

Execution Instructions

Step 1: Pull the ZAP Docker Image

First, ensure you have Docker installed and running on your system. Then, pull the latest OWASP ZAP Docker image:

docker pull zaproxy/zap-stable:latest

Before we start with the zap scans, lets first take a look at the docker image itself

Lets run the following command:

   docker run --rm -it zaproxy/zap-stable:latest bash

This will take us inside the running container of the image and show us the contents in its default working dir, which is /zap

zap@6026a83c9b31:/zap$ ls -la
total 5632
drwxr-xr-x 1 zap  zap     4096 Oct  7 16:47 .
drwxr-xr-x 1 root root    4096 Oct 15 15:26 ..
-rw-r--r-- 1 zap  zap    10324 Oct  7 16:45 CHANGELOG.md
-rw-r--r-- 1 zap  zap     2167 Jan  2  1970 README
-rw-r--r-- 1 zap  zap       18 Oct  7 16:47 container
drwxr-xr-x 2 zap  zap     4096 Jan  2  1970 db
drwxr-xr-x 2 zap  zap     4096 Jan  2  1970 lang
drwxr-xr-x 2 zap  zap     4096 Jan  2  1970 lib
drwxr-xr-x 2 zap  zap     4096 Jan  2  1970 license
drwxr-xr-x 2 zap  zap     4096 Oct  7 16:47 plugin
drwxr-xr-x 3 zap  zap     4096 Jan  2  1970 scripts
drwxr-xr-x 1 zap  zap     4096 Oct  7 16:45 webswing
drwxr-xr-x 2 zap  zap     4096 Jan  2  1970 xml
-rw-r--r-- 1 zap  zap  5450049 Jan  2  1970 zap-2.15.0.jar
-rwxr-xr-x 1 zap  zap    26496 Oct  7 16:45 zap-api-scan.py
-rwxr-xr-x 1 zap  zap    24732 Oct  7 16:45 zap-baseline.py
-rwxr-xr-x 1 zap  zap    20576 Oct  7 16:45 zap-full-scan.py
-rwxr-xr-x 1 zap  zap     2977 Oct  7 16:45 zap-webswing.sh
-rwxr-xr-x 1 zap  zap      319 Oct  7 16:45 zap-x.sh
-rw-r--r-- 1 zap  zap      188 Jan  2  1970 zap.bat
-rw-r--r-- 1 zap  zap   123778 Jan  2  1970 zap.ico
-rwxr-xr-x 1 zap  zap     4231 Jan  2  1970 zap.sh
-rw-r--r-- 1 zap  zap    23306 Oct  7 16:45 zap_common.py

As seen above in the output, we have 3 main python executable files that we can use for scans, later in the article we will see the flag to pass the python file name to use

zap-api-scan.py
zap-baseline.py
zap-full-scan.py

Step 2: Start Your Localhost Application

Make sure your local application is running on localhost. For example, if it's a web application, it might be accessible at
http://localhost:8080.

Step 3: Run the ZAP Scan

Use the following command to run the ZAP scanner against your localhost application. This command assumes your application is accessible on port 8080. Adjust the port number accordingly if your application uses a different port.

Linux/Unix

docker run -v $(pwd):/zap/wrk/:rw --network="host" zaproxy/zap-stable zap-baseline.py -t https://localhost:8080 -r scan-report.html

Windows Powershell

docker run -v ${PWD}:/zap/wrk/:rw --network="host" zaproxy/zap-stable zap-baseline.py -t https://localhost:8080 -r scan-report.html

Explanation of the Command

-v $(pwd):/zap/wrk/:rw

This option mounts the current directory ($(pwd)) to the /zap/wrk/ directory inside the container. The report will be saved in this directory.

--network="host"

This tells Docker to use the host's network, allowing the Docker container to access localhost.

zap-full-scan.py

This is a script provided by ZAP for running a full scan, which performs a deeper and more thorough scan. It will take a while though.

-t http://localhost:8000

Specifies the target URL for the scan. NOTE: You can however provide URL for your own web app as well, be it on localhost or hosted somewhere, hopefully not in production yet :D

-r scan-report.html

Specifies the name of the report file to generate.

Additional Flags to pass to Image

Usage: zap-baseline.py -t <target> [options]
    -t target         target URL including the protocol, eg https://www.example.com
Options:
    -h                print this help message
    -c config_file    config file to use to INFO, IGNORE or FAIL warnings
    -u config_url     URL of config file to use to INFO, IGNORE or FAIL warnings
    -g gen_file       generate default config file (all rules set to WARN)
    -m mins           the number of minutes to spider for (default 1)
    -r report_html    file to write the full ZAP HTML report
    -w report_md      file to write the full ZAP Wiki (Markdown) report
    -x report_xml     file to write the full ZAP XML report
    -J report_json    file to write the full ZAP JSON document
    -a                include the alpha passive scan rules as well
    -d                show debug messages
    -P                specify listen port
    -D secs           delay in seconds to wait for passive scanning 
    -i                default rules not in the config file to INFO
    -I                do not return failure on warning
    -j                use the Ajax spider in addition to the traditional one
    -l level          minimum level to show: PASS, IGNORE, INFO, WARN or FAIL, use with -s to hide example URLs
    -n context_file   context file which will be loaded prior to spidering the target
    -p progress_file  progress file which specifies issues that are being addressed
    -s                short output format - dont show PASSes or example URLs
    -T mins           max time in minutes to wait for ZAP to start and the passive scan to run
    -U user           username to use for authenticated scans - must be defined in the given context file (post 2.9.0)
    -z zap_options    ZAP command line options e.g. -z "-config aaa=bbb -config ccc=ddd"
    --hook            path to python file that define your custom hooks
    --auto            use the automation framework if supported for the given parameters (this will become the default soon)
    --autooff         do not use the automation framework even if supported for the given parameters

Command Output


> docker run -v $(pwd):/zap/wrk/:rw --network="host" zaproxy/zap-stable zap-baseline.py -t https://localhost:8080 -r scan-report.html
Using the Automation Framework
Total of 19 URLs
PASS: Vulnerable JS Library (Powered by Retire.js) [10003]
PASS: In Page Banner Information Leak [10009]
PASS: Cookie No HttpOnly Flag [10010]
PASS: Cookie Without Secure Flag [10011]
PASS: Cross-Domain JavaScript Source File Inclusion [10017]
PASS: Content-Type Header Missing [10019]
PASS: Information Disclosure - Debug Error Messages [10023]
PASS: Information Disclosure - Sensitive Information in URL [10024]
PASS: Information Disclosure - Sensitive Information in HTTP Referrer Header [10025]
PASS: HTTP Parameter Override [10026]
PASS: Open Redirect [10028]
PASS: Cookie Poisoning [10029]
PASS: User Controllable Charset [10030]
PASS: User Controllable HTML Element Attribute (Potential XSS) [10031]
PASS: Viewstate [10032]
PASS: Directory Browsing [10033]
PASS: Heartbleed OpenSSL Vulnerability (Indicative) [10034]
PASS: Server Leaks Information via "X-Powered-By" HTTP Response Header Field(s) [10037]
PASS: X-Backend-Server Header Information Leak [10039]
PASS: Secure Pages Include Mixed Content [10040]
PASS: HTTP to HTTPS Insecure Transition in Form Post [10041]
PASS: HTTPS to HTTP Insecure Transition in Form Post [10042]
PASS: User Controllable JavaScript Event (XSS) [10043]
PASS: Big Redirect Detected (Potential Sensitive Information Leak) [10044]
PASS: Retrieved from Cache [10050]
PASS: X-ChromeLogger-Data (XCOLD) Header Information Leak [10052]
PASS: Cookie without SameSite Attribute [10054]
PASS: CSP [10055]
PASS: X-Debug-Token Information Leak [10056]
PASS: Username Hash Found [10057]
PASS: X-AspNet-Version Response Header [10061]
PASS: PII Disclosure [10062]
PASS: Timestamp Disclosure [10096]
PASS: Hash Disclosure [10097]
PASS: Cross-Domain Misconfiguration [10098]
PASS: Weak Authentication Method [10105]
PASS: Reverse Tabnabbing [10108]
PASS: Dangerous JS Functions [10110]
PASS: Session Management Response Identified [10112]
PASS: Verification Request Identified [10113]
PASS: Script Served From Malicious Domain (polyfill) [10115]
PASS: Private IP Disclosure [2]
PASS: Session ID in URL Rewrite [3]
PASS: Script Passive Scan Rules [50001]
PASS: Stats Passive Scan Rule [50003]
PASS: Insecure JSF ViewState [90001]
PASS: Java Serialization Object [90002]
PASS: Sub Resource Integrity Attribute Missing [90003]
PASS: Insufficient Site Isolation Against Spectre Vulnerability [90004]
PASS: Charset Mismatch [90011]
PASS: WSDL File Detection [90030]
PASS: Loosely Scoped Cookie [90033]
WARN-NEW: Re-examine Cache-control Directives [10015] x 2 
        https://localhost:8080 (200 OK)
        https://localhost:8080/robots.txt (200 OK)
WARN-NEW: Missing Anti-clickjacking Header [10020] x 1 
        https://localhost:8080 (200 OK)
WARN-NEW: X-Content-Type-Options Header Missing [10021] x 6 
        https://localhost:8080 (200 OK)
        https://localhost:8080/index.css (200 OK)
        https://localhost:8080/login?__debugger__=yes&cmd=resource&f=console.png (200 OK)
        https://localhost:8080/login?__debugger__=yes&cmd=resource&f=debugger.js (200 OK)
        https://localhost:8080/login?__debugger__=yes&cmd=resource&f=style.css (200 OK)
WARN-NEW: Information Disclosure - Suspicious Comments [10027] x 3 
        https://localhost:8080/login?__debugger__=yes&cmd=resource&f=debugger.js (200 OK)
        https://localhost:8080/login?__debugger__=yes&cmd=resource&f=debugger.js (200 OK)
        https://localhost:8080/login (500 Internal Server Error)
WARN-NEW: Strict-Transport-Security Header Not Set [10035] x 12 
        https://localhost:8080 (200 OK)
        https://localhost:8080/app/app.py (404 Not Found)
        https://localhost:8080/app/helper/db_manager.py (404 Not Found)
        https://localhost:8080/index.css (200 OK)
        https://localhost:8080/login?__debugger__=yes&cmd=resource&f=console.png (200 OK)
WARN-NEW: Server Leaks Version Information via "Server" HTTP Response Header Field [10036] x 12 
        https://localhost:8080 (200 OK)
        https://localhost:8080/app/app.py (404 Not Found)
        https://localhost:8080/app/helper/db_manager.py (404 Not Found)
        https://localhost:8080/index.css (200 OK)
        https://localhost:8080/login?__debugger__=yes&cmd=resource&f=console.png (200 OK)
WARN-NEW: Content Security Policy (CSP) Header Not Set [10038] x 8 
        https://localhost:8080 (200 OK)
        https://localhost:8080/app/app.py (404 Not Found)
        https://localhost:8080/app/helper/db_manager.py (404 Not Found)
        https://localhost:8080/login?btn=Confirm+Pin&pin=ZAP (404 Not Found)
        https://localhost:8080/sitemap.xml (404 Not Found)
WARN-NEW: Storable and Cacheable Content [10049] x 12 
        https://localhost:8080 (200 OK)
        https://localhost:8080/app/app.py (404 Not Found)
        https://localhost:8080/app/helper/db_manager.py (404 Not Found)
        https://localhost:8080/index.css (200 OK)
        https://localhost:8080/login?__debugger__=yes&cmd=resource&f=console.png (200 OK)
WARN-NEW: Permissions Policy Header Not Set [10063] x 9 
        https://localhost:8080 (200 OK)
        https://localhost:8080/app/app.py (404 Not Found)
        https://localhost:8080/app/helper/db_manager.py (404 Not Found)
        https://localhost:8080/login?__debugger__=yes&cmd=resource&f=debugger.js (200 OK)
        https://localhost:8080/login?btn=Confirm+Pin&pin=ZAP (404 Not Found)
WARN-NEW: Source Code Disclosure - SQL [10099] x 1 
        https://localhost:8080/login (500 Internal Server Error)
WARN-NEW: Modern Web Application [10109] x 2 
        https://localhost:8080 (200 OK)
        https://localhost:8080/login (500 Internal Server Error)
WARN-NEW: Authentication Request Identified [10111] x 1 
        https://localhost:8080/login (500 Internal Server Error)
WARN-NEW: Absence of Anti-CSRF Tokens [10202] x 1 
        https://localhost:8080/login (500 Internal Server Error)
WARN-NEW: Application Error Disclosure [90022] x 1 
        https://localhost:8080/login (500 Internal Server Error)
FAIL-NEW: 0     FAIL-INPROG: 0  WARN-NEW: 14    WARN-INPROG: 0  INFO: 0 IGNORE: 0       PASS: 52

Step 4: Retrieve the HTML Report

Once the scan is complete, an HTML report named scan-report.html will be saved in your current working directory. You can open this file in any web browser to view the detailed security scan report.

Here is the report it generates:

Scan Report - Generated for TIWAP

Your generated report can be shared with your teams to make the proper fixes required.

Additional Tips

Scan Duration: Depending on the complexity and size of your application, the scan may take some time. Ensure your application remains running until the scan is completed.
Security Considerations: Regularly update the Docker image with docker pull owasp/zap2docker-stable to ensure you are using the latest security checks and features.

Why Prometheus and Grafana are Essential for Monitoring in DevOps and How They Enhance the SDLC

Hassan Aftab — Mon, 14 Oct 2024 13:44:07 +0000

In the evolving world of software development and operations, monitoring has become a key pillar for ensuring system reliability, scalability, and performance. With the rise of cloud-native applications, micro services, and containerization, traditional monitoring tools often struggle to keep up with the highly dynamic and distributed nature of modern infrastructures.

Enter Prometheus and Grafana—two powerful open-source tools that have rapidly become the go-to solutions for monitoring and visualization in the DevOps ecosystem. These tools not only help monitor infrastructure in real-time but also contribute significantly to the Software Development Life Cycle (SDLC) by enhancing the feedback loop, improving visibility, and boosting the efficiency of Continuous Integration/Continuous Deployment (CI/CD) pipelines.

In this article, we'll dive into why these tools are critical for DevOps and how they improve the SDLC.

What is Prometheus?

Prometheus is an open-source systems monitoring and alerting toolkit, originally developed by SoundCloud and now maintained by the Cloud Native Computing Foundation (CNCF). It was designed to handle modern, dynamic, and cloud-native infrastructures, making it an ideal choice for monitoring microservices and Kubernetes-based environments.

Key Features of Prometheus:

Real-time Metrics Collection: Prometheus scrapes metrics from endpoints at specified intervals, enabling real-time monitoring of various systems and services. These metrics are then stored in a time-series database, optimized for high performance and scalability.
Service Discovery: Prometheus can automatically discover services using different service discovery mechanisms like Kubernetes, Consul, and others, making it highly suitable for dynamic environments.
Flexible Query Language (PromQL): Prometheus uses a powerful query language called PromQL to aggregate and analyze metrics. You can create complex queries to extract detailed insights from your metrics.
Multi-dimensional Data Model: Prometheus allows metrics to be enriched with labels (key-value pairs), enabling more detailed filtering and aggregation of metrics based on various dimensions like service name, region, instance type, etc.
Alerting: Prometheus integrates with the Alertmanager, enabling teams to configure highly flexible alerting rules. You can trigger alerts based on custom-defined thresholds or unexpected patterns and send notifications through various channels (Slack, PagerDuty, email, etc.).

What is Grafana?

Grafana is a powerful open-source analytics and visualization tool that allows teams to query, visualize, and alert on metrics from multiple sources. While it supports many data sources like Prometheus, Elasticsearch, InfluxDB, and more, it’s particularly known for its seamless integration with Prometheus.

Key Features of Grafana:

Customizable Dashboards: Grafana excels at creating interactive and customizable dashboards. You can visualize metrics in different formats such as time-series graphs, heatmaps, gauges, and more, giving you full control over how your data is presented.
Data Source Agnostic: Grafana can pull in data from multiple data sources and visualize them on the same dashboard. This allows teams to correlate metrics from Prometheus with data from other systems like logs, traces, or business analytics tools.
Alerting and Notifications: Grafana provides advanced alerting capabilities. You can define alert rules and conditions directly from your dashboards and get notified when metrics go out of bounds.
User-friendly Interface: Grafana’s intuitive interface makes it easy for teams to create complex visualizations and share them across the organization. Its wide range of plugins allows for easy customization and integration with other systems.
Templated Dashboards: Grafana’s templating feature allows you to create reusable, dynamic dashboards that can automatically adapt to different environments, services, or teams.

The Role of Monitoring in the DevOps Lifecycle

In a DevOps environment, monitoring plays a crucial role at every stage of the Software Development Life Cycle (SDLC). Let’s break down how Prometheus and Grafana fit into the different stages of the SDLC.

1. Development and Integration

During development, real-time monitoring helps developers understand the impact of code changes on application performance. By integrating Prometheus into your CI/CD pipeline, you can capture metrics from pre-production environments, such as staging or testing environments, and use Grafana dashboards to visualize them.

Benefit: Developers can catch performance issues early in the SDLC and optimize code before pushing it to production.
Example: Monitoring memory usage, API latency, or database query response times during testing can help identify bottlenecks before the software reaches production.

2. Testing and QA

In the testing phase, observability is crucial to ensure that the system behaves as expected under various loads or edge cases. Prometheus can be configured to collect metrics during automated test runs, and Grafana can visualize the data for easy analysis.

Benefit: Teams can validate that the system meets performance benchmarks and remains resilient under stress tests.
Example: By using Prometheus to monitor CPU, memory, and network usage during load testing, teams can see how their system scales under different levels of demand.

3. Deployment

When code is deployed to production, monitoring becomes vital to track the health and performance of the system. Prometheus can provide real-time metrics on service health, error rates, and request latency. Grafana dashboards can give DevOps teams a comprehensive view of the system’s health during and after deployments.

Benefit: Real-time feedback allows for faster rollbacks or corrective measures in case of deployment failures or performance degradation.
Example: Setting up alerts for increased response time or a high rate of 5xx errors ensures that issues are detected immediately after deployment.

4. Operations and Maintenance

Post-deployment, continuous monitoring ensures that the application remains healthy and performant in production. Prometheus and Grafana can provide historical data, enabling teams to identify trends and prevent issues from escalating.

Benefit: By monitoring system behavior over time, teams can detect patterns (e.g., memory leaks, increasing response times) and plan for proactive maintenance.
Example: Grafana can visualize long-term trends of CPU utilization, helping teams predict when infrastructure scaling is necessary.

5. Incident Management

When issues arise in production, Prometheus and Grafana play a critical role in diagnosing the root cause and mitigating the problem. Prometheus’ alerting system can notify teams about potential outages or performance bottlenecks, and Grafana dashboards can be used to analyze the metrics in real-time.

Benefit: Faster incident detection and resolution reduce downtime and maintain service reliability.
Example: When an alert is triggered due to high response latency, Grafana dashboards can help teams drill down into specific microservices or nodes that are underperforming.

Prometheus + Grafana: Driving Continuous Improvement in DevOps

By providing real-time insights, historical analysis, and proactive alerting, Prometheus and Grafana empower DevOps teams to continuously improve their systems and processes. Here’s how they drive continuous improvement across the SDLC:

1. Faster Feedback Loops

In DevOps, continuous feedback is essential for improving software quality and performance. Prometheus enables teams to gather metrics at every stage of development, while Grafana provides an intuitive way to visualize and analyze these metrics. This combination allows for faster iteration, ensuring that teams can quickly adapt to changing requirements or fix issues as they arise.

2. Improved Collaboration Between Development and Operations

Prometheus and Grafana act as a bridge between development and operations teams by providing a shared view of the system’s performance. Developers can see how their code changes affect production, while operations teams can track infrastructure metrics. This visibility fosters better communication and collaboration, breaking down traditional silos between the two teams.

3. Data-Driven Decision Making

The real-time and historical data provided by Prometheus and Grafana enables teams to make informed decisions. Whether it's deciding when to scale infrastructure, identifying the root cause of a performance issue, or planning the next deployment, data-driven insights lead to more confident decision-making.

4. Proactive Problem Solving

With Prometheus’ powerful alerting system and Grafana’s ability to track trends over time, teams can identify potential issues before they become critical. This allows for proactive problem solving, reducing the likelihood of unplanned downtime and improving system resilience.

Conclusion

In the DevOps world, where speed, reliability, and continuous improvement are key, integrating Prometheus and Grafana into your monitoring strategy is a game-changer. These tools not only provide deep visibility into system performance but also enhance collaboration, streamline incident response, and improve the overall Software Development Life Cycle (SDLC).

By embracing Prometheus and Grafana, your team can move beyond reactive troubleshooting and toward a more proactive, data-driven approach to software development and operations.

Are you ready to take your DevOps monitoring to the next level?

Understanding Terraform: A Guide to Effective IaC Practices

Hassan Aftab — Wed, 12 Jun 2024 22:38:45 +0000

What is Terraform?

Terraform is an infrastructure as code (IaC) tool that allows you to build, change, and version cloud and on-premises resources safely and efficiently.

With Terraform, you define your infrastructure using human-readable configuration files, which can be versioned, reused, and shared.

It works with a wide range of platforms and services through their APIs, enabling you to manage both low-level components (such as compute instances, storage, and networking) in a consistent manner.

The 3 Stage Workflow:

The Coding Stage:

Define resources across one or multiple cloud providers and services in your configuration files, depending on your requirements.

Here is a sample project structure:

.
├── bicep
│   ├── deploy.ps1
│   ├── init.bicep
│   ├── params
│   │   ├── dev.bicepparam
│   │   └── test.bicepparam
│   └── storage.bicep
├── LICENSE
├── Makefile
├── README.md
└── terraform
    ├── modules
    │   ├── container_app
    │   │   ├── main.tf
    │   │   ├── outputs.tf
    │   │   └── variables.tf
    │   ├── container_app_environment
    │   │   ├── main.tf
    │   │   ├── outputs.tf
    │   │   └── variables.tf
    │   ├── container_registry
    │   │   ├── main.tf
    │   │   ├── outputs.tf
    │   │   └── variables.tf
    │   ├── resource_group
    │   │   ├── main.tf
    │   │   ├── outputs.tf
    │   │   └── variables.tf
    │   ├── subnet
    │   │   ├── main.tf
    │   │   ├── outputs.tf
    │   │   └── variables.tf
    │   ├── subnet_network_security_group_association
    │   │   ├── main.tf
    │   │   ├── outputs.tf
    │   │   └── variables.tf
    │   └── virtual_network
    │       ├── main.tf
    │       ├── outputs.tf
    │       └── variables.tf
    └── resources
        ├── backend.tf
        ├── data.tf
        ├── main.tf
        ├── outputs.tf
        ├── provider.tf
        ├── tfvars
        │   ├── dev.tfvars
        │   ├── eun_region.tfvars
        │   └── tags.tfvars
        └── variables.tf

You can however, code the entire thing in a single file if you want, but as we all know, it is considered as a best practice, to adhere to separation of concerns

Lets breakdown the project structure:

directories:
- bicep
- terraform
- terraform/module
- terraform/resources

files:
- all files in directories inside terraform/modules/ contain modules for individual resources

    # backend.tf
    # Here we define the provider to use for this directory

    terraform {
    backend "azurerm" {
        storage_account_name = "storageAccountName"
        container_name       = "tfstates"
        resource_group_name  = "resourceGroupName"
        key                  = "resources.tfstate"
    }
    }

    # data.tf
    # Here we define the data sources to use for this directory

    data "terraform_remote_state" "resources" {
    backend = "azurerm"

        config = {
            storage_account_name = "storageAccountName"
            container_name       = "tfstates"
            resource_group_name  = "resourceGroupName"
            key                  = "resources.tfstate"
        }
    }

    # In this case, we use the data source to get the existing resource group

    data "azurerm_resource_group" "existing" {
        name = "resourceGroupName"
    }

As shown above, in the directory structure, the modules are defined in the terraform/modules directory and the resources are defined in the terraform/resources directory. The main codebase of this project resides in the terraform/resources/main.tf file.

Main things to note in the terraform/resources/main.tf file:

source - defines the module to use

module - defines the resources to create

The use of data.azurerm_resource_group.existing.location as well as data.azurerm_resource_group.existing.name to get the location and name of the resource group

The use of depend_on to ensure that the resources are created before the module is executed

Notice the use of $(acrServer) and $(acrUsername) and $(acrPassword) in the container_registry_server, container_registry_username and container_registry_password respectively.

These variables are defined in Pipelines. Since this information is sensitive, we are hiding it in the codebase and storing these secrets in pipeline variable groups/secrets

Let's take a look at the contents below:

    # main.tf
    # Here we define the resources to use for this project

    # Defining the network security group

    module "network_security_group" {
      source              = "../modules/network_security_group"
      name                = "project${var.environment}nsg"
      location            = data.azurerm_resource_group.existing.location
      resource_group_name = data.azurerm_resource_group.existing.name

      rules = [
        {
          name                       = "nsg-rule-1"
          priority                   = 100
          direction                  = "Inbound"
          access                     = "Allow"
          protocol                   = "*"
          source_port_range          = "*"
          destination_port_range     = "*"
          source_address_prefix      = "*"
          destination_address_prefix = "*"
        },
        {
          name                       = "nsg-rule-2"
          priority                   = 101
          direction                  = "Outbound"
          access                     = "Allow"
          protocol                   = "*"
          source_port_range          = "*"
          destination_port_range     = "*"
          source_address_prefix      = "*"
          destination_address_prefix = "*"
        }
      ]
      depends_on = [data.azurerm_resource_group.existing]
      tags       = merge(var.tags)
    }

    # Defining the virtual network to use in resources

    module "virtual_network" {
      source              = "../modules/virtual_network"
      name                = "project${var.environment}vnet"
      location            = data.azurerm_resource_group.existing.location
      resource_group_name = data.azurerm_resource_group.existing.name
      address_space       = ["10.0.0.0/16"]
      depends_on          = [data.azurerm_resource_group.existing, module.network_security_group.this]
      tags                = merge(var.tags)
    }

    # Defining the subnet that will be used to create resources under, later on.

    module "subnet" {
      source                     = "../modules/subnet"
      name                       = "project${var.environment}subnet"
      resource_group_name        = data.azurerm_resource_group.existing.name
      virtual_network_name       = module.virtual_network.virtual_network_name
      subnet_address_prefix      = ["10.0.1.0/24"]
      service_endpoints          = ["Microsoft.Storage", "Microsoft.Web"]
      delegation_name            = "delegation"
      service_delegation_name    = "Microsoft.App/environments"
      service_delegation_actions = ["Microsoft.Network/virtualNetworks/subnets/join/action", "Microsoft.Network/virtualNetworks/subnets/prepareNetworkPolicies/action"]
      depends_on                 = [data.azurerm_resource_group.existing, module.virtual_network.this, module.network_security_group.this]
    }

    # Defining the container app environment, notice the use of module.subnet.subnet_id , this is how we can reference the subnet_id from the subnet module.

    module "container_app_environment" {
      source                         = "../modules/container_app_environment"
      resource_group_name            = data.azurerm_resource_group.existing.name
      location                       = data.azurerm_resource_group.existing.location
      name                           = "project-${var.environment}-cntr-env"
      log_analytics_workspace_id     = module.log_analytics_workspace.log_analytics_workspace_id
      infrastructure_subnet_id       = module.subnet.subnet_id
      internal_load_balancer_enabled = false
      depends_on                     = [data.azurerm_resource_group.existing, module.subnet.this]
      tags                           = merge(var.tags)
    }

    # Defining the container registry

    module "container_registry" {
      source                           = "../modules/container_registry"
      resource_group_name              = data.azurerm_resource_group.existing.name
      location                         = data.azurerm_resource_group.existing.location
      name                             = "project${var.environment}cr"
      sku                              = "Standard"
      is_admin_enabled                 = true
      is_public_network_access_enabled = true
      depends_on                       = [data.azurerm_resource_group.existing, module.key_vault.this]
      tags                             = merge(var.tags)
    }

    # Defining the container apps that will be created under the container app environment created earlier

    module "container_app" {
      source                       = "../modules/container_app"
      resource_group_name          = data.azurerm_resource_group.existing.name
      container_app_environment_id = module.container_app_environment.Environment_ID
      container_registry_server    = "$(acrServer)"
      container_registry_username  = "$(acrUsername)"
      container_registry_password  = "$(acrPassword)"
      container_apps = [

        # Notice the use of $(containerAppSecretKey) and $(containerAppSecretValue) in the secret_name and secret_value respectively

        {
          name                       = "containerapp1-${var.environment}"
          image                      = "mcr.microsoft.com/azuredocs/containerapps-helloworld:latest"
          cpu                        = 0.25
          memory                     = "0.5Gi"
          target_port                = 8080
          transport                  = "http2"
          external_enabled           = true
          allow_insecure_connections = false
          secret_name                = "$(containerAppSecretKey)"
          secret_value               = "$(containerAppSecretValue)"
        },
        {
          name                       = "containerapp2-${var.environment}"
          image                      = "mcr.microsoft.com/azuredocs/containerapps-helloworld:latest"
          cpu                        = 0.25
          memory                     = "0.5Gi"
          target_port                = 8080
          transport                  = "auto"
          external_enabled           = true
          allow_insecure_connections = false
          secret_name                = "$(containerAppSecretKey)"
          secret_value               = "$(containerAppSecretValue)"
        }
      ]
      depends_on = [data.azurerm_resource_group.existing, module.container_app_environment.this, module.container_registry.this]
      tags       = merge(var.tags)
    }

    # Defining the network security group association

    module "subnet_nsg_association" {
      source                    = "../modules/subnet_network_security_group_association"
      subnet_id                 = module.subnet.subnet_id
      network_security_group_id = module.network_security_group.id
      depends_on                = [data.azurerm_resource_group.existing, module.subnet.this, module.network_security_group.this]
    }

In the block below, are the contents of output.tf, which contains all the outputs we want to get when the terraform code is run in the terminal / pipeline.

This can include details such as IPs of services being created, fqdns, etc.

One thing to keep in mind is, since we are using a modular based approach, these outputs must first be exported from an output.tf file inside the module itself, before the implementation of the module that actually outputs it during the run.

    # outputs.tf
    # Here we define the outputs to use for this directory

    # Container App Environment

    output "container_app_environment_default_domain" {
      value = module.container_app_environment.Default_Domain
    }

    output "container_app_environment_docker_bridge" {
      value = module.container_app_environment.Docker_Bridge_CIDR
    }

    output "container_app_environment_environment_id" {
      value = module.container_app_environment.Environment_ID
    }

    output "container_app_environment_static_ip_address" {
      value = module.container_app_environment.Static_IP_Address
    }

    # Container Apps

    output "container_app_latest_fqdn" {
      value = module.container_app.Latest_Revision_Fqdn
    }

    output "container_app_outbound_ips" {
      value = module.container_app.Outbound_Ip_Addresses
    }

    # Container Registry

    output "container_registry_id" {
      value = module.container_registry.id
    }

    output "container_registry_sku" {
      value = module.container_registry.sku
    }

    output "container_registry_registry_server" {
      value = module.container_registry.registry_server
    }

    output "container_registry_admin_enabled" {
      value = module.container_registry.admin_enabled
    }

    output "container_registry_admin_username" {
      value     = module.container_registry.admin_username
      sensitive = true
    }

    output "container_registry_admin_password" {
      value     = module.container_registry.admin_password
      sensitive = true
    }

In the next block, are the contents of provider.tf

We have used skip_provider_registration = true to skip the provider registration, as sometimes during a Pipeline run, it can causes issues if terraform checks for registered providers.

Furthermore, here we define the minimum version of the provider we are using as well as the required version of the Terraform CLI.

    # provider.tf
    # Here we define the providers to use for this directory

    provider "azurerm" {
      features {}
      skip_provider_registration = true
    }

    terraform {
      required_version = ">= 1.7.5"
      required_providers {
        azurerm = {
          source  = "hashicorp/azurerm"
          version = ">=3.96.0"
        }
      }
    }

In the next block, are the contents of tfvars/dev.tf
As discussed before, sensitive values are stored in pipeline variable groups/secrets. In this case sql_administrator_login and sql_administrator_login_password.

    # tfvars/dev.tf
    # This file defines the variables to use for this project for the dev environment

    project_name                 = "Project"
    environment                  = "dev"
    administrator_login          = "$(sql_administrator_login)"
    administrator_login_password = "$(sql_administrator_login_password)"

Similarly:

    # tfvars/eun_region.tf

    region_name  = "northeurope"
    region_short = "eun"

In the last variables file, are the contents of tfvars/tags.tf, which defines the tags to be applied to the resources. Back in the main.tf file, we used this tags = merge(var.tags) key-value pair approach to define the tags.

    # tfvars/tags.tf

    tags = {
        ServiceName    = ""
        Department     = "Cloud"
        Environment    = "dev"
        SubEnvironment = "nonProd"
        SystemName     = ""
    }

And finally the variables file that contains the variables and their types

    # variables.tf

    variable "region_short" {
      type        = string
      description = "Short name of region used in project"
    }

    variable "region_name" {
      type        = string
      description = "Long name of region used in project"
    }

    variable "project_name" {
      description = "Project name"
    }

    variable "environment" {
      type        = string
      description = "Environment name"
    }

    variable "tags" {
      type = map(string)
      default = {
        ServiceName    = ""
        Department     = ""
        Environment    = ""
        SubEnvironment = ""
        SystemName     = ""
      }
    }

    variable "administrator_login" {
      type        = string
      description = "Administrator login"
      sensitive   = true
    }

    variable "administrator_login_password" {
      type        = string
      description = "Administrator login password"
      sensitive   = true
    }

The Plan Stage

In this stage, we are done with defining Infrastructure as Code configurations, now we need Terraform to generate an execution plan based on these configurations and existing infrastructure, describing the changes it will make.

Before we go ahead and generate a plan, it is considered a good practice to make sure your code is valid and there are no syntax errors, or referencing errors.

This can be done by switching to your trusty CLI.. again, and running:


# This command validates your code and makes sure it's good to go

terraform validate

# And it's just as easy to make your code look much more cleaner with just one command again

terraform fmt --recursive

This terraform fmt --recursive command formats the current directory as well the child directories and all .tf and .tfvars files, and properly indents all code.

Below we can see our code is valid!

Finally we can generate a plan by running a simple command

# Command to generate plan

terraform plan -o dev.tfplan

# In our case, we are using tfvars file in a directory called tfvars/, now we will need to modify the command a little bit to get the same result

terraform plan -var-file=tfvars/dev.tfvars -var-file=tfvars/eun_region.tfvars -var-file=tfvars/tags.tfvars -out=dev.tfplan

Sample output can such as:


# module.subnet.azurerm_subnet.this will be updated in-place
  ~ resource "azurerm_subnet" "this" {
        id                                             = "/subscriptions/GUID_HERE/resourceGroups/project-dev/providers/Microsoft.Network/virtualNetworks/project-vnet-dev/subnets/project-subnet-dev"
        name                                           = "project-subnet-dev"
        # (10 unchanged attributes hidden)

      ~ delegation {
            name = "project-subnet-delegation-dev"

          ~ service_delegation {
              ~ actions = [
                    "Microsoft.Network/virtualNetworks/subnets/join/action",
                  + "Microsoft.Network/virtualNetworks/subnets/prepareNetworkPolicies/action",
                ]
                name    = "Microsoft.App/environments"
            }
        }
    }

Plan: 1 to add, 1 to change, 0 to destroy.

Changes to Outputs:
  ~ storage_account_name                        = "projectblobdev" -> "project1blobdev"
  + storage_account_primary_key                 = (sensitive value)

These changes are basically compared with a .tfstate file that can exist in your local machine or on the cloud, hosted in a s3 bucket or blob storage.

In our example above, we used a blob storage service to host the tfstate file.

The Apply Stage

Upon approval of the plan generated in the last step, Terraform applies the proposed changes in the correct order, respecting resource dependencies.


# Apply command for deploying the infrastructure
terraform apply dev.tfplan

This will then finally, deploy your infrastructure to the cloud. You can however, if needed, destroy the infrastructure when done with your usecase.


# Destroy it all by:

terraform destroy -var-file=tfvars/dev.tfvars -var-file=tfvars/eun_region.tfvars -var-file=tfvars/tags.tfvars

All in all, It’s a powerful tool for managing infrastructure, allowing you to track changes, maintain consistency, and avoid manual errors.

And that's not all, at the same time it also ensures, controlled costs in cloud infrastructure, since you can easily create and destroy infrastructure with simple commands.

This automation can be taken further with Pipelines and gitflow that triggers based on branch that reflects a certain environment.. but that's a topic for another day :D

You can find the source code by clicking Here

I hope this article was a fun read and helped you gain some deeper insights into terraform modules and best practices.

Thankyou for the read!

DEV Community: Hassan Aftab

AI-Powered Penetration Testing: How I Used Claude + Kali Linux MCP to Automate Security Assessments

Introduction: The Future of Offensive Security is Conversational

The Problem with Traditional Pen Testing

Enter AI-Assisted Security Testing

The Technology Stack

1. Claude Desktop

2. Kali Linux MCP Server

3. Traditional Security Tools

The Assessment: A Real-World Example

Phase 1: Reconnaissance

Phase 2: Security Header Analysis

Phase 3: Directory Enumeration

Phase 4: Deep Dive on Findings

Phase 5: Comprehensive Reporting

Real Findings (Anonymized)

🟢 Strong Security Controls (Good News)

🟡 Medium Priority Issues

🟢 Low Priority Observations

The Real Value: Beyond Tool Execution

1. Intelligent Analysis

2. Contextual Reasoning

3. Adaptive Methodology

4. Knowledge Transfer

Practical Applications

1. Continuous Security Testing

2. Compliance Audits

3. Security Training

4. Bug Bounty Hunting

5. Red Team Exercises

The Limitations (Let's Be Honest)

❌ Complex Exploitation

❌ Social Engineering

❌ Zero-Day Discovery

❌ Replace Critical Thinking

❌ Handle Authentication

Ethical Considerations

Setting It Up Yourself

Prerequisites

Quick Start: Follow this Guide

The Future of Security Testing

Short Term (Now - 6 months)

Medium Term (6-18 months)

Long Term (18+ months)

My Take: Augmentation, Not Replacement

Conclusion: The Paradigm Shift

Resources & Further Reading

About This Article

App of Apps Pattern: Efficient Kubernetes Deployment Strategies with Terraform and ArgoCD

Table of Contents

1. Introduction

2. Traditional Infrastructure Challenges

3. Modern Infrastructure Approach

4. Terraform: Infrastructure Provisioning

5. ArgoCD Configuration

6. GitOps Workflow

Repository Structures

Implementation Strategies

7. GitOps Philosophical Principles

Core Concepts

Organizational Benefits

8. Practical Implementation Strategy

9. Conclusion

10. Recommended Tools And Resources

Introduction

The Traditional Infrastructure Challenge

Enter the Modern Approach Infrastructure as Code and GitOps

Terraform Infrastructure Provisioning Reimagined

Terraform Kubernetes Cluster Provisioning Example

ArgoCD Initial Setup and Configuration Guide

1. Password Reset and Retrieval

# Command Breakdown

# What This Does

2. RBAC (Role-Based Access Control) Configuration

# Edit RBAC ConfigMap

# RBAC Policy Configuration

# Policy Explanation

3. Restart ArgoCD Server

4. Login to ArgoCD

Important Notes