DEV Community

Cover image for Building Secure Jenkins-Slack Integration with AWS Lambda - Part 1: Complete Setup Guide
AWS Community Builders profile image Sri
Sri for AWS Community Builders

Posted on

Building Secure Jenkins-Slack Integration with AWS Lambda - Part 1: Complete Setup Guide

#aws #terraform #slack #serverless

Building Secure Jenkins-Slack Integration with AWS Lambda - Part 1: Complete Setup Guide

Ever wished you could trigger your CI/CD pipelines directly from Slack without exposing Jenkins to the public internet? In this comprehensive two-part series, we'll build a production-ready solution that lets your team trigger Jenkins jobs using Slack slash commands while maintaining enterprise-grade security.

This is Part 1 of our series, focusing on architecture design, infrastructure setup, and initial implementation. Part 2 will dive deep into troubleshooting real-world issues and production enhancements.

What We'll Build

This solution enables teams to trigger Jenkins CI/CD pipelines from Slack using a secure, serverless architecture. The key innovation is keeping Jenkins completely private while providing a public API endpoint through AWS Lambda and API Gateway.

Tech Stack Overview

  • Infrastructure: Terraform (IaC), AWS VPC, EC2, Lambda, API Gateway, ALB, NAT Gateway
  • Backend: Go 1.20 (Lambda functions), Jenkins (CI/CD orchestration)
  • Storage/Secrets: AWS SSM Parameter Store
  • Integration: Slack API (Slash Commands, Webhooks)
  • Monitoring: CloudWatch Logs
  • OS: Amazon Linux 2

Architecture Deep Dive

Let's start with the complete architecture flow:

┌─────────────────────────────────────────────────────────────────┐
│                        Architecture Flow                        │
└─────────────────────────────────────────────────────────────────┘

👥 Slack User                    🌐 API Gateway
/run_test environment=staging ──→ /prod/trigger
                                        │
                                        ▼
                                ⚡ AWS Lambda
                                jenkins-trigger
                                        │
                    ┌───────────────────┼───────────────────┐
                    │                   │                   │
                    ▼                   ▼                   ▼
            🔐 AWS SSM Parameter    🔧 Jenkins (Private EC2)    🛡️ Bastion Host
            Store                   CI/CD Orchestration        SSH Access Gateway
            Jenkins URL &           ◄─────────────────────────┘
            Credentials             │
                    │               │
                    └───────────────┼───────────────────────────┐
                                    │                           │
                                    ▼                           ▼
                            ⚖️ Internal ALB              Job Status
                            Load Balancer                Notifications
                                    │                           │
                                    └───────────────────────────┘
                                                              │
                                                              ▼
                                                        👥 Slack User
                                                        (Status Updates)

Legend:
🟠 AWS Services (API Gateway, Lambda, SSM, ALB, Bastion)
🟣 Slack Integration
🔴 Jenkins (Private EC2)
Enter fullscreen mode Exit fullscreen mode

Key Components Explained

🔒 VPC with Public/Private Subnets: Jenkins runs in a private subnet with no internet access, while the bastion host provides secure SSH access from the public subnet.

⚡ Lambda Function in VPC: The Lambda function runs in the private subnet to communicate with Jenkins while being invoked by API Gateway from the internet.

🌐 API Gateway: Provides a public HTTPS endpoint that Slack can reach, while the Lambda function handles the secure communication with Jenkins.

🔐 SSM Parameter Store: Stores Jenkins credentials and configuration securely, eliminating the need for hardcoded secrets.

🛡️ Bastion Host: Provides secure SSH access to Jenkins for administrative tasks without exposing Jenkins directly.

⚖️ Internal Load Balancer: Distributes traffic to Jenkins instances and provides health checks.

Security Benefits

  • Zero Public Exposure: Jenkins never faces the internet
  • Encrypted Secrets: All credentials stored in SSM with encryption
  • Network Isolation: Private subnets with controlled access
  • Least Privilege: IAM roles with minimal required permissions

Prerequisites

Before we start, ensure you have:

  • AWS account with appropriate permissions (EC2, Lambda, API Gateway, SSM, VPC)
  • Terraform >= 1.0.0 installed
  • AWS CLI configured with credentials
  • Go 1.20+ installed (for Lambda function builds)
  • Slack workspace with admin privileges
  • SSH key pair for EC2 access
  • Basic understanding of Jenkins, Lambda, and Terraform

Step 0: Initial Setup

Configure AWS CLI

First, configure your AWS CLI with proper credentials:

# Configure AWS CLI
aws configure

# Enter your AWS Access Key ID, Secret Access Key, region (us-east-1), and output format (json)
# Verify configuration
aws sts get-caller-identity
Enter fullscreen mode Exit fullscreen mode

Create SSH Key Pair

Create an EC2 key pair for SSH access:

# Option 1: Create via AWS CLI
aws ec2 create-key-pair --key-name jenkins-slack-key --query 'KeyMaterial' --output text > jenkins-slack-key.pem
chmod 400 jenkins-slack-key.pem

# Option 2: Create via AWS Console
# Go to EC2 → Key Pairs → Create Key Pair → Name: jenkins-slack-key → Download .pem file
Enter fullscreen mode Exit fullscreen mode

Step-by-Step Implementation

Step 1: Clone and Setup 

git clone https://github.com/kasukur/jenkins-slack-aws-integration
cd jenkins-slack-aws-integration
Enter fullscreen mode Exit fullscreen mode

Step 2: Build Lambda Functions

Build the Lambda function binaries:

# Navigate to Lambda function directories
cd lambda/jenkins_trigger
go mod tidy
GOOS=linux GOARCH=amd64 go build -o bootstrap main.go
zip jenkins_trigger.zip bootstrap

cd ../slack_verification
go mod tidy
GOOS=linux GOARCH=amd64 go build -o bootstrap main.go
zip slack_verification.zip bootstrap

# Copy zips to terraform directory
cp jenkins_trigger.zip ../../terraform/
cp slack_verification.zip ../../terraform/
Enter fullscreen mode Exit fullscreen mode

The repository structure:

├── terraform/           # Infrastructure as Code
│   ├── main.tf         # Provider and key pair
│   ├── vpc.tf          # VPC, subnets, security groups
│   ├── jenkins.tf      # Jenkins EC2 and ALB
│   ├── lambda.tf       # Lambda functions
│   ├── api_gateway.tf  # API Gateway configuration
│   ├── iam.tf          # IAM roles and policies
│   └── variables.tf    # Variable definitions
├── lambda/             # Lambda function source code
│   ├── jenkins_trigger/    # Main Lambda function
│   └── slack_verification/ # Slack signature verification
└── jenkins/            # Jenkins pipeline definitions
    └── Jenkinsfile     # Test pipeline configuration
Enter fullscreen mode Exit fullscreen mode

Step 3: Configure Slack App

Create Slack App

  1. Go to Slack API: https://api.slack.com/apps
  2. Click "Create New App" → "From scratch"
  3. App Name: Jenkins CI/CD Integration
  4. Workspace: Select your workspace
  5. Click "Create App"

Configure Slash Command

  1. In your app settings, go to "Slash Commands"
  2. Click "Create New Command"
  3. Fill in the details:
    • Command: /run_test
    • Request URL: https://your-api-gateway-url/prod/trigger (we'll get this after deployment)
    • Short Description: Trigger Jenkins test pipeline
    • Usage Hint: [environment] [test_suite]
  4. Click "Save"

Enable Incoming Webhooks

  1. Go to "Incoming Webhooks" in your app settings
  2. Toggle "Activate Incoming Webhooks" to On
  3. Click "Add New Webhook to Workspace"
  4. Select a channel (e.g., #devops)
  5. Click "Allow"
  6. Copy the webhook URL (starts with https://hooks.slack.com/services/)

Get App Credentials

  1. Go to "Basic Information" in your app settings
  2. In "App Credentials" section, copy:
    • App-Level Token (starts with xapp-)
    • Signing Secret (long random string)

Install App to Workspace

  1. Go to "Install App" in your app settings
  2. Click "Install to Workspace"
  3. Review permissions and click "Allow"
  4. Copy the Bot User OAuth Token (starts with xoxb-)

Save these values for the Terraform configuration:

  • App-Level Token (xapp-...)
  • Signing Secret
  • Webhook URL
  • Bot User OAuth Token (xoxb-...)

Step 4: Prepare Terraform Variables

Copy the example configuration:

cp terraform/terraform.tfvars.example terraform/terraform.tfvars
Enter fullscreen mode Exit fullscreen mode

Edit terraform/terraform.tfvars with your values:

# Basic Configuration
aws_region = "us-east-1"
project_name = "jenkins-slack-demo"

# Network Configuration
vpc_cidr = "10.0.0.0/16"
public_subnet_cidrs = ["10.0.1.0/24", "10.0.2.0/24"]
private_subnet_cidrs = ["10.0.101.0/24", "10.0.102.0/24"]

# SSH Access
key_name = "your-key-pair-name"
public_key = "ssh-rsa AAAAB3NzaC1yc2E... your-email@example.com"

# Jenkins Configuration
jenkins_admin_username = "admin"
jenkins_admin_password = "your-secure-password"

# Slack Configuration
slack_app_token = "xapp-your-app-token"
slack_signing_secret = "your-signing-secret"
slack_webhook_url = "https://hooks.slack.com/services/your/webhook/url"
slack_bot_token = "xoxb-your-bot-token"
slack_channel = "#devops"
Enter fullscreen mode Exit fullscreen mode

Step 5: Deploy Infrastructure 

cd terraform

# Initialize Terraform
terraform init

# Review the plan (should show ~25 resources)
terraform plan

# Deploy the infrastructure
terraform apply
Enter fullscreen mode Exit fullscreen mode

Important: Save the outputs displayed after successful deployment:

api_gateway_url = "https://<your-api-gateway-id>.execute-api.<region>.amazonaws.com/prod/trigger"
bastion_public_ip = "<bastion-public-ip>"
jenkins_url = "http://<internal-alb-url>"
Enter fullscreen mode Exit fullscreen mode

Step 6: Update Slack Configuration

  1. Go back to your Slack app settings
  2. Update the slash command Request URL with the API Gateway URL from Step 5
  3. Save the changes

Step 7: Configure Jenkins

Access Jenkins

Set up SSH tunnel through the bastion host:

# SSH to bastion host with port forwarding
ssh -i your-key.pem -L 8080:<jenkins_private_ip>:8080 ec2-user@<bastion_public_ip> -N

# In another terminal, access Jenkins via localhost
# Open browser: http://localhost:8080
Enter fullscreen mode Exit fullscreen mode

Initial Jenkins Setup

Wait for Jenkins initialization (5-10 minutes), then:

  1. Get the initial admin password: 
   # SSH to Jenkins instance
   ssh -i your-key.pem ec2-user@<bastion_public_ip>
   ssh ec2-user@<jenkins_private_ip>

   # Get initial password
   sudo cat /var/lib/jenkins/secrets/initialAdminPassword
Enter fullscreen mode Exit fullscreen mode
  1. Access Jenkins at http://localhost:8080
  2. Enter the initial admin password
  3. Install suggested plugins
  4. Create admin user:
    • Username: admin
    • Password: <your-secure-password> (use a strong password)
    • Full name: Administrator
    • Email: admin@yourcompany.com

Install Slack Plugin

  1. Go to: Manage Jenkins → Manage Plugins → Available
  2. Search for: "Slack Notification"
  3. Check the box and click "Install without restart"
  4. Wait for installation to complete
  5. Go to: Manage Jenkins → Configure System
  6. Find "Slack" section:
    • Workspace: Your workspace name
    • Default Channel: #devops (or your preferred channel)
    • Integration Token: Paste your Bot User OAuth Token (xoxb-...)
  7. Click "Test Connection" - should show "Success"
  8. Click "Save"

Create Pipeline Job

  1. Click "New Item"
  2. Enter item name: run_test
  3. Select "Pipeline" and click "OK"
  4. In Pipeline section, select "Pipeline script"
  5. Copy the content from jenkins/Jenkinsfile into the script area
  6. Click "Save"

Step 8: Update SSM Parameters

Verify the Jenkins URL in SSM Parameter Store:

aws ssm get-parameter --name "/jenkins-slack-demo/jenkins_url" --region us-east-1
Enter fullscreen mode Exit fullscreen mode

If needed, update it with the correct internal ALB URL:

aws ssm put-parameter \
  --name "/jenkins-slack-demo/jenkins_url" \
  --value "http://<internal-alb-url>" \
  --overwrite \
  --region us-east-1
Enter fullscreen mode Exit fullscreen mode

Verification and Testing

Test 1: Verify Infrastructure 

# Check EC2 instances
aws ec2 describe-instances --filters "Name=tag:Name,Values=jenkins-slack-demo-*" --region us-east-1

# Check Lambda functions
aws lambda list-functions --region us-east-1

# Check API Gateway
curl -X POST https://your-api-gateway-url/prod/trigger
Enter fullscreen mode Exit fullscreen mode

Test 2: Verify Lambda Function

Test the Lambda function directly:

# Test Lambda function
aws lambda invoke \
  --function-name jenkins-slack-demo-jenkins-trigger \
  --payload '{}' \
  /tmp/lambda-response.json && cat /tmp/lambda-response.json
Enter fullscreen mode Exit fullscreen mode

Test 3: Verify Jenkins Connectivity

Test Jenkins accessibility from Lambda:

# SSH to Jenkins and verify it's running
ssh -i your-key.pem ec2-user@<bastion_public_ip>
ssh ec2-user@<jenkins_private_ip>

# Check Jenkins status
sudo systemctl status jenkins
curl -u admin:<your-secure-password> http://localhost:8080/api/json
Enter fullscreen mode Exit fullscreen mode

Test 4: Test Slack Command

In any Slack channel where your app is installed, type:

/run_test
Enter fullscreen mode Exit fullscreen mode

Expected behavior:

  • Immediate response in Slack: "Job triggered successfully!"
  • Jenkins job appears in the dashboard
  • Job runs with default parameters (ENVIRONMENT=dev, TEST_SUITE=smoke)

Test 5: Verify Jenkins Job Execution

  1. Check Jenkins dashboard for the run_test job
  2. Verify job parameters (ENVIRONMENT=dev, TEST_SUITE=smoke)
  3. Check job logs for successful execution
  4. Confirm Slack receives success/failure notifications

Test 6: End-to-End Test with Parameters

Test with custom parameters:

/run_test environment=staging test_suite=regression
Enter fullscreen mode Exit fullscreen mode

Monitor the complete flow:

  1. Slack → API Gateway → Lambda
  2. Lambda → Jenkins (job trigger with parameters)
  3. Jenkins → Slack (status notification)

Key Code Components

Lambda Function (Go)

The Lambda function handles the core integration logic:

func handler(ctx context.Context, request events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) {
    // Get Jenkins URL and credentials from SSM
    jenkinsURL, err := getSSMParameter(ctx, jenkinsURLParam, false)
    credsJSON, err := getSSMParameter(ctx, jenkinsCredsParam, true)

    // Get CSRF crumb for Jenkins security
    crumbField, crumb, err := getJenkinsCrumb(client, jenkinsURL, creds.Username, creds.Password)

    // Trigger Jenkins job
    jobURL := fmt.Sprintf("%s/job/run_test/buildWithParameters", jenkinsURL)
    data := url.Values{}
    data.Set("ENVIRONMENT", "dev")
    data.Set("TEST_SUITE", "smoke")

    // Make authenticated request with crumb
    req.Header.Set("Jenkins-Crumb", crumb)
    req.SetBasicAuth(creds.Username, creds.Password)

    return events.APIGatewayProxyResponse{
        StatusCode: 200,
        Body:       `{"message": "Job triggered! Status: 201"}`,
    }, nil
}
Enter fullscreen mode Exit fullscreen mode

Jenkins Pipeline

The Jenkinsfile defines the test pipeline:

pipeline {
    agent any

    parameters {
        string(name: 'ENVIRONMENT', defaultValue: 'dev', description: 'Environment to run tests against')
        string(name: 'TEST_SUITE', defaultValue: 'smoke', description: 'Test suite to run')
    }

    stages {
        stage('Prepare') {
            steps {
                echo "Preparing to run tests in ${params.ENVIRONMENT} environment"
                slackSend(color: '#FFFF00', message: "STARTED: Job '${env.JOB_NAME} [${env.BUILD_NUMBER}]'")
            }
        }

        stage('Run Tests') {
            steps {
                echo "Running ${params.TEST_SUITE} tests against ${params.ENVIRONMENT}"
                // Your actual test commands here
            }
        }
    }

    post {
        success {
            slackSend(color: '#00FF00', message: "SUCCESS: Job completed successfully")
        }
        failure {
            slackSend(color: '#FF0000', message: "FAILED: Job failed")
        }
    }
}
Enter fullscreen mode Exit fullscreen mode

What's Next?

Congratulations! You've successfully built the foundation of a secure Jenkins-Slack integration. At this point, you should have:

✅ Working Infrastructure: VPC, Jenkins, Lambda, API Gateway

✅ Basic Integration: Slack can trigger Jenkins jobs

✅ Security: Private Jenkins with secure access patterns

✅ Monitoring: CloudWatch logs for observability

Coming Up in Part 2

In Part 2 of this series, we'll dive deep into:

🔧 Real-World Troubleshooting: Common issues you'll encounter and battle-tested solutions

🛠️ Production Enhancements: Security improvements, monitoring, and scaling considerations

🚨 Critical Fixes: Jenkins authentication issues, CSRF token problems, and Slack integration challenges

📊 Advanced Features: Parameter parsing, job status callbacks, and multi-environment support

Key Takeaways

This architecture demonstrates several important DevOps and AWS best practices:

✅ Security First: Jenkins is never exposed to the internet, following zero-trust principles

✅ Infrastructure as Code: Everything is defined in Terraform, making it reproducible and version-controlled

✅ Serverless Integration: Lambda provides cost-effective, scalable integration without managing servers

✅ Secret Management: SSM Parameter Store eliminates hardcoded credentials

✅ Network Isolation: VPC with public/private subnets provides proper network segmentation

✅ Observability: CloudWatch logs provide visibility into system behavior

Ready to tackle the real-world challenges? Check out Part 2 where we'll solve the tricky issues that make or break production deployments!

Have questions or suggestions for improvements? Feel free to reach out or contribute to the project repository!

Cover image by @dlxmedia.hu from unsplash

Top comments (0)