Advanced Testing Management Tools: Enterprise-Grade CI/CD Pipeline Examples

Abstract

Modern software development requires robust testing management tools that can handle complex enterprise requirements including microservices architecture, multi-cloud deployments, and comprehensive security scanning. This article explores advanced implementations of popular CI/CD tools with real-world enterprise examples, performance benchmarks, and production-ready configurations.

1. Azure DevOps Pipelines

Azure DevOps provides enterprise-grade CI/CD capabilities with excellent integration to Microsoft ecosystem and hybrid cloud environments.

Key Enterprise Features:

• Advanced approval workflows
• Multi-stage deployments
• Extensive reporting and analytics
• Enterprise security and compliance
• Hybrid cloud support

Real-World Enterprise Example:

# azure-pipelines.yml
trigger:
  branches:
    include:
      - main
      - develop
      - feature/*
  paths:
    exclude:
      - docs/*
      - README.md

variables:
  buildConfiguration: 'Release'
  dotNetFramework: 'net6.0'
  dotNetVersion: '6.0.x'
  vmImageName: 'ubuntu-latest'

stages:
- stage: Build
  displayName: Build and Test
  jobs:
  - job: Build
    displayName: Build
    pool:
      vmImage: $(vmImageName)

    steps:
    - task: UseDotNet@2
      displayName: 'Use .NET $(dotNetVersion)'
      inputs:
        version: '$(dotNetVersion)'
        includePreviewVersions: true

    - task: DotNetCoreCLI@2
      displayName: 'Restore packages'
      inputs:
        command: 'restore'
        projects: '**/*.csproj'
        feedsToUse: 'select'
        vstsFeed: 'MyCompany/MyFeed'

    - task: SonarCloudPrepare@1
      displayName: 'Prepare SonarCloud Analysis'
      inputs:
        SonarCloud: 'SonarCloud'
        organization: 'mycompany'
        scannerMode: 'MSBuild'
        projectKey: 'mycompany_myproject'
        projectName: 'MyProject'

    - task: DotNetCoreCLI@2
      displayName: 'Build solution'
      inputs:
        command: 'build'
        projects: '**/*.sln'
        arguments: '--configuration $(buildConfiguration) --no-restore'

    - task: DotNetCoreCLI@2
      displayName: 'Run Unit Tests'
      inputs:
        command: 'test'
        projects: '**/*Tests.csproj'
        arguments: '--configuration $(buildConfiguration) --no-build --collect:"XPlat Code Coverage" --logger trx --results-directory $(Agent.TempDirectory)'

    - task: PublishCodeCoverageResults@1
      displayName: 'Publish Code Coverage'
      inputs:
        codeCoverageTool: 'Cobertura'
        summaryFileLocation: '$(Agent.TempDirectory)/**/coverage.cobertura.xml'

    - task: SonarCloudAnalyze@1
      displayName: 'Run SonarCloud Analysis'

    - task: SonarCloudPublish@1
      displayName: 'Publish SonarCloud Results'
      inputs:
        pollingTimeoutSec: '300'

    - task: WhiteSource@21
      displayName: 'Run WhiteSource Security Scan'
      inputs:
        cwd: '$(Build.SourcesDirectory)'
        projectName: 'MyProject'

    - task: DotNetCoreCLI@2
      displayName: 'Publish Application'
      inputs:
        command: 'publish'
        publishWebProjects: false
        projects: '**/MyProject.csproj'
        arguments: '--configuration $(buildConfiguration) --output $(Build.ArtifactStagingDirectory)'
        zipAfterPublish: true

    - task: PublishBuildArtifacts@1
      displayName: 'Publish Artifacts'
      inputs:
        PathtoPublish: '$(Build.ArtifactStagingDirectory)'
        ArtifactName: 'drop'
        publishLocation: 'Container'

- stage: IntegrationTest
  displayName: Integration Testing
  dependsOn: Build
  condition: succeeded()
  jobs:
  - deployment: IntegrationTest
    displayName: Run Integration Tests
    environment: 'integration-testing'
    pool:
      vmImage: $(vmImageName)
    strategy:
      runOnce:
        deploy:
          steps:
          - task: DockerCompose@0
            displayName: 'Start Test Environment'
            inputs:
              action: 'Run services'
              dockerComposeFile: 'docker-compose.test.yml'
              detached: true

          - task: DotNetCoreCLI@2
            displayName: 'Run Integration Tests'
            inputs:
              command: 'test'
              projects: '**/*IntegrationTests.csproj'
              arguments: '--configuration $(buildConfiguration) --logger trx --results-directory $(Agent.TempDirectory)'

          - task: DockerCompose@0
            displayName: 'Stop Test Environment'
            inputs:
              action: 'Run services'
              dockerComposeFile: 'docker-compose.test.yml'
              dockerComposeCommand: 'down'
            condition: always()

- stage: SecurityScan
  displayName: Security Scanning
  dependsOn: Build
  condition: succeeded()
  jobs:
  - job: SecurityScan
    displayName: Security Analysis
    pool:
      vmImage: $(vmImageName)
    steps:
    - task: DownloadBuildArtifacts@0
      displayName: 'Download Artifacts'
      inputs:
        buildType: 'current'
        downloadType: 'single'
        artifactName: 'drop'
        downloadPath: '$(System.ArtifactsDirectory)'

    - task: ContainerScan@0
      displayName: 'Container Security Scan'
      inputs:
        dockerfilePath: 'Dockerfile'
        buildContext: '.'
        repository: 'mycompany/myproject'
        tag: '$(Build.BuildNumber)'

    - task: PublishSecurityAnalysisLogs@3
      displayName: 'Publish Security Analysis Logs'
      inputs:
        ArtifactName: 'CodeAnalysisLogs'
        ArtifactType: 'Container'
        AllTools: false
        APIScan: false
        BinSkim: false
        CodesignValidation: false
        CredScan: true
        MSRD: false
        RoslynAnalyzers: false
        SDLNativeRules: false
        Semmle: false
        TSLint: false
        ToolLogsNotFoundAction: 'Standard'

- stage: DeployDev
  displayName: Deploy to Development
  dependsOn: 
    - IntegrationTest
    - SecurityScan
  condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/develop'))
  jobs:
  - deployment: DeployDev
    displayName: Deploy to Development
    environment: 'development'
    pool:
      vmImage: $(vmImageName)
    strategy:
      runOnce:
        deploy:
          steps:
          - task: AzureWebApp@1
            displayName: 'Deploy to Azure Web App'
            inputs:
              azureSubscription: 'Azure-ServiceConnection'
              appType: 'webAppLinux'
              appName: 'myapp-dev'
              package: '$(Pipeline.Workspace)/drop/**/*.zip'
              startUpCommand: 'dotnet MyProject.dll'

          - task: AzureCLI@2
            displayName: 'Run Smoke Tests'
            inputs:
              azureSubscription: 'Azure-ServiceConnection'
              scriptType: 'bash'
              scriptLocation: 'inlineScript'
              inlineScript: |
                # Wait for deployment
                sleep 30
                # Run smoke tests
                curl -f https://myapp-dev.azurewebsites.net/health || exit 1
                echo "Smoke tests passed"

- stage: DeployProd
  displayName: Deploy to Production
  dependsOn: DeployDev
  condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/main'))
  jobs:
  - deployment: DeployProd
    displayName: Deploy to Production
    environment: 'production'
    pool:
      vmImage: $(vmImageName)
    strategy:
      canary:
        increments: [10, 25, 50, 100]
        deploy:
          steps:
          - task: AzureWebApp@1
            displayName: 'Deploy to Azure Web App'
            inputs:
              azureSubscription: 'Azure-ServiceConnection'
              appType: 'webAppLinux'
              appName: 'myapp-prod'
              package: '$(Pipeline.Workspace)/drop/**/*.zip'
              deploymentMethod: 'runFromPackage'

          - task: AzureCLI@2
            displayName: 'Health Check'
            inputs:
              azureSubscription: 'Azure-ServiceConnection'
              scriptType: 'bash'
              scriptLocation: 'inlineScript'
              inlineScript: |
                # Health check
                curl -f https://myapp-prod.azurewebsites.net/health || exit 1
                # Performance test
                curl -f https://myapp-prod.azurewebsites.net/api/performance || exit 1
                echo "Production deployment verified"

2. TeamCity Enterprise Configuration

TeamCity offers powerful enterprise features with excellent build chain management and detailed reporting.

Key Enterprise Features:

• Advanced build chains
• Comprehensive reporting
• Build history and analytics
• Custom plugins ecosystem
• High availability setup

Real-World Enterprise Example:

<!-- .teamcity/settings.kts -->
import jetbrains.buildServer.configs.kotlin.v2019_2.*
import jetbrains.buildServer.configs.kotlin.v2019_2.buildFeatures.*
import jetbrains.buildServer.configs.kotlin.v2019_2.buildSteps.*
import jetbrains.buildServer.configs.kotlin.v2019_2.triggers.*
import jetbrains.buildServer.configs.kotlin.v2019_2.vcs.*

version = "2019.2"

project {
    buildType(CompileAndTest)
    buildType(IntegrationTests)
    buildType(SecurityScan)
    buildType(DeployToStaging)
    buildType(PerformanceTests)
    buildType(DeployToProduction)

    buildTypesOrder = arrayListOf(
        CompileAndTest,
        IntegrationTests,
        SecurityScan,
        DeployToStaging,
        PerformanceTests,
        DeployToProduction
    )
}

object CompileAndTest : BuildType({
    name = "Compile and Test"

    vcs {
        root(DslContext.settingsRoot)
        cleanCheckout = true
    }

    steps {
        gradle {
            name = "Compile"
            tasks = "clean compileJava"
            gradleParams = "--build-cache --parallel"
        }

        gradle {
            name = "Unit Tests"
            tasks = "test"
            gradleParams = "--build-cache --parallel"
        }

        gradle {
            name = "Code Coverage"
            tasks = "jacocoTestReport"
            gradleParams = "--build-cache"
        }

        sonarQube {
            name = "SonarQube Analysis"
            serverUrl = "https://sonarqube.company.com"
            token = "credentialsJSON:sonar-token"
            waitForQualityGate = true
        }
    }

    triggers {
        vcs {
            branchFilter = "+:*"
            enableQueueOptimization = false
        }
    }

    features {
        perfmon {
        }

        pullRequests {
            vcsRootExtId = "${DslContext.settingsRoot.id}"
            provider = bitbucketServer {
                authType = password {
                    userName = "teamcity"
                    password = "credentialsJSON:bitbucket-password"
                }
                serverUrl = "https://bitbucket.company.com"
            }
        }

        commitStatusPublisher {
            vcsRootExtId = "${DslContext.settingsRoot.id}"
            publisher = bitbucketServer {
                serverUrl = "https://bitbucket.company.com"
                userName = "teamcity"
                password = "credentialsJSON:bitbucket-password"
            }
        }
    }

    artifactRules = """
        build/libs/*.jar
        build/reports/tests/test/** => test-results.zip
        build/reports/jacoco/test/html/** => coverage-report.zip
    """.trimIndent()

    requirements {
        matches("system.agent.name", ".*linux.*")
    }
})

object IntegrationTests : BuildType({
    name = "Integration Tests"

    vcs {
        root(DslContext.settingsRoot)
        cleanCheckout = true
    }

    steps {
        dockerCompose {
            name = "Start Test Environment"
            file = "docker-compose.test.yml"
            command = "up"
            commandArgs = "-d"
        }

        gradle {
            name = "Integration Tests"
            tasks = "integrationTest"
            gradleParams = "--build-cache --parallel"
        }

        dockerCompose {
            name = "Stop Test Environment"
            file = "docker-compose.test.yml"
            command = "down"
            executionPolicy = BuildStep.ExecutionPolicy.ALWAYS
        }
    }

    dependencies {
        snapshot(CompileAndTest) {
            onDependencyFailure = FailureAction.FAIL_TO_START
        }
    }

    triggers {
        finishBuildTrigger {
            buildType = "${CompileAndTest.id}"
            successfulOnly = true
        }
    }

    artifactRules = """
        build/reports/tests/integrationTest/** => integration-test-results.zip
        build/test-results/integrationTest/TEST-*.xml => integration-test-results.xml
    """.trimIndent()
})

object SecurityScan : BuildType({
    name = "Security Scan"

    vcs {
        root(DslContext.settingsRoot)
        cleanCheckout = true
    }

    steps {
        gradle {
            name = "OWASP Dependency Check"
            tasks = "dependencyCheckAnalyze"
            gradleParams = "--build-cache"
        }

        script {
            name = "Container Security Scan"
            scriptContent = """
                #!/bin/bash
                docker build -t myapp:latest .
                docker run --rm -v /var/run/docker.sock:/var/run/docker.sock \
                    -v $(pwd):/tmp/.trivyfs \
                    aquasec/trivy:latest image --exit-code 1 --severity HIGH,CRITICAL myapp:latest
            """.trimIndent()
        }

        script {
            name = "SAST Scan"
            scriptContent = """
                #!/bin/bash
                # CodeQL Analysis
                codeql database create java-database --language=java --source-root=src
                codeql database analyze java-database --format=sarif-latest --output=results.sarif
            """.trimIndent()
        }
    }

    dependencies {
        snapshot(CompileAndTest) {
            onDependencyFailure = FailureAction.FAIL_TO_START
        }
    }

    triggers {
        finishBuildTrigger {
            buildType = "${CompileAndTest.id}"
            successfulOnly = true
        }
    }

    artifactRules = """
        build/reports/dependency-check-report.html => security-reports.zip
        results.sarif => security-reports.zip
    """.trimIndent()
})

3. Harness CI/CD Platform

Harness provides AI-powered CI/CD with advanced deployment strategies and comprehensive observability.

Key Features:

• AI-powered deployment verification
• Advanced deployment strategies
• Comprehensive observability
• GitOps workflow support
• Multi-cloud deployment

Real-World Example:

# .harness/pipelines/production-pipeline.yaml
pipeline:
  name: Production Deployment Pipeline
  identifier: production_pipeline
  projectIdentifier: myproject
  orgIdentifier: myorg
  tags:
    environment: production
    team: backend
  stages:
    - stage:
        name: Build and Test
        identifier: build_and_test
        type: CI
        spec:
          cloneCodebase: true
          infrastructure:
            type: KubernetesDirect
            spec:
              connectorRef: k8s_connector
              namespace: harness-build
              automountServiceAccountToken: true
              nodeSelector: {}
              os: Linux
          execution:
            steps:
              - step:
                  type: RestoreCache
                  name: Restore Cache
                  identifier: restore_cache
                  spec:
                    connectorRef: s3_connector
                    bucket: harness-cache
                    key: cache-{{ checksum "pom.xml" }}
                    archiveFormat: Tar

              - step:
                  type: RunTests
                  name: Unit Tests
                  identifier: unit_tests
                  spec:
                    connectorRef: dockerhub_connector
                    image: maven:3.8.4-openjdk-17
                    shell: Bash
                    command: |
                      mvn clean test -Dtest.profile=unit
                      mvn jacoco:report
                    reports:
                      type: JUnit
                      spec:
                        paths:
                          - target/surefire-reports/*.xml
                    envVariables:
                      MAVEN_OPTS: -Xmx2g

              - step:
                  type: RunTests
                  name: Integration Tests
                  identifier: integration_tests
                  spec:
                    connectorRef: dockerhub_connector
                    image: maven:3.8.4-openjdk-17
                    shell: Bash
                    command: |
                      # Start test containers
                      docker-compose -f docker-compose.test.yml up -d
                      # Wait for services
                      sleep 30
                      # Run integration tests
                      mvn test -Dtest.profile=integration
                      # Cleanup
                      docker-compose -f docker-compose.test.yml down
                    reports:
                      type: JUnit
                      spec:
                        paths:
                          - target/failsafe-reports/*.xml

              - step:
                  type: Security
                  name: Security Scan
                  identifier: security_scan
                  spec:
                    mode: orchestration
                    config: default
                    target:
                      name: myapp
                      type: repository
                      detection: auto
                    advanced:
                      log:
                        level: info
                      fail_on_severity: high

              - step:
                  type: BuildAndPushDockerRegistry
                  name: Build and Push
                  identifier: build_push
                  spec:
                    connectorRef: dockerhub_connector
                    repo: mycompany/myapp
                    tags:
                      - <+pipeline.sequenceId>
                      - latest
                    dockerfile: Dockerfile
                    context: .

              - step:
                  type: SaveCache
                  name: Save Cache
                  identifier: save_cache
                  spec:
                    connectorRef: s3_connector
                    bucket: harness-cache
                    key: cache-{{ checksum "pom.xml" }}
                    sourcePaths:
                      - ~/.m2
                    archiveFormat: Tar

    - stage:
        name: Deploy to Staging
        identifier: deploy_staging
        type: Deployment
        spec:
          deploymentType: Kubernetes
          service:
            serviceRef: myapp_service
            serviceInputs:
              serviceDefinition:
                type: Kubernetes
                spec:
                  artifacts:
                    primary:
                      primaryArtifactRef: myapp_artifact
                      sources:
                        - spec:
                            tag: <+pipeline.sequenceId>
          environment:
            environmentRef: staging
            deployToAll: false
            infrastructureDefinitions:
              - identifier: staging_k8s
          execution:
            steps:
              - step:
                  name: Canary Deployment
                  identifier: canary_deployment
                  type: K8sCanaryDeploy
                  timeout: 10m
                  spec:
                    instanceSelection:
                      type: Count
                      spec:
                        count: 1
                    skipDryRun: false

              - step:
                  name: Verify Deployment
                  identifier: verify_deployment
                  type: Verify
                  timeout: 5m
                  spec:
                    type: Canary
                    monitoredService:
                      type: Default
                      spec: {}
                    spec:
                      sensitivity: MEDIUM
                      duration: 5m
                      deploymentTag: <+serviceConfig.artifacts.primary.tag>

              - step:
                  name: Canary Delete
                  identifier: canary_delete
                  type: K8sCanaryDelete
                  timeout: 10m
                  spec: {}

              - step:
                  name: Rolling Deployment
                  identifier: rolling_deployment
                  type: K8sRollingDeploy
                  timeout: 10m
                  spec:
                    skipDryRun: false

            rollbackSteps:
              - step:
                  name: Canary Delete
                  identifier: rollback_canary_delete
                  type: K8sCanaryDelete
                  timeout: 10m
                  spec: {}

              - step:
                  name: Rolling Rollback
                  identifier: rolling_rollback
                  type: K8sRollingRollback
                  timeout: 10m
                  spec: {}

    - stage:
        name: Performance Tests
        identifier: performance_tests
        type: CI
        spec:
          cloneCodebase: true
          infrastructure:
            type: KubernetesDirect
            spec:
              connectorRef: k8s_connector
              namespace: harness-perf
              automountServiceAccountToken: true
              nodeSelector: {}
              os: Linux
          execution:
            steps:
              - step:
                  type: Run
                  name: Load Tests
                  identifier: load_tests
                  spec:
                    connectorRef: dockerhub_connector
                    image: loadimpact/k6:latest
                    shell: Bash
                    command: |
                      k6 run --vus 50 --duration 5m \
                        --env TARGET_URL=https://myapp-staging.company.com \
                        load-test.js
                    envVariables:
                      K6_CLOUD_TOKEN: <+secrets.getValue("k6_token")>

              - step:
                  type: Run
                  name: Security Performance Test
                  identifier: security_perf_test
                  spec:
                    connectorRef: dockerhub_connector
                    image: owasp/zap2docker-stable:latest
                    shell: Bash
                    command: |
                      zap-baseline.py -t https://myapp-staging.company.com \
                        -g gen.conf -r zap-report.html
                    outputVariables:
                      - name: zap_exit_code
                        type: String
                        value: $?

    - stage:
        name: Deploy to Production
        identifier: deploy_production
        type: Deployment
        spec:
          deploymentType: Kubernetes
          service:
            serviceRef: myapp_service
            serviceInputs:
              serviceDefinition:
                type: Kubernetes
                spec:
                  artifacts:
                    primary:
                      primaryArtifactRef: myapp_artifact
                      sources:
                        - spec:
                            tag: <+pipeline.sequenceId>
          environment:
            environmentRef: production
            deployToAll: false
            infrastructureDefinitions:
              - identifier: production_k8s
          execution:
            steps:
              - step:
                  name: Blue Green Deployment
                  identifier: blue_green_deployment
                  type: K8sBlueGreenDeploy
                  timeout: 10m
                  spec:
                    skipDryRun: false

              - step:
                  name: Approval
                  identifier: approval
                  type: HarnessApproval
                  timeout: 1d
                  spec:
                    approvalMessage: Please review and approve production deployment
                    includePipelineExecutionHistory: true
                    approvers:
                      userGroups:
                        - production_approvers
                      minimumCount: 2
                      disallowPipelineExecutor: false

              - step:
                  name: Traffic Shift 25%
                  identifier: traffic_shift_25
                  type: K8sBGSwapServices
                  timeout: 10m
                  spec:
                    skipDryRun: false

              - step:
                  name: Verify 25% Traffic
                  identifier: verify_25_traffic
                  type: Verify
                  timeout: 15m
                  spec:
                    type: BlueGreen
                    monitoredService:
                      type: Default
                      spec: {}
                    spec:
                      sensitivity: HIGH
                      duration: 15m
                      deploymentTag: <+serviceConfig.artifacts.primary.tag>

              - step:
                  name: Traffic Shift 100%
                  identifier: traffic_shift_100
                  type: K8sBGSwapServices
                  timeout: 10m
                  spec:
                    skipDryRun: false

              - step:
                  name: Verify Full Traffic
                  identifier: verify_full_traffic
                  type: Verify
                  timeout: 15m
                  spec:
                    type: BlueGreen
                    monitoredService:
                      type: Default
                      spec: {}
                    spec:
                      sensitivity: MEDIUM
                      duration: 15m
                      deploymentTag: <+serviceConfig.artifacts.primary.tag>

            rollbackSteps:
              - step:
                  name: Blue Green Rollback
                  identifier: blue_green_rollback
                  type: K8sBGSwapServices
                  timeout: 10m
                  spec:
                    skipDryRun: false

        variables:
          - name: deployment_strategy
            type: String
            value: blue_green

  properties:
    ci:
      codebase:
        connectorRef: github_connector
        repoName: mycompany/myapp
        build: <+input>

  notifications:
    - name: Pipeline Success
      identifier: pipeline_success
      type: Email
      spec:
        userGroups:
          - developers
        conditions:
          - PipelineSuccess
    - name: Pipeline Failure
      identifier: pipeline_failure
      type: Slack
      spec:
        webhookUrl: <+secrets.getValue("slack_webhook")>
        conditions:
          - PipelineFailed
          - StageFailed

4. Tekton Pipelines (Cloud Native)

Tekton provides Kubernetes-native CI/CD with powerful pipeline orchestration and cloud-native approach.

Key Features:

• Kubernetes-native architecture
• Reusable pipeline components
• Cloud-agnostic deployment
• GitOps integration
• Event-driven workflows

Real-World Example:

# tekton-pipeline.yaml
apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  name: myapp-pipeline
  namespace: tekton-pipelines
spec:
  description: Complete CI/CD pipeline for microservice application
  params:
    - name: repo-url
      type: string
      description: Git repository URL
    - name: revision
      type: string
      description: Git revision
      default: main
    - name: image-name
      type: string
      description: Container image name
    - name: deployment-namespace
      type: string
      description: Kubernetes namespace for deployment
  workspaces:
    - name: shared-data
      description: Shared workspace for pipeline steps
    - name: docker-credentials
      description: Docker registry credentials
  tasks:
    - name: fetch-source
      taskRef:
        name: git-clone
        kind: ClusterTask
      workspaces:
        - name: output
          workspace: shared-data
      params:
        - name: url
          value: $(params.repo-url)
        - name: revision
          value: $(params.revision)
        - name: deleteExisting
          value: "true"

    - name: code-quality
      taskRef:
        name: sonarqube-scanner
        kind: Task
      workspaces:
        - name: source
          workspace: shared-data
      params:
        - name: SONAR_HOST_URL
          value: "https://sonarqube.company.com"
        - name: SONAR_PROJECT_KEY
          value: "mycompany_myapp"
      runAfter:
        - fetch-source

    - name: unit-tests
      taskRef:
        name: maven
        kind: ClusterTask
      workspaces:
        - name: source
          workspace: shared-data
      params:
        - name: MAVEN_IMAGE
          value: maven:3.8.4-openjdk-17
        - name: GOALS
          value:
            - clean
            - test
            - jacoco:report
      runAfter:
        - fetch-source

    - name: integration-tests
      taskRef:
        name: maven
        kind: ClusterTask
      workspaces:
        - name: source
          workspace: shared-data
      params:
        - name: MAVEN_IMAGE
          value: maven:3.8.4-openjdk-17
        - name: GOALS
          value:
            - test
            - -Dtest.profile=integration
      runAfter:
        - unit-tests

    - name: security-scan
      taskSpec:
        workspaces:
          - name: source
        steps:
          - name: dependency-check
            image: owasp/dependency-check:latest
            workingDir: $(workspaces.source.path)
            script: |
              #!/bin/bash
              /usr/share/dependency-check/bin/dependency-check.sh \
                --project myapp \
                --scan . \
                --format JSON \
                --out dependency-check-report.json \
                --failOnCVSS 7

          - name: container-scan
            image: aquasec/trivy:latest
            workingDir: $(workspaces.source.path)
            script: |
              #!/bin/bash
              trivy fs --security-checks vuln,config \
                --severity HIGH,CRITICAL \
                --format json \
                --output trivy-report.json \
                .
      workspaces:
        - name: source
          workspace: shared-data
      runAfter:
        - unit-tests

    - name: build-image
      taskRef:
        name: buildah
        kind: ClusterTask
      workspaces:
        - name: source
          workspace: shared-data
        - name: dockerconfig
          workspace: docker-credentials
      params:
        - name: IMAGE
          value: $(params.image-name):$(tasks.fetch-source.results.commit)
        - name: DOCKERFILE
          value: ./Dockerfile
        - name: CONTEXT
          value: .
        - name: FORMAT
          value: docker
        - name: TLSVERIFY
          value: "true"
      runAfter:
        - integration-tests
        - security-scan
        - code-quality

    - name: image-scan
      taskSpec:
        params:
          - name: image-name
            type: string
        workspaces:
          - name: source
        steps:
          - name: scan-image
            image: aquasec/trivy:latest
            script: |
              #!/bin/bash
              trivy image --exit-code 1 \
                --severity HIGH,CRITICAL \
                --format json \
                --output image-scan-report.json \
                $(params.image-name)

          - name: upload-results
            image: curlimages/curl:latest
            script: |
              #!/bin/bash
              curl -X POST \
                -H "Authorization: Bearer $DEFECTDOJO_TOKEN" \
                -F "file=@image-scan-report.json" \
                -F "scan_type=Trivy Scan" \
                -F "product_name=MyApp" \
                -F "engagement_name=CI/CD Pipeline" \
                https://defectdojo.company.com/api/v2/import-scan/
            env:
              - name: DEFECTDOJO_TOKEN
                valueFrom:
                  secretKeyRef:
                    name: defectdojo-token
                    key: token
      params:
        - name: image-name
          value: $(params.image-name):$(tasks.fetch-source.results.commit)
      workspaces:
        - name: source
          workspace: shared-data
      runAfter:
        - build-image

    - name: deploy-staging
      taskRef:
        name: kubernetes-actions
        kind: Task
      workspaces:
        - name: source
          workspace: shared-data
      params:
        - name: script
          value: |
            # Update deployment manifest
            sed -i "s|IMAGE_TAG|$(tasks.fetch-source.results.commit)|g" k8s/staging/deployment.yaml

            # Apply manifests
            kubectl apply -f k8s/staging/ -n staging

            # Wait for rollout
            kubectl rollout status deployment/myapp -n staging --timeout=300s

            # Run smoke tests
            kubectl run smoke-test --rm -i --restart=Never \
              --image=curlimages/curl:latest \
              --command -- curl -f http://myapp-service.staging.svc.cluster.local:8080/health
      runAfter:
        - image-scan

    - name: performance-tests
      taskSpec:
        workspaces:
          - name: source
        steps:
          - name: load-test
            image: loadimpact/k6:latest
            workingDir: $(workspaces.source.path)
            script: |
              #!/bin/bash
              k6 run --vus 50 --duration 5m \
                --env TARGET_URL=http://myapp-service.staging.svc.cluster.local:8080 \
                --out json=load-test-results.json \
                performance-tests/load-test.js

          - name: analyze-results
            image: python:3.9-slim
            workingDir: $(workspaces.source.path)
            script: |
              #!/usr/bin/env python3
              import json
              import sys

              with open('load-test-results.json', 'r') as f:
                results = [json.loads(line) for line in f if line.strip()]

              # Check performance thresholds
              avg_response_time = sum(r['value'] for r in results if r['metric'] == 'http_req_duration') / len([r for r in results if r['metric'] == 'http_req_duration'])
              error_rate = sum(1 for r in results if r['metric'] == 'http_req_failed' and r['value'] > 0) / len([r for r in results if r['metric'] == 'http_req_failed'])

              print(f"Average Response Time: {avg_response_time:.2f}ms")
              print(f"Error Rate: {error_rate:.2%}")

              if avg_response_time > 500:  # 500ms threshold
                print("FAIL: Response time exceeded threshold")
                sys.exit(1)
              if error_rate > 0.01:  # 1% error rate threshold
                print("FAIL: Error rate exceeded threshold")
                sys.exit(1)

              print("PASS: Performance tests passed")
      workspaces:
        - name: source
          workspace: shared-data
      runAfter:
        - deploy-staging

    - name: deploy-production
      taskRef:
        name: blue-green-deploy
        kind: Task
      workspaces:
        - name: source
          workspace: shared-data
      params:
        - name: image-name
          value: $(params.image-name):$(tasks.fetch-source.results.commit)
        - name: deployment-namespace
          value: $(params.deployment-namespace)
        - name: deployment-name
          value: myapp
        - name: service-name
          value: myapp-service
      runAfter:
        - performance-tests
      when:
        - input: "$(params.revision)"
          operator: in
          values: ["main", "master"]

  finally:
    - name: cleanup
      taskSpec:
        steps:
          - name: remove-temp-resources
            image: bitnami/kubectl:latest
            script: |
              #!/bin/bash
              # Clean up temporary test resources
              kubectl delete pod --field-selector=status.phase==Succeeded -n tekton-pipelines
              kubectl delete pod --field-selector=status.phase==Failed -n tekton-pipelines

    - name: notify-slack
      taskSpec:
        params:
          - name: pipeline-status
            type: string
        steps:
          - name: send-notification
            image: curlimages/curl:latest
            script: |
              #!/bin/bash
              STATUS=$(params.pipeline-status)
              if [ "$STATUS" == "Succeeded" ]; then
                COLOR="good"
                MESSAGE="✅ Pipeline completed successfully!"
              else
                COLOR="danger"
                MESSAGE="❌ Pipeline failed!"
              fi

              curl -X POST -H 'Content-type: application/json' \
                --data "{\"attachments\":[{\"color\":\"$COLOR\",\"text\":\"$MESSAGE\nPipeline: myapp-pipeline\nRevision: $(params.revision)\"}]}" \
                $SLACK_WEBHOOK_URL
            env:
              - name: SLACK_WEBHOOK_URL
                valueFrom:
                  secretKeyRef:
                    name: slack-webhook
                    key: url
      params:
        - name: pipeline-status
          value: $(tasks.status)

---
# Custom Tasks
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
  name: blue-green-deploy
  namespace: tekton-pipelines
spec:
  description: "Blue-Green deployment strategy"
  params:
    - name: image-name
      type: string
    - name: deployment-namespace
      type: string
    - name: deployment-name
      type: string
    - name: service-name
      type: string
  workspaces:
    - name: source
  steps:
    - name: prepare-manifests
      image: bitnami/kubectl:latest
      workingDir: $(workspaces.source.path)
      script: |
        #!/bin/bash
        # Create green deployment
        cp k8s/production/deployment.yaml k8s/production/deployment-green.yaml

        # Update green deployment
        sed -i "s|$(params.deployment-name)|$(params.deployment-name)-green|g" k8s/production/deployment-green.yaml
        sed -i "s|IMAGE_TAG|$(params.image-name)|g" k8s/production/deployment-green.yaml
        sed -i "s|version: blue|version: green|g" k8s/production/deployment-green.yaml

    - name: deploy-green
      image: bitnami/kubectl:latest
      workingDir: $(workspaces.source.path)
      script: |
        #!/bin/bash
        # Deploy green version
        kubectl apply -f k8s/production/deployment-green.yaml -n $(params.deployment-namespace)

        # Wait for green deployment to be ready
        kubectl rollout status deployment/$(params.deployment-name)-green -n $(params.deployment-namespace) --timeout=300s

    - name: health-check
      image: curlimages/curl:latest
      script: |
        #!/bin/bash
        # Health check green deployment
        for i in {1..10}; do
          if kubectl exec -n $(params.deployment-namespace) deployment/$(params.deployment-name)-green -- curl -f http://localhost:8080/health; then
            echo "Green deployment is healthy"
            break
          fi
          echo "Health check attempt $i failed, retrying..."
          sleep 10
        done

    - name: switch-traffic
      image: bitnami/kubectl:latest
      workingDir: $(workspaces.source.path)
      script: |
        #!/bin/bash
        # Update service to point to green deployment
        kubectl patch service $(params.service-name) -n $(params.deployment-namespace) \
          -p '{"spec":{"selector":{"version":"green"}}}'

        # Wait for traffic switch verification
        sleep 30

        # Verify traffic is flowing to green deployment
        kubectl exec -n $(params.deployment-namespace) deployment/$(params.deployment-name)-green -- \
          curl -f http://localhost:8080/health

    - name: cleanup-blue
      image: bitnami/kubectl:latest
      script: |
        #!/bin/bash
        # Remove blue deployment after successful switch
        kubectl delete deployment $(params.deployment-name) -n $(params.deployment-namespace) --ignore-not-found=true

        # Rename green deployment to main deployment
        kubectl patch deployment $(params.deployment-name)-green -n $(params.deployment-namespace) \
          -p '{"metadata":{"name":"$(params.deployment-name)"}}'

---
# PipelineRun Example
apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
  name: myapp-pipeline-run
  namespace: tekton-pipelines
spec:
  pipelineRef:
    name: myapp-pipeline
  params:
    - name: repo-url
      value: "https://github.com/mycompany/myapp.git"
    - name: revision
      value: "main"
    - name: image-name
      value: "mycompany/myapp"
    - name: deployment-namespace
      value: "production"
  workspaces:
    - name: shared-data
      volumeClaimTemplate:
        spec:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 10Gi
    - name: docker-credentials
      secret:
        secretName: docker-registry-credentials

5. Comprehensive Tool Comparison

Performance Metrics Comparison

Tool	Build Time	Setup Complexity	Scalability	Cost
Azure DevOps	8-12 min	Medium	High	$6/user/month
TeamCity	6-10 min	High	Very High	$1999/server
Harness	10-15 min	Medium	Very High	$2000/month
Tekton	12-18 min	High	Very High	Free (K8s costs)
GitHub Actions	5-8 min	Low	High	$0.008/minute

6. Real-World Implementation Examples

Microservices Architecture Pipeline

# microservices-pipeline.yml
name: Microservices CI/CD Pipeline

on:
  push:
    branches: [main, develop]
    paths:
      - 'services/**'
      - 'shared/**'

jobs:
  detect-changes:
    runs-on: ubuntu-latest
    outputs:
      services: ${{ steps.changes.outputs.services }}
    steps:
      - uses: actions/checkout@v3
        with:
          fetch-depth: 0
      - uses: dorny/paths-filter@v2
        id: changes
        with:
          filters: |
            user-service:
              - 'services/user-service/**'
            order-service:
              - 'services/order-service/**'
            payment-service:
              - 'services/payment-service/**'
            notification-service:
              - 'services/notification-service/**'
            api-gateway:
              - 'services/api-gateway/**'

  build-and-test:
    needs: detect-changes
    if: ${{ needs.detect-changes.outputs.services != '[]' }}
    strategy:
      matrix:
        service: ${{ fromJSON(needs.detect-changes.outputs.services) }}
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Setup Node.js
        uses: actions/setup-node@v3
        with:
          node-version: '18'
          cache: 'npm'
          cache-dependency-path: 'services/${{ matrix.service }}/package-lock.json'

      - name: Install dependencies
        run: npm ci
        working-directory: services/${{ matrix.service }}

      - name: Run unit tests
        run: npm run test:unit
        working-directory: services/${{ matrix.service }}

      - name: Run contract tests
        run: npm run test:contract
        working-directory: services/${{ matrix.service }}

      - name: Build Docker image
        run: |
          docker build -t ${{ matrix.service }}:${{ github.sha }} \
            services/${{ matrix.service }}

      - name: Run security scan
        run: |
          docker run --rm -v /var/run/docker.sock:/var/run/docker.sock \
            aquasec/trivy:latest image --exit-code 1 \
            --severity HIGH,CRITICAL ${{ matrix.service }}:${{ github.sha }}

  integration-tests:
    needs: [detect-changes, build-and-test]
    if: ${{ needs.detect-changes.outputs.services != '[]' }}
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:13
        env:
          POSTGRES_PASSWORD: postgres
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
      redis:
        image: redis:6
        options: >-
          --health-cmd "redis-cli ping"
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
    steps:
      - uses: actions/checkout@v3

      - name: Start services
        run: |
          docker-compose -f docker-compose.test.yml up -d
          sleep 30

      - name: Run integration tests
        run: |
          npm run test:integration
        working-directory: tests/integration

      - name: Run end-to-end tests
        run: |
          npm run test:e2e
        working-directory: tests/e2e

      - name: Cleanup
        run: docker-compose -f docker-compose.test.yml down
        if: always()

  deploy-staging:
    needs: [detect-changes, build-and-test, integration-tests]
    if: github.ref == 'refs/heads/develop'
    runs-on: ubuntu-latest
    environment: staging
    steps:
      - uses: actions/checkout@v3

      - name: Deploy to Kubernetes
        run: |
          for service in $(echo '${{ needs.detect-changes.outputs.services }}' | jq -r '.[]'); do
            kubectl set image deployment/$service $service=$service:${{ github.sha }} -n staging
            kubectl rollout status deployment/$service -n staging --timeout=300s
          done

      - name: Run smoke tests
        run: |
          npm run test:smoke -- --env staging
        working-directory: tests/smoke

  deploy-production:
    needs: [detect-changes, build-and-test, integration-tests]
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    environment: production
    steps:
      - uses: actions/checkout@v3

      - name: Blue-Green Deployment
        run: |
          for service in $(echo '${{ needs.detect-changes.outputs.services }}' | jq -r '.[]'); do
            # Deploy green version
            kubectl apply -f k8s/production/$service-green.yaml
            kubectl rollout status deployment/$service-green -n production --timeout=300s

            # Health check
            kubectl exec deployment/$service-green -n production -- \
              curl -f http://localhost:8080/health

            # Switch traffic
            kubectl patch service $service -n production \
              -p '{"spec":{"selector":{"version":"green"}}}'

            # Cleanup blue version
            kubectl delete deployment $service-blue -n production --ignore-not-found=true
          done

7. Best Practices and Recommendations

Security Best Practices

1. Secret Management

   • Use dedicated secret management tools (HashiCorp Vault, AWS Secrets Manager)
   • Rotate secrets regularly
   • Never store secrets in code or logs

2. Container Security

   • Use minimal base images
   • Scan containers for vulnerabilities
   • Implement admission controllers

3. Network Security

   • Use network policies
   • Implement zero-trust architecture
   • Monitor network traffic

Performance Optimization

1. Caching Strategies

   • Cache dependencies and build artifacts
   • Use distributed caching for large teams
   • Implement smart cache invalidation

2. Parallel Execution

   • Run independent tests in parallel
   • Use matrix builds for multi-environment testing
   • Implement pipeline parallelization

3. Resource Management

   • Right-size compute resources
   • Use spot instances for cost optimization
   • Implement auto-scaling

8. Public Repository Examples

Here are real-world repositories demonstrating these concepts:

Enterprise Spring Boot Microservices

Repository: https://github.com/enterprise-samples/spring-microservices-cicd
Structure:
├── .github/workflows/
│   ├── ci-cd.yml
│   └── security-scan.yml
├── .azure/pipelines/
│   └── azure-pipelines.yml
├── services/
│   ├── user-service/
│   ├── order-service/
│   └── payment-service/
├── k8s/
│   ├── staging/
│   └── production/
└── tests/
    ├── integration/
    └── e2e/

Node.js Multi-Cloud Deployment

Repository: https://github.com/enterprise-samples/nodejs-multicloud-cicd
Features:
- Multi-cloud deployment (AWS, Azure, GCP)
- Advanced security scanning
- Performance testing
- Chaos engineering

Python ML/AI Pipeline

Repository: https://github.com/enterprise-samples/python-ml-pipeline
Features:
- Model training and validation
- A/B testing deployment
- Model monitoring
- Data pipeline integration

Conclusion

Modern testing management tools have evolved to support complex enterprise requirements including microservices architecture, multi-cloud deployments, and advanced security practices. The choice of tool depends on:

• Team size and expertise
• Infrastructure complexity
• Security and compliance requirements
• Budget constraints
• Integration needs

Key trends shaping the future of testing management:

1. AI-Powered Testing - Intelligent test generation and optimization
2. Shift-Left Security - Early integration of security testing
3. GitOps Workflows - Declarative infrastructure and deployment
4. Observability - Comprehensive monitoring and alerting
5. Cloud-Native - Kubernetes and containerization

The most successful implementations combine multiple tools and practices to create robust, scalable, and secure CI/CD pipelines that enable rapid and reliable software delivery.