Building a Multi-Cloud Bug Bounty Automation Platform

Whoami

My name is Álvaro García. I'm Spanish, I spent 8 years as a backend engineer before going full-time on this project, and I hold a degree in Telecommunications Engineering plus an MSc in Cybersecurity.

Towards the end of that 8-year stretch, I started specializing in AWS and DevOps, and earned all three AWS Associate certs, both Professional certs (Solutions Architect Professional and DevOps Engineer Professional), plus the Security and Advanced Networking Specialties. This served two purposes: a fallback in case this project didn't pan out, and as a way to go deep on AWS, since I knew that knowledge would be useful for this project.

I spent the next three years working full-time, solo, on the project that this technical writeup is about, a fully automated reconnaissance and vulnerability-scanning pipeline for bug bounty work, built across AWS, GCP, and Alibaba Cloud.

Brief Introduction to Bug Bounty

Bug bounty is a crowdsourced cybersecurity practice in which independent researchers ("hackers") are rewarded — typically with money, recognition, or both — for finding and responsibly reporting vulnerabilities in software, websites, or systems. Organizations that adopt this practice do so by running a bug bounty program. Some platforms act as intermediaries between the security researchers and the companies. Some of the major players in this space are Hackerone, Bugcrowd or Intigriti.

Introduction

The bug-bounty side of the project didn't pay off financially for reasons explained in the Retrospective section. But the infrastructure built to run this at scale turned out to be the more valuable outcome of the project. What started as a way to automate subdomain enumeration became an event-driven, cost-aware, multi-cloud job orchestration system. The kind of platform I'd now confidently design for any infrastructure-automation problem, security-related or not.

This write-up is about that platform more than about cybersecurity or bug bounty hunting itself. I'll cover why I chose SQS over a message broker, why Nomad over Kubernetes, how compute gets provisioned and torn down automatically across GCP and Alibaba Cloud based on queue depth, and, in the Retrospective section at the end, what I'd do differently.

Some of the current stats of the project:

• 1,253 active bug bounty programs tracked
• 13,303 root domains tracked
• 5.6 million discovered subdomains tracked
• 5 Nomad system jobs and 12 reconnaissance/attack job types running continuously

Architecture Overview

At its core, the system is a job pipeline coordinated through a message queue, executed by HashiCorp Nomad, and run on compute that's provisioned and destroyed on demand across two cloud providers.

Nomad is the piece most readers won't have touched before, here's a quick introduction: it's a workload orchestrator in the same category as Kubernetes, but lighter. It can schedule and run plain binaries directly on a host (raw_exec) or in a lightweight sandboxed process (exec), without requiring everything to be packaged as a container. Jobs are defined in HCL (HashiCorp Configuration Language) files, and a single job can have a main task plus optional tasks that run before or after it. This pre/post-task pattern is heavily used in this project.

The data flow works like this: a domain enters the pipeline, either a new bug bounty program being added, or a randomly selected domain, and lands as a message on an SQS queue. A poller reads that queue once a minute and dispatches the corresponding job to Nomad, which schedules it on whichever available node meets its resource and instance type constraints. When the job finishes, a poststop task uploads its raw output to S3, writes the parsed results into the database, and pushes a new message describing the job that was completed onto a second queue. That message gets picked up, converted into the next job, and the cycle repeats. A single root domain flows automatically through enumeration, resolution, probing, and scanning with no manual intervention at any step.

The compute nodes that run these jobs are rented on demand. GCP and Alibaba Cloud instances are launched in batches when there's backlog in the queue and torn down again once it clears, using spot/preemptible pricing that cuts compute cost by up to 90%. AWS plays a different role here: not running scan jobs, but acting as the control plane. Lambda and Step Functions drive the scale-up/scale-down and region-rotation logic, Parameter Store holds configuration and secrets, S3 stores the raw output of some jobs, and CloudWatch and the API Gateway (not described in this writeup) round out the operational layer. One always-on server, hosted on Hetzner, runs the Nomad server itself and the SurrealDB database that holds the system's data. Terraform provisions the cloud infrastructure, Packer and Ansible build and configure the machine images that run on it, and a Wireguard mesh gives every node private connectivity back to that central server and between the nodes themselves.

The rest of this write-up walks through each of these pieces.

The following diagram represents a simplified overview of the Architecture:

Why was AWS chosen?: AWS was used as the control plane for the whole infrastructure setup. Features such as the Parameter Store, Lambda Functions, Step Functions, S3 buckets, CloudWatch and the API Gateway have equivalent functionality in the other two major cloud providers, but since I already had more experience with AWS and had already earned multiple AWS certifications, it made sense to put into practice that knowledge.

In the beginning EC2 instances were considered, but were later discarded due to their low price/performance ratio.

Orchestration Core: SQS → Nomad Parameterized Jobs

The backbone of the system is AWS SQS. Here is a simplified diagram of how it works:

There are two queues, job_results and pending_jobs, both are of type FIFO in order to avoid message duplication.

The SQS poller for the pending_jobs queue, which is in itself a Nomad job runs every minute, converts the message read from SQS into a Nomad parameterized job. The parameterized job type allows dispatched jobs to be run using different configurations on the fly.

Here's how a message in SQS in the pending_jobs queue looks like:

{"Meta": {"domain": "tesla.com"}, "jobinfo": {"jobname": "amass"}}

Here is an example configuration for one of the Nomad jobs the automation runs, amass:

parameterized {
    meta_required = ["domain", "message_receipt"]
}

meta_required expects both the domain and the SQS message receipt (which is part of the message metadata, not stored as the actual message data), which gets deleted after the job is run. That message_receipt is used to reference the SQS message that was received and delete it, to avoid that it re-appears in the queue.

Here is the part of the main amass task that is run in Nomad. This is similar to how the job would be run via command line, and it allows us to access special configuration variables that the Nomad environment provides.

config {
  command = "amass"
  args = [
    "enum",
    "-passive",
    "-config", "local/config/config.ini",
    "-d", "${NOMAD_META_DOMAIN}",
    "-json", "../alloc/data/amass.json",
    "-log", "../alloc/data/amass.log",
  ]
}

${NOMAD_META_DOMAIN} allows us to retrieve the value for the domain parameter. e.g. tesla.com.

TL;DR: a message on the queue becomes a running job on whichever node has room for it, and when that job finishes, the results get saved and the next step gets queued automatically.

The steps below are just the mechanics of how that handoff happens:

• Read the messages from SQS. If there are no messages, the dispatcher stops here.
• Dispatch the message to the Nomad server via its API. The API path follows the following structure:

https://{server_address}/v1/job/{jobname}/dispatch

• The message is dispatched as a POST request, with the domain and message_receipt passed as the Meta parameters.

{
    'Meta': {
        'message_receipt': 'SQS_receipt_structure',
        'domain': 'tesla.com',
    }
}

• Once the Nomad server is aware of the job, it immediately attempts to allocate it in one of the hosts available in the pool that match the required system constraints (datacenter/minimum resources needed/type of node). If the job can't be allocated immediately, it stays in a Nomad (internal) queue until the job can be allocated.

• The job gets allocated and runs, after the main task (amass in this case) exits, there's another task executed to do the cleanup. In Nomad this is called a poststop task. Here is where the following actions get performed:

  • Both the .json and the log file get uploaded to S3 in a specific folder for this job name (amass) and job execution. Similar to: s3://bucket_name/amass/execution_id/
  • The new subdomains are stored in the SurrealDB database.
  • The SQS message for this job gets deleted.
  • The successful execution of the amass job gets reported to another SQS queue, the job_results queue.

The job_results queue is read by another Nomad job which converts the job that just run (amass run) into the next job that should be run in the pipeline. For example, an amass run triggers a dnsx (open source resolver) run afterwards.

Why was AWS SQS chosen?: This feature was chosen for its simplicity of use, since it's a managed message queuing service highly scalable and very cheap. For this use case it didn't bring any benefits to implement a message-broker system like RabbitMQ.

Why was Hashicorp Nomad chosen?: This is the most difficult choice to justify out of all the engineering decisions. Before I implemented this process I had zero experience with orchestration of workloads, and for a one person project Kubernetes was very intimidating due to the steep learning curve that it has. But even in retrospect implementing Kubernetes would not have offered any advantage besides scaling up and down the number of nodes/clients to process the work. Hashicorp Nomad allows running many instances of a program without requiring everything to be packaged as a container; it directly allows the orchestration of binaries, which sounded more than good enough at the time this choice was made.

Job Walkthrough: amass and httpx

To make this concrete, here's what happens end to end for two of the jobs in the pipeline.

amass — amass is an open source tool for OSINT (Open Source Intelligence). Among other features, it allows us to list subdomains for a given root domain querying free and paid APIs.

This job file is composed of a main task and a poststop task.

The main job, as previously described, receives the root domain and returns the list of unique subdomains it was able to find querying the API it is configured to use.

The poststop task inserts the new domains into the database, uploads the raw results into S3, and sends a message to the SQS queue of job_results in order to run the next program in the automation pipeline.

httpx — httpx is a "is a fast and multi-purpose HTTP toolkit that allows running multiple probes using the retryablehttp library" as stated in the repo. This is the default go-to tool in bug bounty to check if a remote HTTP/HTTPs is listening on a given port.

This job file is composed of three tasks, a prestart task, the main task, and a poststop task.

The prestart task, calls the database and retrieves the list of all the subdomains that have been marked as resolving. It excludes domains that are wildcards, since some resolvers respond with an IP even when queried with domains that are clearly made up.

The main job, does the actual job, opens an HTTP/HTTPs connection to a list of ports that have been configured, retries, connects to a local DNS resolver, and is greatly rate limited to avoid being blocked by WAFs.

The poststop task inserts the results into the DB, it inserts information into several tables and relates it to the subdomains table, since the relationship is one-to-multiple. A single subdomain can have multiple services listening on it. Afterwards, just like most of the other jobs, it uploads the raw results into S3, and sends a message to the SQS queue of job_results in order to run the next program in the automation pipeline.

Here are some of the jobs that are run. This is both a simplification and standard workflow in the Bug Bounty/Pentesting space.

Failure Handling

The retry mechanism is implicit rather than explicit: every job comes from a message on an SQS queue, and if a job doesn't complete, or the poststop task never runs to delete the message receipt, that message reappears once its visibility timeout expires, and the dispatcher picks it up again.

A dead-letter queue backs both queues: after enough failed delivery attempts, a message stops re-queuing and lands there instead. Any job that crashes multiple times, or has a malformed message that Nomad won't dispatch, gets pulled out for a human to look at rather than cycling forever.

Observability

CloudWatch collects logs across the pipeline, and alarms are configured to email me when errors in a given window cross a threshold. That catches the failures worth checking on, but it's alerting on an aggregate error count, not a view of pipeline health over time. I couldn't easily tell whether a specific job type had quietly stopped producing results unless I checked CloudWatch manually. Closing that gap is what the Grafana/Prometheus work described in What's Next is for.

Terraform

Three different terraform projects were created. But they can be divided in two groups:

Cloud scaffolding: Two projects belong in this category, one of them for Alibaba Cloud, and one for GCP. They are triggered from Gitlab CI/CD; that's where the terraform state gets saved. These projects describe the VPCs and their private IP ranges, Security Groups, Regions and Availability Zones to use in each of those two clouds.

They contain the code that sets up the ground floor for the rest of the infrastructure to run on. And since all they set up are free, it makes no sense to destroy and apply this infrastructure every time some work load needs to be run.

Here is the code to create the skeleton for GCP. It will first create /27 subnets in multiple regions and will also create the compute instance templates that the instance groups (autoscaling groups) will be based on.

Instance groups/autoscaling groups: This Terraform project is for GCP. This Terraform code is the one that gets up and running a group of, for example, 20 instances in 5 different regions (20*5 instances). It is being triggered from Terraform Cloud, which is a service offered by Hashicorp that allows me to make an API call via an API and apply or destroy the Terraform infrastructure and keep the Terraform state information in that cloud service.

This Terraform project had 20+ regions that it could launch the autoscaling groups on and every time it receives a call to apply the Terraform infrastructure, it randomly selects 5 regions and applies the changes.

Why was Terraform chosen?: It is the standard go-to tool for IaC (Infrastructure as Code), specially for a project that involves multiple cloud providers. It didn't make sense to learn each cloud's specific flavor of IaC. Using something like AWS CloudFormation makes sense to achieve very detailed fine tuning or for features that are not exposed in Terraform's AWS provider.

Why were GCP and Alibaba chosen?: GCP was chosen with the hopes of it having better IP reputation than other cloud providers commonly used for bug bounty such as Digital Ocean. Alibaba was also chosen for this reason and for pricing reasons. Alibaba is very competitive price wise.

Why was Terraform Cloud chosen?: At least at the time I was not aware of any other solution that would allow with such simplicity to perform an API call to apply (or re-apply) and destroy Terraform IaC with such ease.

Preemptible Instances

As stated previously in the Architecture Overview, the system uses preemptible/spot instances to run the workload. For those who are not familiar with the concept here is a small definition:

Preemptible/spot instances are spare compute capacity that cloud providers sell at a large discount (up to 90%) compared to on-demand instance pricing. In exchange for the lower cost, the provider can terminate the instance(s) with a few minutes warning.

Why were Preemptible Instances chosen?: Besides the steep discount they offer, these instances are perfect to run tasks that are repeatable and can run asynchronously (e.g., rendering jobs), and since the jobs to be run on this project are loaded from SQS, unfinished work re-appears in the queue after the visibility timeout expires, as discussed in the Failure Handling section.

Multi-Cloud Provisioning and Region Rotation

The main service used for this is AWS Step Functions. Step Functions could be defined as a managed service that helps users "build workflows, handle errors, and repeat failed tasks without writing complex custom code". It is perfect when one or more Lambda functions need to be run with some conditional logic/workflow in between.

The system needed to provision and tear down instances across providers dynamically. I used Step Functions to orchestrate the different steps necessary to provision instances in a cloud, and after a certain threshold of time, rotate the autoscaling groups to another region.

The next diagram is an example of the use of Step Functions to run this automation system. Here is this Step Function as ASL (Amazon States Language) code.

TL;DR: checks the queue roughly every 30 minutes, spins up GCP/Alibaba capacity if there are jobs in the queue, maintains the infrastructure up for up to ~5 hours if jobs keep arriving, and rotates the region every 30 minutes until it tears it down completely.

Step Functions Workflow

It consists of the following steps:

• First, a Lambda function is triggered which changes a variable in the Nomad Server that allows the SQS poller job to run.

• Pass state contains a simple JSON block that declares the number of machines to use in each region for each cloud. A key (acting as a variable) called retryCount is initialized to 0.

• SQS: GetQueueAttributes allows us to retrieve a specific attribute from an SQS queue. The attribute that gets fetched is "ApproximateNumberOfMessages". Which returns the number of messages in the SQS queue that keeps track of the jobs that need to be run and forwards that value to the next step.

• Afterwards there's a conditional block. Depending if there are more than 10 jobs to be run or not, one action or the other is taken.

• If there are enough jobs to be run, the clouds get started (or restarted if they were already running). AWS Step Functions allow calling other Step Functions from within them, when this step is reached, the execution flow jumps to that other Step Function.

• A wait/sleep section is reached, for 1800s the machines that were just launched are kept running.

• Another retryCount block is reached. If the value is less than 10, add one to the variable and go to "Get number of jobs", check again if there are jobs in the queue.

• If the retryCount value is reached, the shutdown sequence is started. This step checks if there is already an execution of the Step Function "ShutDownEverything" running and forwards that information to the next step.

• Another conditional block. If there was already an execution of ShutDownEverything (for example due to a manual trigger), the flow simply exits. And if there isn't an execution, the Step Function ShutDownEverything is triggered.

This workflow serves two purposes. One of them is to automatically enable auto scaling groups both in GCP and Alibaba that are automatically shutdown when they are no longer necessary, and the other purpose is that it brings IP rotation capabilities, because at least with Alibaba the process checks in which region are the current machines up and running, and avoids using the same region in the next iteration of the loop.

Having that many IPs available at once also matters for a second, separate reason: instead of running the full set of requests for one scan through a single source IP, individual requests get spread across the pool; 100s of servers each sending a handful of requests to a target instead of one server sending all of them. Each source IP stays under whatever rate limit or blocking threshold the target enforces, even though aggregate throughput across the pool stays high.

Why were AWS Step Functions chosen?: They were the best tool for the job since they are managed service, very low cost, and I am at least not aware of any other sensible alternative to achieve the same result. If this service did not exist it would have been necessary to run this workflow using some scripting language like Python but it would have been much more difficult to develop, maintain and handle errors without Step Functions.

Cost-Aware Region Selection and Auto Scaling Group Provisioning in Alibaba Cloud

Instead of using Terraform to provision the Auto Scaling Groups in Alibaba, Alibaba's SDK in Python was used for this task. The main reason to do it this way was that it allowed more control over which instance types to launch in each Region in a way that is more difficult to achieve with Terraform.

Because most providers use dynamic pricing, in Alibaba a library was developed to help select the best prices in a specific region.

For example, two types of machines might have the same capabilities with regards to CPU and memory (although not the same performance). But machine of type A might be 60% cheaper in the London region than machine B. Again, with similar specs. But the opposite situation can happen in the Los Angeles region.

Example: using Alibaba's SDK and this dynamic pricing feature within a Lambda function, the system can provision 300 instances, divided in Auto Scaling groups of 100 instances, at the 3 cheapest prices available. This will create auto scaling groups starting from the cheapest instances for a given list of instance types.

I have released this pricing matcher as a standalone tool. It will not create any autoscaling groups but for the given regions and instance types it will retrieve which regions and AZs have the lower pricing, either for spot instances or for normal instances.

Here is an example of this standalone tool:

Even in the same region, eu-central-1, and for the same instance type, there's a difference of 5 times in price between Availability Zone A or B with respect to C.

Golden Images and Configuration Management

The compute nodes present in the cloud are built from Packer-created golden images. The computers on-prem, the main Nomad server (and database) and the auxiliary controller instances were configured using Ansible roles. Packer builds the images in GCP and Alibaba by applying -nearly- the same Ansible roles that get applied to the on-prem computers.

Here is an example of how to install and configure Nomad using an Ansible role, and here is an example of the Packer configuration to create golden images in Alibaba.

The golden images were re-built when there were changes that affected the compute nodes. The auto-scaling groups in the cloud automatically pick the newest built image to launch the VMs.

The VMs are built on top of a Ubuntu minimal 2024 LTS base image.

Here's an example of the playbooks.yml file for the golden images:

- hosts: golden_image
  user: ubuntu
  become: yes
  become_user: root
  become_method: sudo
  roles: 
    - base
    - attack_tools
    - nomad
    - client

Base: installs the basic configuration, since the base image is Ubuntu minimal, many tools are missing.

Attack tools: copies the binaries to the VMs and installs some tools from private GitLab repositories using pipx.

Nomad: Installs nomad as a client.

Client: This is the code that wraps all the tools such as amass, nuclei, dnsx, etc, and also contains the code that is run in the special nodes as system jobs (such as fetching new jobs)

Both the Ansible pipeline (which applies directly to the VMs and on-prem machines that don't get destroyed after every run), and the golden images pipeline, use pipeline variables and files stored in a secure way in Gitlab (e.g. certificates). This is discussed in the section IAM and Secrets Management.

Why were Packer and Ansible chosen?: Packer and Ansible were chosen because they are cloud agnostic and they are the default go-to tools in the industry to achieve these objectives.

Private connectivity solution (Wireguard)

In order to deal with the many network ranges and different clouds and regions/zones of the different providers, it was necessary to interconnect the networks using a VPN.

Due to the simplicity to implement the technology and the low extra CPU and header data, Wireguard was chosen. Using a hub-and-spoke architecture.

The central server is the hub and the cloud instances and OpenWRT routers used on-prem act as hubs. Because the network ranges of the cloud for every region are already mapped, and the regions limited (excluded Chinese regions in Alibaba and expensive regions in GCP), Wireguard's client public/private keys and configuration files for each private IP had already been pre-generated.

Each instance just has to select the Wireguard client configuration that matches their private IP in order to get connectivity to the main server, and from there, to the rest of the clients. This is done in a Python script that runs in a Linux service triggered when the instance has booted.

Why was Wireguard chosen?: Wireguard was chosen due to the simplicity to get it up and running and the low extra CPU and header data.

Distributing Scan Traffic: HAProxy and Dynamic SOCKS5 Backends

As discussed in the section Multi-Cloud Provisioning and Region Rotation, to avoid being blocked by WAFs (Web Application Firewalls) it is important to distribute the security tests across as many IPs as possible, and in this system this task is handled by HAProxy acting as a forward proxy. Each compute node, which has its own public IPv4 address, runs a SOCKS5 proxy (microsocks), and those proxies register themselves as Nomad services as instances come up.

HAProxy reads that service list as its backend pool, so as capacity gets provisioned and torn down by the corresponding Step Function, the set of proxies it balances across updates automatically.

HAProxy's configuration file is created from within the Nomad's job template:

{{range nomadService "microsocks"}}
    server microsocks-{{ .Name }}-{{ .ID }} {{ .Address }}:{{ .Port }} check inter 2s rise 1 fall 1
{{ end }}

This statement acts as a for loop, when the nomadService list gets updated (a new microsocks service appears or disappears), new backend information is printed to the configuration file of HAProxy, and this in turn triggers a refresh of HAProxy with the new list of backends.

Scan jobs like httpx and nuclei route their outbound requests through a SOCKS5 proxy (HAProxy) located at 127.0.0.1:8080 rather than connecting directly to their target, which spreads a single job's requests across the whole pool of source IPs.

IAM and Secrets Management

IAM permissions were scoped by functionality rather than shared broadly: the credentials used to store results in S3 were strictly scoped to that bucket, and the credentials used to provision infrastructure, whether Terraform, Alibaba SDK calls, or golden image creation, were similarly scoped to what each task actually needed rather than given admin-level access.

There are secrets being managed in three distinct places:

Hashicorp Nomad, which accepts the use of variables which have Encryption at Rest. Nomad allows granular control in order for one job to not be able to access the variables of another.

AWS SSM Parameter Store, which allows for the secure storage of variables of type secret and it's free unlike AWS Secrets Manager.

GitLab CI/CD, used both for storing variables and files securely. The variables are accessible from the Pipeline (.gitlab-ci.yml) and used as environment variables, and the values can be masked to avoid that they show up in the logs of the build. And Gitlab CI/CD also allows uploading files securely. Those files are no committed to the repository and they can only be fetched when the pipeline gets built.

Why wasn't a solution like Hashicorp Vault used?: There wasn't a need for a centralized secrets store which would have added a learning curve, extra implementation, and above all, extra management. This is not a project open to the public and/or with third party users. But this doesn't mean that security didn't have to be taken into consideration.

Data Layer: SurrealDB

What is SurrealDB?: SurrealDB, as described by its creators, "is an innovative, cloud-native, multi-model database designed to act as a single, unified data layer for modern applications and artificial intelligence. It combines document, graph, relational, time-series, and vector search capabilities into one engine using its SQL-like query language, SurrealQL".

This is very powerful because it allows its users to query NoSQL documents with an SQL-like query language. Another feature, that removes the complexity of JOINs, is a special table type called RELATE. They allow to establish by-directional relationships between two tables.

As an example, the root_domain table (an entry might be tesla.com), via a simple syntax, allows the user to retrieve all the entries in the domain that have tesla.com as the root_domain entry.

And the way to create this by-directional relationship is very simple. As an example:

CREATE root_domain:['tesla.com'];
CREATE domain:['dev.tesla.com'];
RELATE domain:['dev.tesla.com']->has_root_domain->root_domain:['tesla.com'];

Here is a real example using another subdomain:

Between the subdomains for tesla.com and the root domain of tesla.com there's now a by-directional relationship established, and it becomes trivial to retrieve, starting from any of the edges of the Relation, the other piece of information.

The direction of the arrow determines if the Relation that wants to be retrieved goes from input to output or vice versa.

As it can be seen in the image, it's also substantially faster than performing a JOIN query. It took less than 100 ms to count that there are 2861 domain rows whose root domain is tesla.com in a table that contains 5.6 million entries.

Why was SurrealDB chosen?: SurrealDB was chosen due to the flexibility it offers. Especially because it was difficult to predict what the future needs would be. And it was the only DB I was aware of that allows the insertion of unstructured/document data, relational data and time series (useful to detect changes across different sweeps) while at the same time providing an SQL like language to query, not only the relational data, but also document and time series data.

Hetzner: Main Server and Controller

The main server of this infrastructure exists in the Hetzner cloud. This server performs two main functions, ideally these functions should be split into different machines, but due to the cost constraints a single server had to perform both functions.

The server acts as the Nomad server. As stated in the documentation, there should be a minimum of three Nomad servers to have some fault tolerance and still be able to reach a quorum between the remaining servers, but since I am the only user of this server this wasn't deemed necessary.

The server hosts the SurrealDB database. A backup of the database is performed automatically every 6 hours and gets uploaded to S3. To avoid that the stored data grows ad infinitum, S3 bucket lifecycle rules expire and delete the objects (files) after 10 days.

Here's how that simple configuration looks like:

A second instance, the controller, also exists in the Hetzner cloud, in the same private subnet as the server. This instance performs several functions:

The controller hosts a Gitlab runner. This is necessary for the pipeline that runs Ansible. Ansible is connecting privately to several on-premises machines that are also used in this project, and those machines, which are Nomad clients, don't allow public connectivity to them. The only way to run those playbooks in those machines is to run them from an instance that is already connected to the VPN.

The controller runs special Nomad jobs. Those jobs are not actual attack jobs, they are similar to cron jobs. Those jobs could be run in some of the on premise nodes in case of need, but conceptually it made more sense that those jobs were performed by a different machine (plus Hetzner has a higher SLA than an on-prem machine hosted at home). Some of those jobs that run in the controller instance are the SQS orchestration jobs, and a daily job that parses the new programs available in the bug bounty platforms I have access to.

Why was Hetzner chosen?: It offers the best price/performance ratio out of all the clouds tested. These instances run 24/7 so cost had to be taken into consideration. Using one extra cloud solution barely had any downside since only the compute service from Hetzner is used; there was no learning curve to adopt it.

CI/CD with GitLab

As stated in the previous section, GitLab CI/CD is used with a private runner.

GitLab is also used as the main repo for this automation project, with already 40+ different repositories, here are the different categories:

- Tools. The go tools get built in the CI/CD pipeline.

- Terraform code. Two of the Terraform projects, as previously discussed, run from Gitlab CI/CD.

- AWS Lambda functions. A single repo hosts all lambda functions and is capable of deploying only to the Lambda functions that have been modified. I decided to use a single repository since each function is generally just one file and the number of Lambdas could easily explode (17 Lambda functions as of now). Here is the pipeline configuration to achieve this feature with Gitlab CI/CD.

- AWS Lambda layers. One of them contains Alibaba's SDK. And others contain private libraries with functionality used by multiple Lambda functions.

- Nomad jobs, they get deployed to the Nomad server via the use of Nomad Pack, which allows the jobs to be created as templates with reusable stanzas/sections.

- Ansible and golden image creation (Hashicorp Packer). Each of them has their own repository and they both run from Gitlab CI/CD. As previously stated, Ansible runs in its own private runner.

Why was GitLab chosen?: In the past I used Azure DevOps and it doesn't offer the same flexibility as GitLab. Using GitHub Actions could have been another viable option, but being more familiar with GitLab and GitHub Actions not providing any meaningful benefit I decided to go ahead with GitLab.

Retrospective

This is the part I want to be honest about rather than just presenting a clean success story.

What worked well

The DevOps/Platform Engineering side of the project I'd consider a success. I already had theoretical depth in AWS from my Solutions Architect Professional certification, but this pushed that into practice, and I got exposed to GCP and Alibaba along the way. In effect, I was filling multiple roles at once: Cloud Engineer and internal Solutions Architect being the two most relevant for this writeup. It was a good -although, very challenging- experience to design and interconnect so many moving pieces together. There was also real engineering work behind some of the security tools I built or modified, though that's outside the scope of this writeup. It was very satisfying to be able to launch 300 instances in Alibaba in a matter of minutes in order to have a pool of 300 IPs for around $0.5 per hour and be able to run thousands of testing jobs per hour in them.

What I'd do differently

If I had to do it over again, I would use Kubernetes instead. Not because it has any features that I needed that Nomad was actually lacking, but to gain real world experience using the default go-to tool when a system requires orchestration.

Another change is that I'd consolidate spot instances onto a single cloud instead of splitting across two. Probably Alibaba due to its very competitive pricing and the elevated number of instances that it allows users to launch (hundreds in one go). GCP has much more modest limits and requires constantly to be opening tickets to increase the default per region limits (8).

Lastly, there were some over-engineered tools that in retrospect were a time sink. As an example: to deduplicate near-identical web pages, I had a job open a Chrome window through Playwright for every host httpx flagged as live and fingerprint (hash) both the rendered page and its favicon; a later job then discarded the duplicates, reducing the number of targets tested further down the pipeline. For some root domains this helped collapse thousands of hosts down to a few dozen. The approach worked, but a full browser per host was slow and resource-hungry, and it became one of the pipeline's worst bottlenecks.

Why the economics didn't work out

Being strong at automation and strong at vulnerability research are different skill sets, and I had invested far more in the former. There were also some external factors, bug bounty has a public surface and a hidden surface (private programs/companies). Security researchers doing well in the bug bounty space are getting most of their rewards in private programs, which have a limited number of security experts looking into them. The public surface is contested by an estimated 100,000+ researchers, and only after finding medium-to-high security vulnerabilities in public programs are researchers invited to those private programs. This creates a structural chicken-and-egg problem.

What's Next

About this project:

• I will be looking to collaborate with a security researcher who has created a PoC (Proof-of-concept) or an attack chain that can be used to prove exploitation in a system, and using my automation pipeline integrate the job and scan for that vulnerability across all the bounty programs/domains/live hosts that I have identified while, obviously, splitting the reward with the security researcher.

• I'm currently adding a Grafana/Prometheus observability layer on top of this system to get some dashboards for pipeline health, and will update this post once that's live.

On a personal note:

• I'm working towards the CKA (Certified Kubernetes Administrator) certification to stay current with market demand for DevOps/DevSecOps Engineers. As of late August 2026, I'm looking for a remote Cloud Engineer or DevSecOps Engineer role.

How to reach out?

If you think my skillset could be useful, feel free to reach out to me via LinkedIn or via email at alvaro[at]agarfer[dot]dev

---

Lastly, here are the code examples of selected subsystems that I've shared in my repo.