DEV Community

Cover image for From SSH Config to Running CDC Pipelines: Host-Based Deployment for Debezium Platform
divyanshu_Kumar
divyanshu_Kumar

Posted on

From SSH Config to Running CDC Pipelines: Host-Based Deployment for Debezium Platform

#opensource #automation #beginners #distributedsystems

For Google Summer of Code 2026, I worked on adding host-based pipeline deployment to the Debezium Platform.

Debezium is a change data capture platform. It observes changes in databases and turns them into events that downstream systems can consume. The Debezium Management Platform provides a control plane for defining sources, destinations, transforms, and pipelines. Its Conductor component orchestrates deployments, while Stage provides the user interface. Debezium’s Platform documentation explains these concepts in more detail.

Before this project, the Platform deployment path was centered on Kubernetes and the Debezium Operator. That model is useful for teams already running Kubernetes. But some teams run their databases and services on VMs, traditional servers, or on-premise infrastructure.

The goal of my project was to add a host-based deployment path without removing the existing operator-based path.

The work is now merged upstream.

The idea

The Platform should be able to deploy a Debezium Server pipeline in two ways:

pipeline.deployment.mode=operator
Enter fullscreen mode Exit fullscreen mode

or:

pipeline.deployment.mode=host
Enter fullscreen mode Exit fullscreen mode

Operator mode remains the default for backward compatibility.

When host mode is selected, the Conductor discovers remote hosts through SSH configuration, provisions them with Ansible, chooses a host for each pipeline, and manages the resulting Debezium Server container.

For host container lifecycle operations, there are also two runtime choices:

pipeline.host.container-runtime=ansible
Enter fullscreen mode Exit fullscreen mode

or:

pipeline.host.container-runtime=agent
Enter fullscreen mode Exit fullscreen mode

Ansible remains the default runtime. Agent mode is explicitly selected when the Host Agent should handle container lifecycle operations through HTTP.

This is important: Ansible is still used for provisioning in both modes. The Agent does not replace Ansible as the machine-setup tool. It replaces repeated lifecycle commands such as deploy, stop, start, status, logs, and undeploy after the host is ready.

Architecture 

flowchart LR
    UI[Platform Stage or API] --> C[Conductor]
    C --> D[Host Pipeline Controller]
    C --> W[SSH Config Watcher]
    W --> P[Ansible Provisioner]
    P --> H[Remote Host]

    D --> R{Container runtime}
    R -->|ansible| P
    R -->|agent HTTP| A[Host Agent]

    H --> A
    A --> Docker[Docker Engine]
    Docker --> S[Debezium Server container]

    C --> Poller[Status poller]
    Poller -->|Ansible or Agent HTTP| R

The diagram is intentionally split into two responsibilities:

  • Ansible prepares the host.
  • The selected runtime manages the pipeline container.

Host discovery starts with SSH configuration

Rather than creating a separate host-registration API, host mode uses a file operators already know:

~/.ssh/config
Enter fullscreen mode Exit fullscreen mode

For example:

Host cdc-host-1
    HostName 192.168.1.20
    User deploy
    IdentityFile ~/.ssh/cdc-host-key
Enter fullscreen mode Exit fullscreen mode

SshConfigWatcherService watches this configuration and reconciles it with the Platform database.

When it finds a new host entry, the Platform creates host state and starts provisioning. When an entry is removed, the Platform can mark that host as removed.

The watcher also has a scheduled reconciliation fallback. This matters because file-system watch events are not equally reliable on every environment or mounted filesystem. The fallback gives the system another chance to detect changes even if a watch event is missed.

One important implementation detail came from review: @LookupIfProperty only affects CDI Instance<T>.get() lookups. It does not disable @Observes StartupEvent or @Scheduled methods. Therefore, host-specific scheduled and startup logic has an explicit runtime guard:

if (!"host".equals(deploymentMode)) {
    return;
}
Enter fullscreen mode Exit fullscreen mode

Without this guard, the SSH watcher could try to read ~/.ssh/config even when Conductor runs in operator mode.

Provisioning the host with Ansible

After discovery, HostProvisioningService delegates to AnsibleHostProvisioner.

The provisioning playbook prepares a remote host for Debezium Server deployments. Depending on the runtime configuration, it can also install the Host Agent.

For Agent mode, the playbook:

  1. Installs the required host prerequisites.
  2. Obtains the Host Agent artifact through Maven coordinates.
  3. Creates the Agent configuration and systemd service.
  4. Supplies a per-host bearer token through the service environment.
  5. Starts the Agent service.
  6. Prepares Docker for Debezium Server containers.

Using a Maven artifact is important. Earlier local development used a JAR from the developer checkout. Review feedback correctly pointed out that a real playbook should not depend on a contributor’s local file path. The final approach retrieves the Agent artifact from a Maven repository. For local snapshot testing, I used a local Nexus repository. In a normal environment, Maven Central is the default unless another repository is configured.

Selecting a host and deploying a pipeline

When a pipeline is deployed, HostPipelineController coordinates the work.

At a high level, the flow is:

Pipeline deployment request
  -> map pipeline configuration to Debezium Server properties
  -> choose a ready host
  -> allocate a port
  -> create deployment state
  -> ask selected runtime to deploy the container
  -> poll until the container is running
Enter fullscreen mode Exit fullscreen mode

The deployment state is owned by Conductor. The Host Agent does not own pipeline orchestration records. It only owns local facts and local actions: Docker containers, configuration files, data directories, logs, and status information.

This separation matters. Conductor knows concepts such as:

  • DEPLOYING
  • RUNNING
  • FAILED
  • CONFIG_DRIFT

The Agent only knows local facts such as:

  • whether a container exists
  • whether it is running
  • the hash of its deployed configuration file

Keeping those responsibilities separate prevents the standalone Agent from becoming coupled to Conductor’s database model.

The Host Agent

The Host Agent is a standalone Quarkus application packaged as an uber-JAR. It runs as a systemd service on a remote host.

Its job is deliberately small:

HTTP request
  -> validate request
  -> authenticate request
  -> run local Docker command
  -> return local result
Enter fullscreen mode Exit fullscreen mode

The Agent exposes these endpoints:

Method Endpoint Successful response Purpose
POST /api/agent/deploy 202 Accepted Writes configuration and begins container deployment
POST /api/agent/undeploy/{containerName} 204 No Content Removes a deployed container and local files
POST /api/agent/stop/{containerName} 204 No Content Stops a container
POST /api/agent/start/{containerName} 204 No Content Starts a container
GET /api/agent/status/{containerName} 200 or 404 Returns running state and config hash
GET /api/agent/logs/{containerName} 200 Returns container logs

202 Accepted on deploy is intentional. Docker might need time to pull an image or start a container. The Agent accepts the work and starts the Docker command asynchronously. Conductor’s status poller later confirms whether the deployment reached RUNNING.

The Agent uses Java virtual threads for this asynchronous Docker work, keeping the REST request responsive while the command runs.

Authentication and input validation

Each provisioned host can have a bearer token. Conductor stores the token as host deployment metadata, while the Agent receives it through the AGENT_TOKEN environment variable in its systemd service.

The Conductor REST client automatically sends:

Authorization: Bearer <token>
Enter fullscreen mode Exit fullscreen mode

The Agent’s AgentTokenFilter validates the token before the REST resource executes.

The comparison uses MessageDigest.isEqual() instead of a normal string comparison. This is a constant-time comparison approach that avoids leaking how much of a token matches through response timing.

For local development, authentication can be disabled when no Agent token is configured.

The Agent also uses Jakarta Validation for request safety:

  • container names must match a safe pattern
  • Docker image names cannot be blank
  • ports must be between 1 and 65535
  • deployment configuration content must be present

For example, a container name such as ../outside is rejected with HTTP 400 before the Agent touches the filesystem or Docker.

OpenAPI documentation

The Agent now generates an OpenAPI contract at:

/q/openapi
Enter fullscreen mode Exit fullscreen mode

The document describes:

  • every lifecycle endpoint
  • request schemas
  • validation constraints
  • successful responses
  • invalid-request responses
  • authentication failures
  • missing-container responses
  • runtime failures

This is useful for both humans and software. A maintainer can inspect the API through Swagger UI in development, while a future client can rely on the generated contract instead of reverse-engineering Java code.

Polling and retry behavior

HostDeploymentStatusPoller runs on a schedule and checks deployments in DEPLOYING or RUNNING state.

It supports both runtimes:

Ansible runtime
  -> check Docker state through Ansible

Agent runtime
  -> call GET /api/agent/status/{containerName}
Enter fullscreen mode Exit fullscreen mode

The state-transition logic is shared. Only the source of the status information changes.

The poller uses Debezium’s existing RetryingRunnable utility for transient failures. This was another useful review lesson: when a mature codebase already has a utility that solves a problem, reuse it instead of introducing a small custom retry implementation.

A temporary Agent connectivity failure can be retried. A definite stopped or missing container is not retried as if it were a network failure.

Verification

I verified the Host Agent path with focused unit and wiring tests, plus manual end-to-end testing on a remote test host.

The manual flow covered:

local Maven/Nexus artifact publication
  -> Ansible provisioning
  -> systemd Agent startup
  -> bearer-token protection
  -> request validation
  -> deploy
  -> status
  -> logs
  -> stop
  -> start
  -> undeploy
Enter fullscreen mode Exit fullscreen mode

I also verified both valid and invalid Agent API behavior, including:

  • 401 Unauthorized when authentication is missing
  • 404 Not Found for a valid but missing container
  • 400 Bad Request for an unsafe container name

What I learned

This project taught me that infrastructure work is mostly about boundaries.

A clean boundary between Conductor and the Host Agent made the system easier to reason about. Conductor decides what should happen. The Agent performs local Docker work. Ansible prepares machines. The poller observes state over time.

It also reminded me that existing project patterns matter:

  • use the project’s logger conventions
  • use the project’s retry utility
  • respect CDI lifecycle behavior
  • preserve backward-compatible defaults
  • avoid local-only deployment assumptions
  • test the real deployment path, not only mocked code

I am grateful to Mario Fiore Vitale and Giovanni Panice for the technical guidance and detailed review throughout the project. Their feedback made this implementation substantially better.

Links

Thank you for reading. If you found this useful, I’d love to hear your thoughts in the comments.

Top comments (0)