From Scheduled Ansible to Continuous Reconciliation on Linux Hosts

Danylo Kochetov — Fri, 06 Nov 2026 16:05:43 +0000

A scheduled Ansible playbook is a reasonable way to manage a small Linux fleet.

It is understandable, widely supported, and easy to introduce:

*/15 * * * * ansible-playbook -i inventory production.yml

Every 15 minutes, the runner connects to the hosts and applies the playbook again.

This works. But it answers a narrower question than many teams eventually need.

It tells us what happened during the last playbook run. It does not necessarily tell us whether the fleet matches Git right now.

The gap between two runs

Imagine that a playbook runs successfully at 10:00.

At 10:04, someone edits /etc/nginx/nginx.conf while investigating an incident.

Until the next run:

Git contains one value;
the server contains another value;
no deployment has failed;
the configuration has drifted;
the team may not know about it.

The next scheduled run may repair the file. But the operating model remains periodic:

Run playbook
    ↓
Wait
    ↓
Something may drift
    ↓
Run playbook again

A successful cron job also does not automatically provide a current fleet-level answer to questions such as:

Which hosts are converged now?
Which resource is different from Git?
When did the drift begin?
Did a host fail to apply the latest revision?
Is the service healthy after the change?
Who approved or initiated the reconciliation?

It is possible to build these capabilities around Ansible using CI jobs, logs, monitoring, AWX, additional databases, and custom automation.

The question is whether maintaining that control loop is where the team wants to spend its time.

A different operating model

I built hostcd around continuous reconciliation.

A control plane reads the desired state from Git. A lightweight agent on each host connects outbound to the control plane, receives the accepted desired state, compares it with the host, applies changes, and reports the result.

Git repository
       ↓
hostcd control plane
       ↓ outbound agent connection
Linux hosts
       ↓
packages, files, services, users, cron and sysctls

Instead of waiting for the next scheduled job, each application has an explicit state:

Synced or OutOfSync;
Healthy, Progressing, Degraded, or Missing;
the Git revision accepted by the host;
the resource that differs;
the result of the last reconciliation.

The difference is not YAML versus YAML. It is periodic execution versus a continuously observed control loop.

This does not require replacing Ansible

“Replace Ansible” is usually the wrong migration goal.

Ansible is still useful for:

creating and provisioning machines;
installing the hostcd agent;
one-off administrative operations;
network or cloud orchestration;
unsupported resource types;
workflows that intentionally run only on demand.

A more practical division is:

Ansible:
provision the machine and perform one-off orchestration

hostcd:
continuously defend the configuration that must remain true

Both tools can coexist, but they should not manage the same resource at the same time.

If Ansible writes a file every 15 minutes while hostcd continuously reconciles that file to a different value, the two systems will fight each other.

Ownership must be explicit.

Start with one small scope

Do not begin by translating the largest production playbook.

Choose:

one to three non-critical hosts;
one package;
one configuration file;
optionally one service;
resources whose expected state is easy to verify.

In my pilot, I created an application called ansible-migrated.

It manages two resources on one Ubuntu host:

the htop package;
/etc/ansible-demo.conf.

The application reports one of one hosts Synced and Healthy. Its reconciliation history records the accepted Git revision and the automated sync result.

This is intentionally unimpressive. A migration test should be easy to understand and difficult to misinterpret.

A safe migration sequence

1. Record the existing state

Before changing ownership, record:

the playbook and role responsible for the resources;
the current values on the target hosts;
the cron or CI schedule;
the expected rollback procedure;
any variables or templates that affect the result.

The existing system remains the reference during the first comparison.

2. Describe the same state in hostcd

Create the hostcd application and target only the selected hosts.

Do not enable destructive behaviour until the preview matches expectations. Review the package, file, service, user, and other resource differences before applying them.

3. Install the agents

The agents connect outbound to the control plane. Regular operation does not require opening inbound agent ports or allowing the control plane to SSH into every managed host.

Ansible can remain responsible for installing the agent during this stage.

4. Compare without creating competing writers

Run hostcd in a non-destructive or review-oriented mode first.

Confirm that:

the expected resources are discovered;
the calculated differences are correct;
the selected hosts are correct;
no unrelated resource is included;
secrets and generated files are handled appropriately.

5. Remove overlapping Ansible ownership

Once the desired states match, disable the relevant Ansible tasks for the pilot scope.

Do not disable the entire playbook if only two resources are moving. Remove ownership incrementally.

6. Reconcile and verify

Apply the accepted Git revision and verify:

package state;
file content and permissions;
service state;
application health;
agent reconnect behaviour after restart.

The UI should report the application as Synced and Healthy, but the verification should also inspect the real host.

7. Introduce controlled drift

For a non-critical test file, make a manual change on the host.

Verify that hostcd:

detects the difference;
identifies the affected resource;
reports the application as out of sync;
restores the Git-managed value when self-heal is enabled;
records the reconciliation result.

Do not run destructive drift tests against infrastructure anyone depends on.

8. Observe before expanding

Keep the pilot small for several days.

Watch for:

repeated false drift;
package-manager behaviour;
service restart loops;
repository authentication failures;
host reconnects;
incorrect targeting;
resources that still need to remain under Ansible.

Only then move the next scope.

What about rollback?

With GitOps, the cleanest rollback is usually another Git revision.

Revert the problematic commit, review the resulting difference, and reconcile the accepted revision.

This keeps the desired state, approval history, and actual host state aligned.

A UI-level rollback can be useful, but it needs a clear interaction with automated sync. If the application continuously follows main, rolling back only the live host without changing the desired revision will cause the controller to move forward again.

Rollback must change or temporarily override the desired target, not merely copy old files onto a machine.

Operational trade-offs

Continuous reconciliation is not automatically better for every resource.

It introduces an always-running agent and control plane. A bad desired state can be propagated continuously. Automated self-heal can undo an emergency manual change. The agent may require root privileges to manage operating-system resources.

That means the system needs:

narrow targeting;
dry runs and previews;
approval gates for sensitive applications;
audit history;
per-host authentication;
signed releases;
backup and recovery procedures;
an explicit emergency process for pausing reconciliation.

The control loop is valuable only if its authority is bounded and observable.

When scheduled Ansible is enough

Keep the existing approach when:

the fleet is very small;
configuration changes rarely;
manual drift is not a meaningful risk;
current-state visibility is unnecessary;
the playbook is primarily provisioning;
another monitoring system already provides sufficient evidence;
operating an additional control plane would add more complexity than value.

A cron job is not inherently wrong. It becomes insufficient when the team starts building a second system around it to answer questions the playbook runner was not designed to answer.

The actual migration goal

The goal is not to replace one YAML format with another.

It is to move from:

“The playbook succeeded this morning.”

to:

“These hosts match this Git revision now.
This resource drifted.
This reconciliation corrected it.
This operator approved it.”

For some teams, scheduled Ansible remains enough.

For teams that need continuous evidence and automatic drift correction, the safest migration is not a rewrite. It is one package, one file, one host, and one clearly transferred ownership boundary at a time.

The migration guide is available at:

https://hostcd.dev/guides/ansible-cron-to-hostcd/

hostcd itself is self-hosted and free for up to 10 hosts:

https://hostcd.dev/

DEV Community: Danylo Kochetov