A production release was blocked by a nearly full disk. The useful part of the AI workflow was not a clever command—it was keeping the investigation on the affected host while preserving a human approval boundary.
Disclosure: I’m affiliated with KeySync, the remote-access product used in this incident.
At 4:09 p.m. on August 31, I triggered the production release for version 0.1.115.
Builds started for macOS, Windows, Linux x64, Linux ARM64, and the standalone CLI. Installation, startup, signing, and notarization checks passed one after another.
Thirty-nine minutes later, all five build groups had succeeded.
But the release did not continue.
The final CI gate failed:
The server has only 6.5 GiB free. A safe release requires 17.4 GiB.
Every candidate artifact had passed validation, yet production was blocked by a disk that was almost full.
The most dangerous response was not moving slowly. It was moving too quickly.
Deleting an old-looking directory might recover enough space, or it might remove a live database, a model, or the rollback package we would need ten minutes later. Expanding the disk would be easy, but if a retention job had silently stopped working, more capacity would only postpone the next incident.
I needed three answers:
- What was actually consuming the space?
- Which data was safe to remove, and which data was still in use?
- Was this a one-time capacity problem, or was a process continuously generating garbage?
Two clicks from a failed CI gate to the affected server
I did not start by looking up an IP address or searching for an SSH configuration and key.
The production Linux server was already connected to KeySync. I opened the device list, selected the server, and clicked Remote Open. DeepSeek Harness (DSH), which was installed on that host, opened in the browser.
The same production Linux host, online with DSH installed and available to open remotely.
From the failed CI gate to the correct production server, the only decisions were selecting the device and opening its DSH instance.
KeySync handled the first part of remote diagnosis: getting the agent to the correct machine. DSH handled the second part: continuing the investigation once it was there.
The copy-and-paste version would have taken at least six round trips
In the usual workflow, I would log in, run df, and paste the output into an AI chat.
The AI would ask for du, so I would run that and paste another result. Once Docker and KeySync appeared near the top, the next requests would be container state, mounted volumes, directory details, historical files, and the cleanup scripts.
The investigation would look like this:
run df -> copy output -> wait for an answer
run du -> copy output -> wait for an answer
inspect Docker -> copy output -> wait for an answer
expand large directories -> copy output -> wait for an answer
read cleanup scripts -> copy output -> wait for an answer
run validation -> copy the result again
It looks like the human is investigating. In practice, the human is acting as a remote keyboard for the AI.
Because DSH was running on the target host, it could inspect the disk, directories, processes, containers, and scripts within the permissions I granted.
My first instruction was deliberately not “clean this up.” It was:
Perform a read-only investigation only. Do not delete anything, restart services, or modify configuration. First determine why the disk is full. Separate live data, recoverable duplicates, and backups that must not be removed. Then decide whether the right answer is cleanup or expansion.
I did not prescribe a sequence of commands. I stated the objective and the boundaries. The agent chose each next command from the result of the previous one.
One objective, a continuous investigation
DSH first used df to verify that this was not a false CI alert: the root filesystem was genuinely at 98%.
It then continued narrowing the scope:
df confirms the root filesystem is at 98%
-> du narrows the problem to KeySync and Docker
-> container and volume inspection rules out blind Docker cleanup
-> KeySync directories reveal 77.85 GiB of staged uploads
-> historical version directories show abnormal retention
-> the release cleanup script is inspected
-> real directory names are compared with the matching rule
-> a one-character mismatch is found
The time saving did not come from typing fewer commands. It came from keeping the investigative context intact. The agent could see the last result and immediately test the next hypothesis.
The first branch: do not prune Docker
This Ubuntu production server had been running for more than a year. It hosted KeySync, databases, object storage, monitoring, internal services, and several containerized applications.
Docker had 37 containers, and every one of them was running. Some test environments had been alive for ten months.
So “Docker is large” was not enough evidence to run docker system prune.
After checking mounts and volumes, DSH found that the largest volumes were still attached to services. There were no stopped containers that could be reclaimed safely. System logs were not the main consumer, and inode exhaustion was not the problem.
The important disk consumers were:
| Major consumer | Usage at the time |
|---|---|
| KeySync upload staging | 77.85 GiB |
| Docker data, images, and containers | 57.08 GiB |
| KeySync public downloads | 34.42 GiB |
| KeySync release and candidate archives | 33.96 GiB |
| KeySync backups | 18.50 GiB |
| Ollama models | 16.82 GiB |
The disk had a total capacity of 294.9 GiB, with 276.3 GiB used. The system reported 98% usage and only about 6.4 GiB available to a normal process.
The first conclusion was clear: immediate expansion was unnecessary, and active Docker or database data should not be touched. The 77.85 GiB upload-staging directory deserved the next investigation.
A nearly invisible error in the retention rule
If I had pasted only the du output into a chat, the AI could have told me that the directory was large.
But DSH was still on the machine. It could follow the evidence from the large directory to its file structure, and from the file structure to the release script responsible for cleaning it.
Production upload directories were named like this:
client-0.1.113
client-0.1.114
client-0.1.115
The cleanup rule matched this instead:
client-v*
The configuration said to keep only the newest two versions, and the cleanup script completed without an error. But it never matched the real directories.
No alert fired. No red status appeared. Fifteen upload sets accumulated over time.
There was a second problem: the public-download cleanup script scanned only the top level. Old installers under the Windows, macOS, and Linux subdirectories were never considered.
All the tests had passed. The test fixtures used names such as client-v..., exactly matching the incorrect rule, and never represented the real client-... production convention.
One letter—v—turned “keep the latest two” into “never clean anything.”
Finding deletable data did not mean deleting it
DSH remained inside the read-only boundary after identifying the likely cause.
It selected three upload directories for which complete release copies already existed elsewhere, then calculated hashes for the files in both locations. All 5,487 files matched.
Only then did it recommend a concrete action:
Delete only the three verified duplicate upload directories. Expected recovery: approximately 15.4 GiB. Release artifacts, databases, and backups are outside the scope of this operation.
This distinction matters. Allowing an agent to enter a production server is not the same as allowing it to delete production data.
The investigation could proceed continuously. Destructive action still required a human decision about scope and consequences.
After I approved that exact scope, the cleanup removed only those three duplicate directories.
Available space increased from 6.41 GiB to 21.84 GiB. The other 39,392 protected files, 11 business-configuration tables, database backups, and deployed versions were unchanged.
At 5:22 p.m., free space exceeded the 17.40 GiB release budget and CI resumed. Ten minutes later, eight classes of downloadable packages, six platform-update configurations, and 24 web files passed validation. Version 0.1.115 went live.
A green release did not end the investigation
Recovering 15.4 GiB solved the immediate release failure, but it did not answer why retention had failed or whether the directory would keep growing.
In the same DSH conversation, I asked a follow-up:
Why is KeySync using so much disk space? Find the process that will cause it to continue growing.
The conversation still had the server’s directory layout, container state, release paths, and scripts in context. There was no need to reconstruct the incident in a new chat.
Using the real production naming convention, DSH reproduced the bug: with four upload directories present and retention set to two, the script deleted none of them.
It also found nine independent copies of the same Windows portable package across three release locations. Their hashes were identical, and together they consumed 1.62 GiB.
The real problem was not an undersized disk. Release files were duplicated, and the automatic cleanup rules did not cover the names and directory structure used in production.
A one-line fix required much more than a one-line safety process
The matching rule needed a one-line change. Production remediation required much more:
- Add ten focused tests covering old and new naming conventions, retention counts, protection of the current version, unrelated directories, and symbolic links.
- Check 20,083 files in eleven historical upload sets.
- Create recoverable deduplicated backups for 1,512 files whose contents were not duplicated elsewhere.
- Preserve approximately 7.89 GiB of unique data.
- Limit the operation to upload staging instead of opportunistically cleaning public downloads, Docker, databases, or model files.
After the rule passed its tests and the recovery data was verified, I approved the final execution scope.
Eleven historical upload sets were removed, leaving only the two newest. The operation recovered 48.46 GiB, bringing sampled free space to approximately 63.19 GiB.
DSH then verified the current downloads, update feeds, web files, and business configuration on the same host. Protected files were unchanged except for the expected hard-link differences.
The result was a complete operational chain:
detect the anomaly -> rule out unsafe shortcuts -> locate the root cause -> control risk -> wait for authorization -> execute -> verify
Space the agent correctly refused to reclaim
The investigation also found several running containers with names beginning with test-. Some had been running for about ten months, alongside even older testing and tooling services.
That exposed a separate lifecycle problem: temporary deployments had gradually become part of a production server without a consistent retirement process.
But old does not mean safe to delete.
The containers were running, and some of their volumes contained real data. DSH classified them as systems requiring owner review, not as immediately reclaimable garbage.
Each service needs its dependencies, data value, and shutdown window confirmed before migration, archival, or deletion. Otherwise, an attempt to improve a disk-usage number simply trades a capacity risk for data loss.
Where remote agents are useful—and where authority should stop
If I already know the exact command I need, SSH remains simple and reliable.
The strongest case for a remote agent is the unknown phase of an incident: a Linux host is unhealthy, but nobody yet knows whether to start with logs, storage, containers, processes, configuration, or source code.
In this workflow, KeySync answers “how does the agent reach the correct machine?” DSH answers “once it is there, how does it continue the investigation?”
The convenience is that I do not need a separate remote-operation setup for every Linux host. The intelligence is that the agent does not wait passively for the next copied fragment of evidence.
The safety model is equally important:
- Start with a read-only objective and explicit prohibitions.
- Let the agent choose investigative commands based on evidence.
- Keep deletion, restarts, and configuration changes behind human approval.
- Define exact targets and expected effects before approval.
- Verify protected data and service health after execution.
My view is that agents operating directly on remote hosts will become an increasingly common part of operations work. The concern is equally obvious: a single incorrect command on a production system can cause an outage or irreversible data loss.
Would you allow an AI agent to investigate or maintain production systems remotely? What safeguards would you require—read-only credentials, restricted shells, sudoers policies, command allowlists, sandboxes, approval gateways, snapshots, or something else?
Try the workflow
- Download KeySync to connect a Linux device and remotely open its DSH instance.
- Visit the DeepSeek Harness website for the official introduction, installation instructions, and open-source project.
Top comments (0)