DEV Community

Cover image for Machine0's Agent-First VM CLI: Per-Minute Billing and Persistent State for Long-Horizon Workflows
mech.app
mech.app

Posted on Originally published at mech.app

Machine0's Agent-First VM CLI: Per-Minute Billing and Persistent State for Long-Horizon Workflows

Machine0 is a YC-backed CLI that provisions persistent VMs billed by the minute, explicitly designed for agent workloads that run for hours or days. The product exposes a gap in existing infrastructure. Serverless platforms charge per invocation and time out after 15 minutes. Container orchestrators assume stateless workloads. Traditional VMs bill hourly with manual provisioning workflows.

The HN discussion (83 points, 44 comments) focused on cost controls and security isolation for agent-driven infrastructure, particularly the risks of giving agents autonomous provisioning access.

Agents that scrape datasets, train models, or run multi-step research workflows need compute that persists across crashes, resumes from snapshots, and bills only for active minutes. Machine0's CLI lets agents call machine0 new mybox to spin up a VM, machine0 suspend mybox to freeze state and stop billing, and machine0 resume mybox to continue. This is infrastructure designed around the agent as the operator, not the human.

Security Model for Agent-Driven Infrastructure

Machine0 VMs are single-tenant. Each VM runs on dedicated vCPUs with no noisy neighbors. For agent workloads, this isolation model prevents specific threats that multi-tenant container platforms face. An agent that loads API keys into memory or writes credentials to /tmp cannot have those secrets leaked to another tenant's process through shared memory pages or kernel vulnerabilities. The hypervisor enforces process isolation at the VM boundary, not just the container boundary.

Credential Handling and Exfiltration Risks

The CLI authenticates via API key, stored in ~/.machine0/config. The agent running in shared environments (CI runners, multi-user notebooks) should use short-lived tokens or inject credentials via environment variables instead of writing them to disk.

Snapshots introduce a credential exfiltration vector. When an agent snapshots a VM, the snapshot includes the entire disk, including any secrets written to /tmp, /var/log, or application directories. If the agent shares the snapshot (for cloning or backup), those secrets travel with it. Mitigation strategies:

  • Store secrets in memory only, never on disk
  • Use a secrets manager (HashiCorp Vault, AWS Secrets Manager) and fetch credentials at runtime
  • Scrub sensitive files before snapshotting: machine0 exec <vm> "shred -u /tmp/api_key.txt"
  • Verify snapshot encryption with Machine0 support before storing sensitive data

Cost-Based Denial of Service

Per-minute billing creates a new attack surface. An agent with provisioning access can spin up expensive GPU VMs and run them indefinitely. A compromised agent or a buggy orchestrator loop could provision 100 H100 instances at $0.836/hr each, costing $83.60/hr or $2,006/day.

Mitigation requires multiple layers:

Account-level quotas: Set a maximum number of VMs per account (Machine0 does not document quota APIs, so this may require manual coordination with support).

Budget alerts: Monitor total spend per hour and kill all VMs if spend exceeds a threshold. Implement this in the orchestrator, not the agent.

Approval gates: Require human approval for GPU instances or VMs larger than a certain size. The agent requests provisioning, the orchestrator queues the request, and a human approves or denies.

Time-to-live (TTL) enforcement: Every provisioned VM gets a TTL (e.g., 4 hours). The orchestrator force-suspends the VM when the TTL expires, regardless of task status.

Lateral Movement Between Agent-Provisioned VMs

Machine0 VMs within the same account can reach each other over the public internet using their assigned IPs. If an agent provisions 10 VMs for parallel work and one VM is compromised (via a vulnerable dependency or a prompt injection attack that writes malicious code), the attacker can pivot to other VMs in the fleet.

Network segmentation strategies:

Firewall rules: Each VM should run iptables or nftables to block all inbound traffic except SSH from known IPs. The agent can configure this during provisioning:

machine0 new worker-1 --size large

# Replace ORCHESTRATOR_IP with your orchestrator's public IP
machine0 exec worker-1 "iptables -A INPUT -p tcp --dport 22 -s $ORCHESTRATOR_IP -j ACCEPT"
machine0 exec worker-1 "iptables -A INPUT -j DROP"
Enter fullscreen mode Exit fullscreen mode

Overlay networks: Use Tailscale or WireGuard to create isolated communication channels between VMs. The agent provisions a Tailscale network, adds each VM to the network, and blocks public internet access. VMs can only reach each other via the overlay.

Separate accounts: For high-security workloads, provision VMs in separate Machine0 accounts. This prevents lateral movement at the account boundary but increases operational complexity (multiple API keys, separate billing).

Audit Logging Gaps

Machine0 does not document audit logging for VM provisioning, suspension, or snapshot operations. An agent that provisions 50 VMs, suspends 30, and deletes 20 leaves no audit trail unless the orchestrator logs every CLI call. For compliance-sensitive workloads (HIPAA, SOC 2), this is a blocker.

Workarounds:

Orchestrator-level logging: Wrap every machine0 CLI call in a logging function that records the command, timestamp, and result. Ship logs to a tamper-proof sink (S3 with versioning, CloudWatch Logs with retention policies).

Periodic inventory scans: Run machine0 list --json every 5 minutes and diff the results. Alert on unexpected VMs (provisioned outside the orchestrator) or missing VMs (deleted without logging).

Custom audit layer: Build a proxy API that wraps the Machine0 CLI, similar to how Teleport proxies SSH. All agent calls go through the proxy, which logs every operation before forwarding to Machine0. This adds latency but provides a complete audit trail.

Public IP and HTTPS Endpoint Risks

VMs get a public IP and an HTTPS endpoint at <vm>.mac0.io. This is convenient for agents that need to expose a webhook or API, but it also means the VM is internet-reachable by default. The agent should:

  • Run a firewall (iptables or nftables) to block unused ports
  • Use SSH key authentication, not passwords
  • Rotate credentials after each task
  • Disable the HTTPS endpoint if not needed (Machine0 does not document how to disable <vm>.mac0.io, so assume it's always active)

For agents running untrusted code (e.g., executing user-submitted scripts), the public IP creates a data exfiltration path. The untrusted code can POST results to an attacker-controlled server. Mitigation: run untrusted code in a nested sandbox (gVisor, Firecracker) or block outbound traffic except to whitelisted domains.

Why Persistent VMs Matter for Long-Horizon Agents

Most agent frameworks assume short-lived execution. A tool call completes in seconds, the agent returns a response, and the runtime shuts down. That model breaks when an agent needs to:

  • Clone a 50 GB repository and run a multi-hour build
  • Train a model over 12 hours with checkpoints every 30 minutes
  • Scrape a dataset across 200 paginated API calls with rate limits
  • Run a browser automation workflow that waits for human approval mid-flow

Serverless functions time out. Containers restart and lose disk state. Hourly VMs waste money during idle periods. Machine0's primitives address these gaps:

Persistent disk state: VMs keep their filesystem across stops and resumes. An agent can write intermediate results to /tmp, suspend the VM, and resume hours later without re-downloading dependencies.

Per-minute billing: A VM that runs for 37 minutes costs 37 minutes of compute, not a full hour. The agent can suspend during rate-limit waits or human approval loops without burning budget.

Snapshot and clone: An agent can snapshot a VM after installing dependencies, then spin up 10 clones for parallel work. Each clone starts from the same golden image.

CLI-driven provisioning: The agent calls machine0 new --size xl --region nyc and gets a VM in 30 seconds. No Terraform, no console clicks, no approval queue.

Architecture: How Agents Drive VMs

Machine0 exposes two interfaces: a CLI and a Model Context Protocol (MCP) server. Both let agents treat VMs as programmable resources.

CLI Workflow

An agent running in a local environment or orchestrator can shell out to the machine0 CLI:

# Agent provisions a VM
machine0 new research-vm --size xl --region nyc --os nixos

# Agent SSHs in and runs a long task
ssh research-vm "git clone https://github.com/large-repo && make build"

# Agent suspends to stop billing during a 2-hour wait
machine0 suspend research-vm

# Agent resumes when ready
machine0 resume research-vm

# Agent snapshots the VM after setup
machine0 snapshot research-vm golden-image

# Agent spins up 5 clones for parallel work
for i in {1..5}; do
  machine0 new worker-$i --from-snapshot golden-image
done
Enter fullscreen mode Exit fullscreen mode

Every CLI command supports --json output, so agents can parse structured responses without regex scraping.

MCP Integration

The Model Context Protocol server exposes VM operations as tool calls. An agent using Claude or another MCP-compatible runtime can invoke machine0_new_vm, machine0_suspend_vm, and machine0_get_vm_status directly. The agent doesn't need shell access or credential management; the MCP server handles authentication.

This matters for agents running in sandboxed environments (browser extensions, hosted notebooks, or multi-tenant orchestrators) where shelling out to a CLI is blocked.

State Persistence and Recovery

Machine0 VMs are persistent by default. When an agent calls machine0 suspend, the hypervisor freezes RAM and disk state, then stops billing. When the agent calls machine0 resume, the VM restarts from the exact same state: open files, running processes, and network connections.

This differs from container orchestration, where a pod restart loses ephemeral disk and in-memory state. The agent using Machine0 can:

  • Write checkpoint files to /var/agent/state and resume from the last checkpoint after a crash
  • Keep a SQLite database open across suspend/resume cycles
  • Run a long-polling HTTP server that survives restarts

The trade-off: VMs are heavier than containers. A Machine0 VM takes 30 seconds to provision (vs. 2 seconds for a container) and uses more disk (25 GB minimum vs. a few hundred MB for a slim container image). For workloads under 5 minutes, containers win. For workloads over 30 minutes, the persistent state and per-minute billing make VMs cheaper and simpler.

Billing Model and Cost Control

Machine0 bills by the minute with no minimum runtime. A VM that runs for 3 minutes costs 3 minutes of compute. Compare this to:

Platform Billing Unit Idle Cost Suspend Cost Best For
AWS Lambda Per invocation + GB-seconds $0 when idle N/A Sub-minute tasks
AWS EC2 Per hour Full hour even if stopped after 5 min Full hour Always-on services
GCP Cloud Run Per 100ms + memory $0 when idle N/A HTTP request handlers
Machine0 Per minute Billed per minute No charge when suspended Multi-hour agent workflows

An agent running a 2-hour data pipeline on a large VM (2 vCPU, 4 GB RAM) pays 120 minutes × $0.052/60 = $0.104. If the agent suspends for 8 hours during a rate-limit wait, billing stops. The same workflow on an hourly EC2 instance costs 10 hours × $0.052 = $0.52, even if the VM is idle for 8 of those hours.

For GPU workloads, Machine0 offers H100 instances starting at $0.836/hr (billed per minute). An agent running a 4-hour fine-tuning job pays $3.34. On AWS SageMaker, the same H100 instance costs $4.10/hr with hourly billing, totaling $16.40 for a 4-hour job. The per-minute model saves 80% for bursty GPU workloads.

The risk: agents that forget to suspend VMs will rack up charges. Machine0 doesn't auto-suspend idle VMs (unlike Cloud Run, which scales to zero). An agent orchestrator needs to track active VMs and call machine0 suspend after task completion.

Observability and Failure Modes

Machine0 provides VM-level uptime (99.99% SLA) but no built-in observability for agent workflows. An agent running a multi-hour task needs to:

  • Write logs to a persistent location (/var/log/agent.log) or ship them to an external sink
  • Emit heartbeat signals so the orchestrator knows the agent is alive
  • Handle VM restarts (rare, but possible during maintenance windows)

The CLI exposes machine0 logs <vm> to tail system logs, but this doesn't capture application-level telemetry. The agent should integrate with an observability stack (Prometheus, Datadog, or a custom metrics endpoint) if it needs to track task progress or debug failures.

Common failure modes:

Agent forgets to suspend. The VM runs indefinitely, burning budget. Mitigation: orchestrator sets a max runtime and force-suspends after a timeout.

VM restarts mid-task. The agent loses in-progress work unless it writes checkpoints. Mitigation: the agent should checkpoint every N minutes and resume from the last checkpoint on restart.

Snapshot restore fails. A corrupted snapshot can't be resumed. Mitigation: implement a snapshot validation pattern. After creating a snapshot, provision a test VM from it, run a smoke test (check for expected files, run a simple command), then destroy the test VM. Store validation results alongside snapshot metadata.

Agent provisions too many VMs. Without rate limits, a buggy agent can spin up hundreds of VMs. Mitigation: set account-level quotas or implement a provisioning approval step in the orchestrator.

OS and Environment Setup

Machine0 supports two OS options: NixOS and Ubuntu. NixOS uses declarative configuration (flakes) for reproducible builds. Ubuntu comes pre-installed with Docker, Node, Python, and agent tools (Claude Code, Codex).

The agent that needs a custom environment can:

  1. Provision a base VM
  2. SSH in and run setup scripts (apt install, pip install, etc.)
  3. Snapshot the configured VM
  4. Spin up clones from the snapshot for parallel work

This is slower than container-based workflows (where you build an image once and deploy it everywhere) but simpler for agents that need full OS control or GPU drivers.

Machine0 offers GPUs (H100, H200, L40S) starting at $0.836/hr. The agent running a fine-tuning job can provision a GPU VM, train for 4 hours, snapshot the model weights, and suspend. The agent pays for 4 hours of GPU time, not a full day.

Technical Verdict

Machine0 fills a real gap: infrastructure designed for agents that run longer than a Lambda timeout but shorter than an always-on server. The per-minute billing and suspend/resume primitives make it cheaper than hourly VMs for bursty workloads, and the persistent state makes it simpler than container orchestration for agents that need to checkpoint progress.

The CLI-driven provisioning model is a good fit for agents that self-drive infrastructure, but it requires careful cost controls (max runtime limits, auto-suspend after idle periods) to avoid runaway charges. The lack of built-in observability and audit logging means you'll need to integrate your own logging and metrics stack.

Use Machine0 when your agent needs to train a model for 12 hours with per-minute billing ($0.052/hr for a large CPU VM, $3.34 total for a 4-hour GPU job) instead of hourly EC2 pricing that bills for idle time. Use it when your agent writes intermediate state to disk and needs to resume after interruptions without re-downloading dependencies or re-running setup scripts.

Avoid Machine0 for sub-minute tasks like Lambda invocations, which cost $0.20 per million requests and scale to zero automatically. Avoid it for stateless HTTP handlers where Cloud Run's per-100ms billing and auto-scaling make more sense. Avoid it for compliance-heavy environments requiring detailed audit logs, since Machine0 lacks native audit logging and you'll need to build orchestrator-level logging yourself.

For production deployments, implement budget alerts, TTL enforcement, and orchestrator-level audit logging before giving agents provisioning access. Set account-level quotas to prevent cost-based denial of service.

Top comments (0)