DEV Community: Arpan Shah

How I Ran a Containerized Playwright Scraper on a Free VM — and What Actually Broke

Arpan Shah — Tue, 28 Jul 2026 05:43:38 +0000

The goal was simple: download data on a schedule, automatically, with no local machine involved. No manual triggers, no keeping a laptop on overnight. Just a container, a free VM, and a cron job.

Getting there took longer than expected. Here's what actually broke and how I fixed it.

The setup

Oracle Cloud Always Free AMD micro VM (Ubuntu 24.04, x86_64, ~954MB RAM)
Playwright running Microsoft Edge (headless, via Xvfb) inside a Docker container
Two .NET apps — a Playwright downloader and a CSV consolidator — combined into one multi-stage image
GHCR as the image registry
Cron for scheduling on the VM itself — no CI, no external orchestrator

The build and push happen from a local WSL machine via VS Code tasks. The VM just pulls and runs.

Why containerize at all?

The alternative was installing the .NET SDK directly on the VM and running the app with dotnet run. That works, but it ties the VM to a specific runtime version, makes the setup hard to reproduce, and diverges from how the rest of the project is deployed.

Containerizing means the VM needs exactly two things: Docker and a cron job. Everything else — the SDK, Xvfb, Edge, both apps — lives inside the image. If the VM is rebuilt or replaced, setup is: install Docker, add the cron entry, pull the image, done.

The first real problem: OOM kills

The AMD Always Free shape has ~954MB RAM and zero swap by default. Playwright launching a real browser under load hits memory hard. Without swap, the OOM killer terminates the browser process outright — no graceful degradation, no error message in the logs, just a hard kill.

The fix is a 2GB swap file, persisted to /etc/fstab so it survives reboots:

sudo fallocate -l 2G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

After this, sustained load across a full run completed with OOMKilled: false. This isn't a one-time workaround — it's a permanent property of the VM shape.

One other thing worth noting: the free ARM64 shape (Ampere A1) has more RAM and would avoid the swap requirement entirely — but Microsoft Edge has no native Linux ARM64 build. Staying on AMD x86_64 was the only viable path for this stack.

The second problem: `xvfb-run` hangs silently

Playwright needs a display. The standard approach is xvfb-run, but in a minimal container image it hangs indefinitely with zero output.

The reason: xvfb-run uses xdpyinfo (from x11-utils) to confirm the virtual display is ready before handing off. That package isn't installed by default. Xvfb itself starts fine — the wrapper just never moves past its own readiness check, silently.

The fix: start Xvfb manually in the entrypoint, poll for the actual X11 socket file, then export DISPLAY and exec the app directly:

Xvfb :99 -screen 0 1280x800x24 &
while [ ! -S /tmp/.X11-unix/X99 ]; do sleep 0.1; done
export DISPLAY=:99
exec "$@"

No dependency on xdpyinfo, no silent hang.

The third problem: output vanishing silently

The apps write files to paths configured in appsettings.json. Those paths are fine on a dev machine. Inside a container, the app creates those paths on its own ephemeral filesystem — and when the container is removed, the files go with it. Logs still report success. Only running ls on the host mount folder reveals the problem.

The fix: both apps call .AddEnvironmentVariables() after .AddJsonFile(...) so environment variables override config file values. The Dockerfile sets container-native ENV defaults for all output paths. The bind mounts in docker run wire the container paths to real host folders:

docker run \
  --name my-pipeline \
  -v /opt/myapp/downloads:/app/downloads \
  -v /opt/myapp/diagnostics:/app/diagnostics \
  -v /opt/myapp/logs:/app/logs \
  -v /opt/myapp/consolidated:/app/consolidated \
  -v /opt/myapp/archive:/app/archive \
  ghcr.io/username/my-pipeline:latest

One subtle catch: Docker won't auto-create bind-mount source directories reliably. Create them explicitly with mkdir -p before the first run, or output silently goes to the container's own filesystem. This is now part of the run task — the directories are created automatically before docker run fires.

The fourth problem: the entrypoint script encoding traps

The entrypoint is a shell script. Copying it from a host file into the image hit two separate encoding issues, both producing the same error — exec format error (exit 255):

CRLF line endings — Windows editors save files with \r\n. The \r becomes part of the interpreter name in the shebang line, breaking it.
UTF-8 BOM — some editors prepend a BOM before the shebang. The kernel sees the BOM first and can't identify the interpreter.

The standard fix for CRLF is sed 's/\r$//' — but this does not fix a BOM. A BOM-aware sed pattern can also silently fail under a UTF-8 locale. Both issues produce the same exit code, making it easy to think you've fixed one when the other is still present.

The actual fix: write the entrypoint directly in the Dockerfile via a heredoc:

COPY <<'EOF' /app/entrypoint.sh
#!/bin/sh
Xvfb :99 -screen 0 1280x800x24 &
while [ ! -S /tmp/.X11-unix/X99 ]; do sleep 0.1; done
export DISPLAY=:99
exec "$@"
EOF
RUN chmod +x /app/entrypoint.sh

The heredoc guarantees LF line endings and no BOM regardless of host OS or editor. The encoding problem simply can't occur.

Scheduling: cron on the VM itself

No CI runner, no GitHub Actions schedule, no external trigger — cron calls docker run directly on the VM. The local machine doesn't need to be on.

One thing to know: cron is not installed by default on minimal Ubuntu cloud images. It needs to be installed and enabled explicitly:

sudo apt-get install -y cron
sudo systemctl enable cron
sudo systemctl start cron

The cron entry cleans up any leftover container from a previous run before starting a new one:

30 1 * * 2-6 docker rm -f my-pipeline 2>/dev/null; docker run --name my-pipeline [mounts and flags] >> /opt/myapp/logs/cron.log 2>&1

All container output is appended to a cron log — one place to check after any scheduled run.

What's proven

After a full run across all configured targets:

Exit code 0
78 files saved, 0 failed, 0 skipped
OOMKilled: false

The 2GB swap held under sustained load. The pipeline now runs on schedule with no local machine involvement.

What's still open

A few things are deliberately deferred:

Off-site backup — currently data lives only on the VM's local disk. A Google Drive backup via service account is planned but not started.
Diagnostics under real failure — the screenshot/HTML capture on Playwright startup failure is wired up but hasn't been verified end-to-end under a real failure yet.

None of these block the automated pipeline. They're next.

If you've hit any of these in your own container/VM setup — especially the Xvfb hang or the BOM trap — I'd be curious what your fix looked like.

I Built a CLI to Use Free Web-Based AI Chatbots for Real Development Work — No API Keys, No Extensions

Arpan Shah — Wed, 22 Jul 2026 18:42:09 +0000

I wanted to use web-based AI chatbots — Claude, Gemini, ChatGPT, Qwen — for actual development work, not just Q&A. The free tiers are generous, and I didn't want to be locked into a single coding agent or pay for API access just to get an assistant to touch my code. But the moment you try to actually use a web chat for real dev work, you hit the same wall every time: you either paste in your whole codebase manually every session, or you give up and reach for a paid extension with an API key behind it.

So I built AI Bridge — a CLI tool that bridges a local codebase and any browser-based AI chatbot. No API keys, no extensions running in the background, no vendor lock-in. You pack your code, paste it into whichever AI chat you're using, and apply the changes back with a command.

Why not just use Copilot, Cursor, or an API key?

Coding agent apps and API-based tools work, but they come with tradeoffs I wanted to avoid:

Web-based AI chat plans are usually more generous on the free tier than API usage
I'm not locked into one provider — I can switch models mid-project depending on which one is handling a task better
There's no background agent or extension — it's just a CLI and whatever chat tab I already have open

The tradeoff is that browser chats don't have direct filesystem access. AI Bridge closes that gap without turning it into full manual copy-pasting.

How it works

There are two modes, depending on project size.

Simple Mode is for projects small enough to fit in a single prompt. You pack the codebase, upload it along with a couple of prompt templates, and apply the AI's response back to your files:

dotnet tool install --global Tools.AIBridge
cd /path/to/your-project
ai-bridge init
ai-bridge pack
# Upload ai-bridge/1-SimpleMode/*.md + the generated context files to your AI
# Copy the AI's response, then:
ai-bridge apply --paste

Advanced Mode is for larger codebases, where uploading everything every time burns tokens and adds noise. Instead, you generate a one-time index.xml map of the project. From then on, the AI requests only the specific files it needs for a given task, and AI Bridge gathers and bundles just those into context — so you're not re-uploading the whole repo on every prompt.

It respects .gitignore automatically (via git ls-files), and there's an .aiignore file for excluding things like test fixtures or generated code that the AI doesn't need to see.

It's also not tied to .NET specifically — it detects your ecosystem (.csproj, package.json, pyproject.toml, go.mod, Cargo.toml) and groups context files accordingly, so it works with Node.js, Python, Go, and Rust projects too.

Applying changes back works the same way regardless of mode — copy the AI's XML response and run ai-bridge apply --paste, which reads straight from your clipboard on Windows, macOS, and Linux (X11/Wayland), with a stdin fallback for headless/SSH setups.

Quick demo

Here's a short demo of Simple Mode in action.

Try it

dotnet tool install --global Tools.AIBridge

Requires the .NET 10 SDK and Git on your PATH. Full setup and command reference are in the repo.

I built this mainly to solve my own workflow friction, but I'm curious whether it's useful outside that — if you try it, I'd genuinely like to know whether it fits how you work, or where it breaks down. It's MIT licensed, so issues and PRs are welcome too.