I Trained RL Agents on a $0 Server. All Three Environments Solved in Under an Hour.

2 CPU cores. 12GB RAM. No GPU. Can you train a reinforcement learning agent on that?

Yes. I solved three standard RL benchmarks in under an hour. Here's the data.

Setup

Hardware: Oracle Cloud free tier. ARM64, 2-core Neoverse-N1, 12GB RAM. Total cost: $0.

Algorithm: DQN (Deep Q-Network) with experience replay, target network, epsilon-greedy exploration. No fancy tricks (no prioritized replay, no double DQN, no dueling networks). Vanilla 2015-era DQN.

Framework: PyTorch 2.14.0 CPU build, Gymnasium.

Results

Environment	Obs Dim	Act Dim	Solved At	Time	Memory	Ep/sec
CartPole-v1	4	2	Episode 975	10.1 min	324 MB	1.6
Acrobot-v1	6	3	Episode 728	12.8 min	324 MB	2.0
LunarLander-v3	8	4	Episode 677	36.8 min	341 MB	0.9
Total				59.7 min	341 MB

All three solved. Total wall-clock time: under an hour. Peak memory: 341 MB. That's less RAM than most browser tabs.

What Does "Solved" Mean?

Each environment has a standard threshold:

• CartPole: balance the pole for 475+ steps averaged over 100 episodes
• Acrobot: swing up in under 100 steps on average
• LunarLander: land safely with 200+ reward on average

The agent met all three thresholds during training.

Training Curves

CartPole ramped gradually. Random performance (reward ~40) at episode 100, near-optimal (~460) by episode 1000. The loss kept increasing throughout, which is expected: longer episodes mean larger Q-values mean larger squared errors.

Acrobot converged fastest in episodes (728) despite being "harder" than CartPole. The shorter episode length (max 500 steps) means more updates per wall-clock second.

LunarLander was the most expensive. 8D observations, 4 actions, longer episodes. Took 37 minutes but reached an average reward of 270 by episode 1700. Some degradation in the last 300 episodes, likely from Q-value overestimation (the known weakness of vanilla DQN).

Memory Profile

This surprised me. All three environments peaked between 324-341 MB. The replay buffer (50K transitions, ~5MB) is almost nothing. The dominant cost is the PyTorch runtime itself, not the RL data.

A Raspberry Pi 4 with 1GB RAM could run these experiments.

Where's the Wall?

These are the easy benchmarks. The hardware wall for this setup is at:

Image observations (Atari): Convolving 84x84 frames with no GPU would be extremely slow. This is probably where CPU-only RL stops being practical.

Large replay buffers: Atari needs 1M+ transitions (2-4 GB). The 12GB RAM can handle it, but barely.

Parallel environments: PPO and A3C benefit from running dozens of environments simultaneously. 2 cores can't do that.

Long training horizons: MuJoCo continuous control needs millions of timesteps. At 0.9 ep/sec, that's days.

I plan to test Atari on this hardware next to find the exact breaking point.

Takeaways

1. You don't need a GPU for classical RL. CartPole, Acrobot, LunarLander all solve on a free cloud instance.
2. Memory isn't the bottleneck. 341 MB peak. The bottleneck is CPU throughput on forward/backward passes.
3. Simple algorithms still work. Vanilla DQN from 2015, no modifications, solves all three. The algorithm research has moved on, but the baselines haven't gotten harder.
4. Reproducibility is better on CPU. No CUDA version issues, no GPU nondeterminism, no driver compatibility. Same code runs everywhere.

Code

Everything is public and runs on any machine with Python and 1GB RAM:

pip install torch gymnasium psutil
python3 rl_experiment.py

Full paper (LaTeX/PDF) and code: github.com/turingrtss/vulndetect

---

Next experiment: finding the hardware wall. Atari on 2 CPU cores. How slow is too slow?