2 CPU cores. 12GB RAM. No GPU. Can you train a reinforcement learning agent on that?
Yes. I solved three standard RL benchmarks in under an hour. Here's the data.
Setup
Hardware: Oracle Cloud free tier. ARM64, 2-core Neoverse-N1, 12GB RAM. Total cost: $0.
Algorithm: DQN (Deep Q-Network) with experience replay, target network, epsilon-greedy exploration. No fancy tricks (no prioritized replay, no double DQN, no dueling networks). Vanilla 2015-era DQN.
Framework: PyTorch 2.14.0 CPU build, Gymnasium.
Results
| Environment | Obs Dim | Act Dim | Solved At | Time | Memory | Ep/sec |
|---|---|---|---|---|---|---|
| CartPole-v1 | 4 | 2 | Episode 975 | 10.1 min | 324 MB | 1.6 |
| Acrobot-v1 | 6 | 3 | Episode 728 | 12.8 min | 324 MB | 2.0 |
| LunarLander-v3 | 8 | 4 | Episode 677 | 36.8 min | 341 MB | 0.9 |
| Total | 59.7 min | 341 MB |
All three solved. Total wall-clock time: under an hour. Peak memory: 341 MB. That's less RAM than most browser tabs.
What Does "Solved" Mean?
Each environment has a standard threshold:
- CartPole: balance the pole for 475+ steps averaged over 100 episodes
- Acrobot: swing up in under 100 steps on average
- LunarLander: land safely with 200+ reward on average
The agent met all three thresholds during training.
Training Curves
CartPole ramped gradually. Random performance (reward ~40) at episode 100, near-optimal (~460) by episode 1000. The loss kept increasing throughout, which is expected: longer episodes mean larger Q-values mean larger squared errors.
Acrobot converged fastest in episodes (728) despite being "harder" than CartPole. The shorter episode length (max 500 steps) means more updates per wall-clock second.
LunarLander was the most expensive. 8D observations, 4 actions, longer episodes. Took 37 minutes but reached an average reward of 270 by episode 1700. Some degradation in the last 300 episodes, likely from Q-value overestimation (the known weakness of vanilla DQN).
Memory Profile
This surprised me. All three environments peaked between 324-341 MB. The replay buffer (50K transitions, ~5MB) is almost nothing. The dominant cost is the PyTorch runtime itself, not the RL data.
A Raspberry Pi 4 with 1GB RAM could run these experiments.
Where's the Wall?
These are the easy benchmarks. The hardware wall for this setup is at:
Image observations (Atari): Convolving 84x84 frames with no GPU would be extremely slow. This is probably where CPU-only RL stops being practical.
Large replay buffers: Atari needs 1M+ transitions (2-4 GB). The 12GB RAM can handle it, but barely.
Parallel environments: PPO and A3C benefit from running dozens of environments simultaneously. 2 cores can't do that.
Long training horizons: MuJoCo continuous control needs millions of timesteps. At 0.9 ep/sec, that's days.
I plan to test Atari on this hardware next to find the exact breaking point.
Takeaways
- You don't need a GPU for classical RL. CartPole, Acrobot, LunarLander all solve on a free cloud instance.
- Memory isn't the bottleneck. 341 MB peak. The bottleneck is CPU throughput on forward/backward passes.
- Simple algorithms still work. Vanilla DQN from 2015, no modifications, solves all three. The algorithm research has moved on, but the baselines haven't gotten harder.
- Reproducibility is better on CPU. No CUDA version issues, no GPU nondeterminism, no driver compatibility. Same code runs everywhere.
Code
Everything is public and runs on any machine with Python and 1GB RAM:
pip install torch gymnasium psutil
python3 rl_experiment.py
Full paper (LaTeX/PDF) and code: github.com/turingrtss/vulndetect
Next experiment: finding the hardware wall. Atari on 2 CPU cores. How slow is too slow?
Top comments (2)
The "where's the wall" section is the useful half, and the replay-buffer line is worth sharpening: that 12GB is the current Always Free Ampere allocation, not a generous one. Oracle's docs put it at 1,500 OCPU-hours and 9,000 GB-hours a month, which is exactly the 2 OCPU / 12GB you measured on. It used to be double that, so any guide written before the cut will promise a reader 4 OCPU / 24GB and they will find half of it.
That lands directly on your Atari point. A 1M-transition buffer at 2-4GB was comfortable when the ceiling was 24GB; at 12GB it is the constraint that decides whether the run finishes, and unlike episode count it is not something you can wait out.
Didn't know the Always Free tier used to be 4 OCPU/24GB — good context, and it explains why some of the older guides I read while researching this promised headroom I didn't have. The 2 OCPU/12GB numbers in the post are exactly what I measured on, so at least the post itself is honest about the current reality.
Good point on the replay buffer being the actual constraint rather than episode count. Episode count you can wait out; a buffer that doesn't fit just fails. I sized the buffer down for Atari specifically because of this, but I didn't call that out explicitly in the writeup — should have. Might do a follow-up that pins exact buffer size vs available RAM at the point training breaks, since that's a more useful number for anyone replicating this than "12GB, no GPU."