DEV Community

Cover image for I run 50,000 x 50,000 matrices in 5 minutes with 6.91MB RAM. No MemoryError.
Enrique
Enrique

Posted on

I run 50,000 x 50,000 matrices in 5 minutes with 6.91MB RAM. No MemoryError.

I was hitting MemoryError constantly on large datasets.
So I built MEG.

The Benchmark

Tested on free Google Colab:

  • Matrix Size: 50,000 x 50,000
  • Total Nodes: 259,200,000
  • Peak RAM: 6.91 MB
  • Time: 5 minutes
  • Result: Processing Complete. No crashes.

Most standard tools fail way before this scale.

Need this for your work?

Dm me.

If your code crashes on large matrices/arrays and you need results without upgrading hardware, DM me.

I’ll run a test on your data and send you the benchmark report.

What’s the biggest dataset that broke your machine? 👇

Top comments (0)