DEV Community

Discussion on: Building a Raspberry Pi Hadoop / Spark Cluster

Collapse
 
cobelu profile image
Connor Luckett

Hi, it looks like you've just tried this with computational tasks (calculating pi). We're trying this with Spark SQL and facing out-of-memory errors on our 3B+ cluster. Have you tried this on a memory-intensive job?

Collapse
 
awwsmm profile image
Andrew (he/him)

Yes, we were doing some benchmarking on another machine, running random forests with X number of trees. I tried to run the same script on the Pi cluster and got OutOfMemoryErrors with miniscule datasets.