DEV Community

Discussion on: Struggling with micro-optimizations on large scale data processing

Collapse
 
rslak profile image
rslak

The calculations are CPU bound and hence you can try parallelism (Data Parallelism) by distributing your workloads across multiple cores of your server CPU. You can fork them if there are dependencies. If you have C#, Java stack these languages support that

Collapse
 
roelofjanelsinga profile image
Roelof Jan Elsinga

I use PHP for this, but there is a way to distribute processes across multiple threads. I'm going to look into this, thank you for your reply! :)

Collapse
 
serializator profile image
Julian

I'd say that it might be worth looking into other languages that might be better at this specific scenario than PHP.

I don't know exactly what your code does and what it's based on, because obviously that idea might not be feasible if you're tied to a framework of some kind to get the data from.