DEV Community

Cover image for Hot experts, cold RAM: running a big MoE
Private Local AI
Private Local AI

Posted on

Hot experts, cold RAM: running a big MoE

How expert offload actually works on a box with more system RAM than VRAM: hot experts stay pinned on the card, cold ones sleep in ordinary memory. The mechanics, measured:
https://blog.localai.hex7.com/hot-experts-cold-ram-running-a-big-moe-on-a-machine-with-more-memory-than-vram/

https://blog.localai.hex7.com/hot-experts-cold-ram-running-a-big-moe-on-a-machine-with-more-memory-than-vram/

Top comments (0)