Hey everyone!
So, a few days ago, I came across a project on GitHub called MLX-DLSS, which attempted to port NVIDIA's DLSS 5 technology to macOS using the MLX backend.
I was genuinely disappointed when I saw the performance numbers: 6 FPS, and that too on a pre-rendered video with just a single pass. 💀
So I decided to experiment with it myself.
Before going further, I should mention that I did take partial help from a coding agent, mainly for testing and analysis. The actual coding and implementation work was done by me.
Anyway, here's what I ended up doing.
I ported the project into OBS Studio as a plugin, extracted the model weights from nvngx_dlssnr.dll (the main Windows DLSS 5 neural-rendering DLL provided by NVIDIA), and started engineering a different, hybrid backend.
The idea was to distribute the workload across Apple's available hardware rather than relying entirely on GPU inference.
Here's how my current pipeline works:
Apple Neural Engine (ANE) via Core ML: Handles the main deep-learning transformer inference passes using the 31 HT and 71 HT models.
MLX: Handles AI upscaling and experimental frame generation.
Temporal stability: I've implemented temporal stabilization to reduce texture flickering and RGB poisoning.
A small warning about frame generation: it's still experimental, partially implemented, and quite buggy. I'm actively working on improving it.
The good news is that temporal stability has improved significantly. Texture flickering is much less noticeable, and RGB poisoning has been reduced by a huge margin compared to my earlier tests.
How am I feeding frames into the pipeline?
Right now, I'm using a PS5 connected to my Mac through an HDMI capture card, with the captured frames being processed through OBS Studio.
However, the pipeline isn't limited to PS5 footage. It can process other OBS video sources as well.
The three rendering modes
I've implemented three different modes: Fidelity, Balanced, and Performance.
- Fidelity Mode
This is the heaviest configuration.
It uses a double 71 HT transformer pass, prioritizing neural-rendering quality over performance.
Current results:
Without frame generation: 8–11 FPS
With frame generation: ~35 FPS
It looks impressive, but as mentioned earlier, frame generation is still experimental and can introduce visual artifacts and instability.
- Balanced Mode
This one is particularly interesting.
Instead of running the same model twice, Balanced uses a heterogeneous pipeline, with the first pass running the 31 HT model and the second pass running the 71 HT model.
Current results:
Without frame generation: 12.5–15 FPS
With frame generation: 45–48 FPS
It's basically my attempt to combine the advantages of both models without going all-in on the heavier double-71 configuration.
- Performance Mode
This is the lightweight configuration.
It runs a single 31 HT transformer pass.
Current results:
Without frame generation: ~20 FPS
With frame generation: 70–80 FPS
Obviously, there's a trade-off in rendering quality compared to Fidelity, but this mode provides considerably better performance.
Again, the frame-generated numbers shouldn't be confused with native inference performance. FG is still a work in progress, and visual stability isn't completely solved yet.
How am I even getting double-digit FPS?
Now, you might be wondering how I went from the original project's 6 FPS on a pre-rendered video to these numbers.
Well, I changed quite a few things in the original approach.
(A) The 480p neural-inference trick
Instead of directly feeding full 1080p frames into the neural inference backend, I do something different.
First, I capture the console output at 1080p through the HDMI capture card.
Then I downscale the frames to 480p before feeding them into the neural-rendering backend.
Once neural inference is complete, I upscale the output back to 1080p using an MLX-based AI upscaler.
The idea is to reduce the computational cost of inference while using AI upscaling to reconstruct and preserve details that conventional spatial or temporal upscaling might soften.
In some scenes, the resulting details can even appear better than the original input.
Of course, this approach has its own limitations, but the performance improvement has been pretty substantial.
(B) Moving neural inference from MLX to the ANE
This was another major change.
The original project relied on MLX for GPU-based neural inference.
Instead of following the same approach, I moved major parts of the deep-learning network onto Apple's Neural Engine through Core ML.
Think of it this way:
On NVIDIA GPUs, you have dedicated Tensor Cores for accelerating certain AI workloads.
On Apple Silicon, you have the Apple Neural Engine, which provides dedicated hardware acceleration for supported machine-learning operations.
They're not architecturally identical, obviously, but the general idea is to take advantage of dedicated neural acceleration hardware.
My goal was to avoid putting all the inference pressure on the GPU, especially since the GPU is already doing other work.
MLX is still useful, and newer Apple GPU architectures, particularly the M5 series and beyond, may offer additional advantages for GPU-based machine learning.
But for my M4 Pro, using Core ML to leverage the ANE for the main transformer inference stages made much more sense for this particular workload.
So instead of having MLX handle everything, I ended up with a heterogeneous backend where Core ML/ANE and MLX handle different parts of the pipeline.
Other things I've implemented
I've also recreated a RenoDX/OptiScaler-style configuration menu for DLSS 5, allowing users to tweak different model and pipeline settings.
Alongside that, I've implemented an RTSS-style performance metrics overlay to monitor performance while the pipeline is running.
These are still evolving, just like the rest of the project.
What's next?
The biggest thing I'm working on right now is frame-generation stability.
Temporal stabilization has already helped reduce flickering and RGB artifacts significantly in the normal neural-rendering pipeline, but enabling frame generation can still reintroduce some of those issues.
So that's my current priority.
The project is very much experimental, and there's still a lot of testing, optimization, and debugging left to do.
But considering where the original experiment started, I'm pretty happy with the progress so far.
GitHub repository
Here's the project:
https://github.com/Anonymous137-sudo/NVDLSS5_CML_MTL
Feel free to check it out, give it a try, and share your feedback!
I'm particularly interested in hearing from people working with Apple Silicon, Core ML, MLX, Metal, neural rendering, or similar graphics pipelines.
Top comments (0)