Executive Summary
GPU-accelerated optimization algorithms have expanded far beyond their traditional home of deep learning training into virtually every branch of optimization: mathematical programming, combinatorial optimization, evolutionary computation, Bayesian optimization, robot motion planning, and even quantum-inspired annealing. The underlying logic is consistent everywhere: restructure the most time-consuming operators of an iterative optimizer (sparse matrix–vector products, batched candidate evaluation, matrix factorization, Monte Carlo sampling) into a massively parallel, memory-bandwidth-friendly form, compressing "overnight" problems into minutes or even milliseconds. This report systematically surveys the mainstream GPU-accelerated optimization algorithms and representative tools across five domains, presenting publicly reported performance figures (e.g., NVIDIA cuOpt delivering up to 5,000× speedup on a large multicommodity-flow LP instance; cuLoRADS achieving 100×+ over CPU solvers on hundred-million-scale SDPs; cuRobo motion planning at 60× over CPU planners), and concludes with a scenario-by-scenario engineering selection table. It must be stressed that reported speedups come from very different measurement setups, and GPUs do not win on every problem — small instances, high-accuracy requirements, and branch-and-bound-style search remain CPU territory. Selection should always be validated by benchmarking on your own problem sizes and accuracy needs.
1. Overview: The Landscape of GPU-Accelerated Optimization
"GPU-accelerated optimization algorithms" form a large family spanning multiple disciplines. Broadly speaking, they follow two lines of thinking. The first is algorithms that are naturally parallel: in evolutionary algorithms every individual in the population can be evaluated independently; in Bayesian optimization thousands of candidate points can be scored in a batch; in motion planning hundreds of trajectories can be rolled out simultaneously. Such algorithms routinely gain one to several orders of magnitude on GPUs. The second line is algorithms redesigned for GPUs: the most typical example is the shift in linear programming from the simplex method and interior-point methods (which depend on hard-to-parallelize sparse matrix factorization) toward the first-order PDHG/PDLP family (whose iteration core is just sparse matrix–vector products), which finally unlocks the bandwidth advantage of GPUs (NVIDIA). Representatives of the first line include EvoX, evosax, BoTorch, and cuRobo; representatives of the second include cuPDLP, cuOpt, ClarabelGPU, and cuLoRADS.
From an engineering perspective, the first step in selection is identifying which class of problem you face. For continuous differentiable problems (deep learning training, nonlinear least squares), look first at optimizers inside deep learning frameworks and differentiable optimization libraries. For constrained continuous problems of enormous scale (LP/QP/SDP), look at GPU first-order solvers. For discrete/combinatorial problems (MILP, VRP, scheduling), GPUs currently play the role of heuristics and relaxation solvers, working best in hybrid combination with CPU branch-and-bound. For black-box, gradient-free problems (hyperparameters, structural design, expensive-simulation optimization), look at GPU implementations of evolutionary computation and Bayesian optimization. Each of the following sections expands on these in turn.
2. Mathematical Programming and Operations Research Solvers
Mathematical programming is the domain where GPU-ification has advanced fastest over the past two years and had the greatest industrial impact. The turning point came from Google's PDLP, proposed in 2021: LP is reformulated as a minimax saddle-point problem and solved with the primal-dual hybrid gradient method (PDHG) augmented by restarts, diagonal preconditioning, and adaptive step sizes, so the iteration core reduces to sparse matrix–vector products that are naturally suited to GPUs and distributed hardware (Google Research). The work received the 2024 Beale–Orchard-Hays Prize, and its C++ implementation is open-sourced as part of OR-Tools (Kavour).
2.1 Linear Programming (LP): The PDHG/PDLP Family and cuOpt
The academic pathbreaker for GPU-ified LP is cuPDLP.jl (a Julia implementation) from Haihao Lu's group at MIT: on the LP relaxations of MIPLIB 2017, at medium accuracy (1e-4) it solves more instances than all CPU versions of PDLP and is comparable to Gurobi; on large instances with over ten million nonzeros it is 3.7× faster than Gurobi's best barrier method, and up to 20× faster than its own Julia CPU version (arXiv:2311.12180). It was followed by a C implementation, cuPDLP-C (roughly another 50% faster), and by cuPDLP+, which adopts restarted Halpern PDHG (rHPDHG) with reflection updates and PID-controlled primal-weight updates — a further 2–4× speedup over cuPDLP on MIPLIB LP relaxations (2.9× at high accuracy with presolve, 4.63× on hard instances) (arXiv:2507.14051). This line of work directly influenced the design of commercial and open-source solvers including Gurobi, COPT, FICO Xpress, and NVIDIA cuOpt (EmergentMind).
The industry benchmark is NVIDIA cuOpt: open sourcing was announced in March 2025 in collaboration with the COIN-OR Foundation, and from June 2025 the complete LP/MIP/VRP solver source code has been released under the Apache 2.0 license (NVIDIA Developer). On the Mittelmann LP benchmark (1e-4 tolerance), cuOpt's PDLP solver is faster than a top CPU LP solver on 60% of the instances where both converge, more than 10× faster on 20% of instances, and up to 5,000× on a single large multicommodity-flow instance; against CPU PDLP implementations it is consistently 10×–3,000× faster (NVIDIA Developer). In October 2025 cuOpt added a GPU-accelerated barrier (interior-point) method, which internal benchmarks show averaging more than 8× faster than a leading open-source CPU solver and more than 2× faster than a popular commercial CPU solver, filling the high-accuracy gap (NVIDIA Developer).
Among commercial solvers, Gurobi has landed GPU support along two routes: its barrier method uses NVIDIA cuDSS for sparse linear-system solves, and its PDHG method uses cuSPARSE for sparse matrix–vector products with data resident in GPU memory throughout; Gurobi also cautions that GPU PDLP's advantage depends strongly on problem characteristics, and in its internal test set CPU solvers still win on the majority of instances (Gurobi). The Chinese solver COPT (Shanshu Technology) open-sourced cuPDLP-C, and its GPU version COPTG performs strongly on third-party leaderboards (arXiv:2506.02174). Nothing illustrates the shifting landscape better than Hans Mittelmann's LPfeas leaderboard (data of 12 June 2026): the top three — COPTG (1.00), cuOpt 26.06 (1.07), HPRLP (1.36) — are all GPU solvers, ahead of CPU COPT (1.67), MOSEK (5.28), HiGHS (17.24), and PDLP (27.85); three of the top four entries are GPU-accelerated (Mittelmann Plots, ASU).
Boundary conditions that should be stated honestly: PDLP-type methods scale linearly with memory bandwidth, so reproducing the paper numbers requires relatively recent server-grade GPUs; on small LPs the GPU advantage is limited (which is why cuOpt offers a batch mode that solves hundreds of small LPs in parallel); and on the 49 public instances, 8 timed out due to slow convergence (NVIDIA Developer). Gurobi's engineering advice is therefore "pick the algorithm by model characteristics" rather than blanket GPU-ification (arXiv:2506.02174).
2.2 Quadratic Programming (QP) and Conic Programming (SOCP)
GPU-ification of QP advances along two routes. One is first-order methods: restarted accelerated PDHG (rAPDHG) targets large-scale convex QP, and its successor PDHCG (PDHG combined with conjugate gradients) is about 5× faster again than rAPDHG on specific large-scale tests and nearly two orders of magnitude faster than other solvers (Front. Eng. Mgmt.). The other route is moving the linear-system solves of interior-point methods onto the GPU: ClarabelGPU, the GPU version of the open-source conic solver Clarabel, factorizes the KKT system with the NVIDIA cuDSS library, and on large QP benchmarks it is the fastest of all solvers tested — more than 2× faster than Gurobi, about 4× faster than MOSEK, and about 10× faster than its own multithreaded Rust CPU implementation (arXiv:2412.19027).
These two routes correspond to the "factorization-free, bandwidth-hungry" and "GPU-accelerated factorization" paradigms respectively, and they also foreshadow the GPU-ification path for more general conic programs such as SOCP and SDP. cuOpt has listed QCQP and SOCP support in beta (GitHub). For engineering users, if you already model with CVXPY, both cuOpt and ClarabelGPU can be swapped in as backends at very low migration cost (NVIDIA).
2.3 Semidefinite Programming (SDP): cuLoRADS
SDP has long been bottlenecked by the factorization of huge matrix variables inside interior-point methods. cuLoRADS (open-sourced by the COPT team at Shanshu Technology) ports a two-stage algorithm — Burer–Monteiro low-rank factorization with an augmented-Lagrangian warm start followed by ADMM splitting — wholesale onto the GPU, using custom kernels and compressed storage exploiting the column sparsity of the constraint matrices to cut quadratic memory overhead down to something proportional to the number of nonzeros (arXiv:2407.15049). Its reported numbers are striking: on an H100 it solves MaxCut instances with 10⁷×10⁷ matrix variables in just 10–60 seconds each, whereas previous CPU solvers needed at least tens of hours for the same scale; a MaxCut with a 170M×170M matrix variable and 170 million constraints is solved in 160 seconds; on matrix-completion problems it is over 100× faster than the CPU version of LoRADS, and can solve a problem with an 8-million×8-million matrix variable and 320 million constraints in about 89 seconds — in the same time the CPU version can only handle 20,000×20,000 (arXiv:2407.15049).
On Mittelmann's sparse SDP leaderboard, cuLoRADS 1.0.0 already ranks second (score 3.01, behind only CPU COPT at 1.00, ahead of MOSEK at 3.67), making it the only GPU SDP solver currently in the leaderboard's top tier (Mittelmann Plots). Its limitations are that it does not yet support mixed problems containing linear variables, and robustness on general SDPs is still being improved (GitHub).
2.4 Mixed-Integer Programming (MILP), Combinatorial Optimization, and Vehicle Routing (VRP)
MILP is the hardest nut for GPU-ification: in branch-and-bound trees the LP subproblems at different nodes vary enormously in size and structure, and the highly irregular workload is a poor fit for the GPU's SIMT model (Front. Eng. Mgmt.). The currently viable paradigm is a hybrid architecture: the GPU runs massively parallel heuristics (to find high-quality feasible solutions quickly), while CPU branch-and-bound uses those solutions to prune the tree. This is exactly the idea behind NVIDIA cuOpt's MIP beta — GPU-accelerated primal heuristics (including three evolutionary algorithms, stagnation detection, subMIP evolution, and diving threads with branch-and-bound) rapidly produce feasible solutions, and it has even found new solutions for four open instances in MIPLIB (AlphaSignal). In a commercial case study, SimpleRose combined its parallel MIP solver with cuOpt (cuOpt provides heuristic solutions, Rose prunes in parallel), accelerating overall MILP solve time by up to 61.7× and the root LP by up to 50.2× (SimpleRose); combining HiGHS with cuOpt cut the MIP gap from 28% to 21% on one problem class (AlphaSignal).
VRP and routing optimization is cuOpt's most mature commercial module. By combining GPU-accelerated heuristics with evolutionary strategies, as of March 2024 it had broken world records on 15 instances of the Gehring & Homberger benchmark and 8 instances of the Li & Lim benchmark, and it has held all records in the CVRPTW and PDPTW categories for three consecutive years; NVIDIA claims routing solutions up to 100× faster (NVIDIA Developer). Practical deployments include: a 10,000-node VRP with real-world constraints solved on a single GPU in about 30 seconds; large-scale power unit commitment accelerated by up to 20×; and o9 Solutions achieving more than 10× planning speedup on production supply-chain data, turning "overnight batch processing" into "minute-level on-demand solving" (SimpleRose, AP News).
3. Deep Learning Training Optimizers
Deep learning is the most mature market for GPU optimization algorithms. "GPU acceleration" here operates at three levels: the implementation layer (fusing optimizer updates into CUDA kernels), the memory layer (low-bit/low-rank optimizer states so larger models fit in GPU memory), and the algorithm layer (new optimizers designed for GPU matrix compute).
3.1 Implementation Layer: Fused and Multi-Tensor Kernels
PyTorch provides foreach (horizontal fusion across multiple tensors) and fused (vertical fusion into a single CUDA kernel) implementations for Adam/AdamW/SGD and others. The principle is that multi-tensor-apply combines pointer management, memory alignment, vectorization, and the Adam math into one kernel, significantly reducing kernel-launch overhead and memory round-trips; this is the de facto default for large-scale training (GPU MODE). There is also the more advanced trick of "fusing the optimizer into backpropagation": using register_post_accumulate_grad_hook to apply the update as soon as each parameter's gradient is ready, eliminating the persistent gradient tensor — an official PyTorch tutorial claims this can be done in 10 lines of code (PyTorch).
These tricks do not change the optimizer's mathematical behavior, only constant factors, typically delivering single-digit to two-digit percentage speedups — but they are risk-free and tuning-free, and should be the baseline configuration for any GPU training task. In the JAX ecosystem the equivalents are Optax combinators and jax.jit XLA fusion, with similar effects.
3.2 Memory Layer: Low-Bit and Low-Rank Optimizers
Adam maintains two 32-bit states (first and second moments) per parameter, so optimizer memory often dominates training memory. bitsandbytes' 8-bit optimizers (the full family: Adam8bit/AdamW8bit/Lion8bit and more) use block-wise dynamic quantization to store states in 8 bits, dequantizing to FP32 for the update and re-quantizing each step, saving roughly 75% of optimizer memory with negligible accuracy loss; this is the quantization backend for Hugging Face Transformers/PEFT/Diffusers (bitsandbytes, Leeroopedia). Paged variants can additionally swap states out to CPU automatically when GPU memory runs low.
Another route is structural rank reduction: GaLore projects gradients onto a low-rank subspace before feeding them to Adam; GaLore 2 further combines low-bit projection matrices with deep FSDP sharding integration, substantially reducing per-GPU memory for Llama-3-class models on 2 GPUs (arXiv:2504.20437). Adafactor (row/column factorization of second moments) and DeepSpeed ZeRO's optimizer-state sharding with CPU offload are system-level approaches: ZeRO-Offload moves optimizer computation to the CPU (using AVX instructions) in exchange for memory (arXiv:2209.00099). The engineering rule of thumb: if memory is tight, start with 8-bit Adam (almost lossless); if still insufficient, add GaLore/FSDP sharding; ZeRO-Offload is the last resort (with obvious throughput loss).
3.3 Algorithm Layer: A New Generation of GPU-Born Optimizers
Over the past two years a batch of "matrix-aware" methods has emerged in LLM pretraining optimizers, sharing a common idea: trade GPU-friendly dense matrix operations for fewer training steps. Muon (proposed by Keller Jordan) orthogonalizes the momentum of hidden-layer 2D weight matrices via Newton–Schulz iteration; it maintains only momentum (no second moments, halving state memory), and achieves roughly 30–50% wall-clock advantage over AdamW in the NanoGPT speedrun and Karpathy's nanochat, i.e. about 2× compute efficiency (Grokipedia). Muon has been integrated into NVIDIA Megatron Core, and through layered distributed optimization and distributed Newton–Schulz iteration achieves throughput on GB300 nearly on par with AdamW (NVIDIA Developer); Turbo-Muon further accelerates the orthogonalization subroutine itself by 2.8× (Triton fused kernels + AOL preconditioning), cutting per-step time on a 1.3B model by about 8–10% (EmergentMind).
In the second-order and full-matrix preconditioning direction, Sophia (a lightweight second-order optimizer using diagonal Hessian estimates with element-wise clipping) reports roughly 2× speedup over Adam in steps, compute, and wall-clock; however, independent reproductions by Fraunhofer IIS show that although Sophia achieves the lowest training and validation loss, Lion is fastest in GPU-hours and AdamW scores best on downstream evaluation — demonstrating that "new optimizers are faster" depends heavily on the evaluation protocol (Facebook/DeepNet, arXiv:2507.08472). The Shampoo/SOAP family trades full-matrix preconditioning for stronger per-step progress: distributed Shampoo's per-step overhead, amortized across multiple GPUs, is typically only about 10% higher than Adam (EmergentMind); NVIDIA's large-scale pretraining study shows both SOAP and Muon consistently beat AdamW and scale to larger batch sizes, recommending KL-SOAP when memory is not tight (arXiv:2607.20548). Lion (a sign-momentum optimizer discovered by Google through evolutionary search) needs only one state tensor and suits memory-constrained scenarios where hyperparameter tuning is acceptable (arXiv:2312.03863).
4. Evolutionary Computation and Swarm Intelligence
Evolutionary algorithms (EAs) are naturally GPU-friendly: fitness evaluation of every individual in the population is fully independent, and crossover and mutation can also be vectorized. The bottleneck has historically been software frameworks rather than the algorithms — traditional Python EA libraries (such as DEAP) are CPU-serial. A new generation of GPU frameworks fills this gap: EvoX (EMI Group) is built on PyTorch and ships 50+ algorithms including GA, DE, PSO, CMA-ES, NSGA-II/RVEA/MOEA/D together with 100+ benchmark problems; it officially claims 100×+ acceleration on heterogeneous hardware, supports distributed multi-node execution, and interfaces with the Brax physics engine for evolutionary reinforcement learning (EvoX, GitHub). On the JAX side there is evosax (focused on evolution strategies, a GPU-optimized collection of ES algorithms) and EvoJAX (oriented toward neuroevolution); on the PyTorch side there is also EvoTorch (arXiv:2412.20980).
Academic evaluations highlight an important fact: different EAs gain very differently on GPUs — parallel granularity (individual-level vs gene-level), problem dimension, and the computational intensity of the fitness function determine the speedup, which cannot be summarized by a single number (arXiv:2601.18446). Practical experience says: the heavier the fitness evaluation itself (e.g., neuroevolution, simulation rollouts), the greater the GPU benefit; on pure numerical benchmark functions, GPUs can actually lose to CPUs at small population sizes due to kernel-launch overhead, and populations of thousands or more are needed to saturate the GPU. The non-dominated sorting step of multi-objective algorithms is highly irregular and harder to GPU-ify than single-objective ones; EvoX handles such problems through a unified tensorized programming model (arXiv:2609.02387).
5. Bayesian Optimization and Hyperparameter Search
GPU-ification of Bayesian optimization (BO) shows up in two ways: Gaussian-process (GP) matrix computations on the GPU, and batched parallel evaluation of the acquisition function over tens of thousands of candidate points. Meta's BoTorch (built on GPyTorch) supports batch evaluation in all components; increasing the number of Monte Carlo samples of the acquisition function has little impact on wall-clock time, and the official appendix reports significant GPU speedups; multi-objective acquisition functions (qLogNEHVI and others) can be solved efficiently in practical scenarios thanks to automatic differentiation and GPU acceleration (NeurIPS, BoTorch). Trust-region variants for high-dimensional problems, TuRBO/FuRBO, also run on GPUs and suit expensive-simulation optimization (CFD, chip design).
In hyperparameter optimization (HPO), Optuna's TPE sampler with pruning algorithms is the mainstream choice for single machines and small clusters, scalable to thousands of workers through a shared database backend (LabOak); Ray Tune targets parallel trial scheduling on GPU clusters, integrating search algorithms such as Optuna/HyperOpt with early-stopping schedulers such as ASHA/HyperBand/PBT. Industry practice shows that ASHA early stopping can cut total GPU compute cost of HPO by 70–80% without accuracy loss, roughly a 5–10× compute saving over grid/random search (Swfte, ML Journey). Selection advice: for a single machine with flexibility as priority, use Optuna; for multi-machine multi-GPU setups with fault tolerance and productionization as priorities, use Ray Tune (ML Journey).
6. Robot Motion Planning and Differentiable Nonlinear Optimization
The flagship in robotics is NVIDIA cuRobo: collision-free minimum-jerk trajectory generation for robot arms is formulated as a global optimization problem, solved by a combination of particle-sampling exploration, L-BFGS gradient refinement, and parallel noisy line search, optimizing hundreds of trajectories simultaneously on the GPU. The official paper reports solving hard motion-generation problems within an average of 50 ms — 60× faster than SOTA trajectory-optimization methods — with collision-free IK solved over 7,000 times per second (80× faster than common libraries), and on the embedded Jetson Orin NX platform it processes 512 trajectories × 64 steps in parallel at 500 Hz (Hugging Face, arXiv:2508.04146).
The more general tool is Meta's Theseus: a differentiable nonlinear least-squares library built on PyTorch, providing second-order optimizers such as Gauss–Newton, Levenberg–Marquardt, and Dogleg, with built-in batching, automatic vectorization, GPU sparse solvers (including its self-developed batched sparse Cholesky solver Baspacho), and support for implicit differentiation that embeds the entire optimization process into a neural network for end-to-end learning (NeurIPS 2022). The two represent "domain-specific parallel global optimization" and "general-purpose differentiable optimization infrastructure" respectively: the former is ready out of the box for motion planning, the latter suits scenarios like SLAM, pose-graph optimization, and inverse dynamics where optimization must be a layer inside a network.
7. Quantum-Inspired and Annealing Optimization
Combinatorial optimization in QUBO/Ising form has spawned a family of "physics-inspired" solvers, several offering GPU implementations: Fujitsu's Digital Annealer (improved simulated annealing with parallel trial moves and dynamic escape, on digital CMOS hardware, with CPU/GPU software modes; about two orders of magnitude faster than single-core SA on fully connected spin glasses, though with no advantage on sparse graphs) (Frontiers); Toshiba's Simulated Bifurcation Machine (CPU/GPU/FPGA supported); and the pure-GPU route of Compal's Quantix GPUA (multi-body adaptive search) (arXiv:2509.09862). Cross-annealer benchmark studies show: on classic problems like Max-Cut, TSP, and job-shop scheduling, GPU annealers match Fujitsu's DA in solution quality with wins on both sides, but on many small and medium instances well-tuned classical algorithms (such as OR-Tools) actually have the shortest solve times — the annealers' advantage lies mainly in extremely large, fully connected QUBOs (IEEE/InspireHEP, arXiv:2507.22117).
The lesson for engineering users: if your problem maps cleanly onto a large-scale QUBO and classical solvers are already struggling, GPU annealing/bifurcation methods are worth trying as a complement; otherwise the mature mathematical-programming route (GPU LP relaxations + heuristics + CPU branch-and-bound) is usually more controllable and easier to integrate.
8. Speedup Overview and Boundary Conditions
Placing the public figures of each domain on one chart makes the "order-of-magnitude distribution" of GPU acceleration immediately visible: in any domain whose bottleneck can be restructured as massively parallel regular computation (first-order LP methods, low-rank ADMM for SDP, evolutionary evaluation, trajectory sampling), speedups reach the 10²–10³ range; in domains where algorithms are already highly optimized and the GPU only contributes a constant-factor improvement (deep learning optimizers, QP interior points), typical gains are 2–10×. Note that the bars in the figure below come from mutually incomparable measurement setups (different hardware, different accuracy tolerances, different baselines); they are for perceiving magnitudes only, not for direct cross-comparison (NVIDIA Developer, arXiv:2407.15049, EvoX).
Typical situations where GPU acceleration fails or its benefit shrinks include: problems that are too small (kernel-launch and PCIe transfer overheads outweigh parallel gains — this is why cuOpt provides a small-LP batch mode); requirements for 1e-8-level high accuracy (first-order methods converge slowly, and GPU barrier/interior-point methods are more appropriate here); irregular tree search like branch-and-bound (hard for GPUs to schedule efficiently); and problems that do not fit in GPU memory (in the PDLP paper, some very large instances exceed single-GPU memory and can only be handled by large-memory CPU machines) (NVIDIA Developer, arXiv:2501.07018). Before selecting, it is advisable to run a small-scale benchmark on your own 3–5 most representative instances rather than taking vendor numbers at face value.
9. Engineering Selection Table
The table below aggregates the main GPU optimization tools covered in this report by problem type, together with their key attributes, for quick reference. Licenses and feature status are as of mid-2026; please check each project's latest documentation before deployment.
| Problem Type | Representative Tools | Core Algorithm | Hardware/Ecosystem | Representative Public Performance | License |
|---|---|---|---|---|---|
| Large-scale LP | NVIDIA cuOpt (GitHub) | PDLP (PDHG) + GPU barrier | CUDA; Python/C/Server APIs; plugs into CVXPY/PuLP/AMPL | #2 on Mittelmann LPfeas (1.07); up to 5,000× on a single instance (NVIDIA) | Apache 2.0 |
| Large-scale LP | cuPDLP.jl / cuPDLP-C / cuPDLP+ (arXiv) | Restarted (Halpern) PDHG | Julia/C + CUDA; research-friendly | 3.7× vs Gurobi barrier on large LP; cuPDLP+ another 2–4× (arXiv) | Open source |
| LP (commercial) | COPT GPU / Gurobi GPU (Gurobi) | GPU PDHG / cuDSS barrier | Commercial solver + CUDA | COPTG tops LPfeas (1.00) (Mittelmann) | Commercial |
| QP / SOCP | ClarabelGPU (arXiv); cuOpt QP (beta) | Interior point + cuDSS / PDHG family | Rust/CUDA; CVXPY backend | 2×+ faster than Gurobi, ~4× faster than MOSEK on large QP | MIT / Apache 2.0 |
| SDP | cuLoRADS (GitHub) | Low-rank Burer–Monteiro + ALM/ADMM | Julia/CUDA | 10⁷-scale MaxCut in 10–60 s; 100×+ vs CPU | Open source |
| MILP | cuOpt MIP (beta); Rose+cuOpt (SimpleRose) | GPU primal heuristics + CPU B&B hybrid | CUDA; composable with HiGHS etc. | Up to 61.7× overall MILP (hybrid scheme) | Apache 2.0 / Commercial |
| VRP / Routing | cuOpt Routing (NVIDIA) | GPU heuristics + evolutionary strategy | CUDA; Python/REST | 23 world records; 10k-node VRP in ~30 s | Apache 2.0 |
| DL training (general) | fused AdamW / torch.compile (GPU MODE) | Fused kernels | Built into PyTorch/JAX | Constant-factor speedup; risk-free baseline | Open source |
| DL training (memory-saving) | bitsandbytes 8-bit; GaLore 2 (arXiv) | Block-wise quantization / low-rank gradient projection | PyTorch; HF ecosystem default | −75% optimizer memory | MIT |
| DL training (new algorithms) | Muon / KL-SOAP / Sophia / Lion (NVIDIA) | Matrix orthogonalization / full-matrix preconditioning / diagonal second-order | Megatron Core, pytorch-optimizer | ~1.5–2× compute efficiency vs AdamW (protocol-dependent) | Open source |
| Evolutionary / Swarm | EvoX (GitHub); evosax | GA/DE/PSO/CMA-ES/NSGA-II fully tensorized | PyTorch / JAX + GPU | Officially claimed 100×+; 50+ algorithms | Open source |
| Bayesian optimization | BoTorch + GPyTorch (BoTorch) | GP + batched MC acquisition evaluation | PyTorch GPU | Significant batched/multi-objective acceleration | MIT |
| Hyperparameter search | Optuna / Ray Tune (ML Journey) | TPE + ASHA/PBT early stopping | Single machine → GPU cluster | Early stopping saves 70–80% GPU compute | MIT / Apache 2.0 |
| Robot planning | cuRobo (arXiv) | Particle optimization + parallel L-BFGS | CUDA; ROS2 interface | 60× vs CPU planners; IK at 7,000 queries/s | Apache 2.0 |
| Differentiable NLS | Theseus (NeurIPS) | Gauss–Newton / LM / Dogleg + batched sparse Cholesky | PyTorch GPU | Significant batched-vectorization speedup | MIT |
| QUBO / Annealing | Fujitsu DA / Toshiba SBM / Quantix GPUA (arXiv) | Parallel simulated annealing / simulated bifurcation | Dedicated hardware / GPU | ~100× vs single-core SA on fully connected spin glass | Mostly commercial |
10. Conclusion
The landscape of GPU-accelerated optimization algorithms can be summed up in one sentence: rewriting an optimization problem into a "massively parallel + bandwidth-friendly" form usually pays far more than simply porting an existing algorithm. In mathematical programming, the paradigm replacement of simplex/interior-point methods by PDHG first-order methods is the most profound change, and cuOpt, COPT GPU, and cuLoRADS have already surpassed traditional CPU commercial solvers on authoritative leaderboards. In deep learning, fused kernels and 8-bit states are "free lunches" that must be taken, while matrix-aware optimizers like Muon/SOAP represent the next wave of algorithm-level dividends. Evolutionary computation, Bayesian optimization, and robot planning gain order-of-magnitude improvements from their naturally batched parallel structure. For engineering deployment, three principles apply: first, use the selection table to lock in one or two candidate tools by problem type; second, benchmark on your own instances under real accuracy requirements; third, keep CPU options as a fallback for small-scale, high-precision, or branch-heavy problems — the hybrid architecture (GPU for speed, CPU for accuracy and global certificates) will remain the most robust deployment form for the foreseeable few years.
Note: This report is a general technical survey. All cited speedups come from public tests conducted by their respective sources, with varying measurement setups and hardware environments, and do not constitute a guarantee of product selection or commercial advice.
Top comments (0)