Target: QuantConnect/Lean
Issue: QuantConnect/Lean#7031
PR: QuantConnect/Lean#9611
Field test status: new feature contribution submitted; upstream review pending.
The feature request
Issue #7031 requested walk-forward optimization support in LEAN.
Walk-forward optimization allows an algorithm to optimize its parameters repeatedly over time rather than selecting one fixed parameter set before execution.
The feature request therefore involved more than adding another optimizer command.
LEAN needed a way for algorithms to:
- schedule optimization during execution
- run optimization through different backends
- receive the selected parameter set
- inspect the resulting backtest summary
- avoid recursive or nested optimization runs
The challenge was to introduce that capability without coupling the algorithm API to one execution environment.
Why this is a boundary problem
Walk-forward optimization spans several responsibilities:
- algorithm scheduling
- optimization execution
- backtest orchestration
- parameter selection
- local infrastructure
- QuantConnect cloud infrastructure
- result delivery back into the running algorithm
Those responsibilities belong together as a feature, but they should not collapse into one implementation.
The central design question was:
How can an algorithm request walk-forward optimization without needing to know whether the optimization runs locally or through QuantConnect infrastructure?
That is the boundary.
The algorithm should express when optimization runs and how its results are consumed.
The provider should own where and how the optimization is executed.
Without that separation, the public algorithm API would become tied to one backend and make future execution modes harder to support.
The feature lane
The PR introduces QCAlgorithm.Optimize(...) overloads that schedule walk-forward optimization in a way similar to Train(...).
The new API gives algorithms a core contract for requesting optimization over time.
Execution is routed through a provider abstraction rather than being hard-coded into the algorithm layer.
The implementation includes:
- a local walk-forward optimization provider
- a QuantConnect API-backed provider
- scheduling through
QCAlgorithm.Optimize(...) - result plumbing for selected parameter sets
- backtest summary delivery
- safeguards against nested child optimizations
This creates one algorithm-facing API with multiple execution paths.
Why Train(...) was the right model
LEAN already has an established scheduling model through Train(...).
Walk-forward optimization has a similar temporal shape:
- schedule work
- execute it at a defined time
- return a result to the algorithm
- allow the algorithm to update its behavior
Using a comparable API pattern makes the new feature easier to understand and keeps it aligned with an existing LEAN concept.
The new Optimize(...) API does not treat optimization as a one-time command-line concern.
It makes optimization part of the algorithm lifecycle.
That is the important shift.
The provider boundary
The provider abstraction is the architectural center of the feature.
The algorithm asks for optimization.
The configured provider decides how that optimization is executed.
This allows LEAN to support both:
- local optimization
- cloud-backed optimization through the QuantConnect API
The public algorithm contract remains the same in either case.
That separation matters because local and cloud execution have different operational requirements, but algorithms should not need separate optimization APIs for each environment.
The provider boundary keeps backend choice out of strategy logic.
Result plumbing
Starting an optimization is only half of the feature.
The algorithm also needs a usable result.
The PR adds plumbing for:
- the selected parameter set
- associated backtest summaries
- applying the chosen parameters after optimization completes
This turns optimization into an actionable algorithm event rather than a detached batch process.
The algorithm can schedule an optimization window, receive the chosen values, and continue using those values during later execution.
That is what makes the feature genuinely walk-forward rather than simply repeated offline optimization.
Nested optimization safeguards
Walk-forward optimization can create child backtests.
Without an explicit guard, those child backtests could attempt to start their own optimization runs.
That could produce recursive execution, duplicated work, or an uncontrolled expansion of optimization jobs.
The feature therefore includes safeguards against nested child optimizations.
This is a small implementation detail with an important systems consequence.
A scheduled execution feature must define not only how work starts, but also where that work is prohibited from starting again.
Why this was not an optimizer rewrite
The feature does not replace LEAN’s optimizer.
It does not introduce a new optimization algorithm.
It does not redesign backtesting.
It does not force one optimization backend.
It adds the missing orchestration layer between a running algorithm and the optimization infrastructure that already exists around it.
The new capability is therefore bounded to:
- scheduling
- provider routing
- execution context
- result delivery
- recursion safeguards
That scope keeps the feature additive and non-breaking.
The Scarab reading
This was not a defect-repair case.
It was a missing-capability case.
LEAN already had algorithm scheduling, optimizer infrastructure, backtesting, and provider-based execution concepts.
What it lacked was a governed boundary connecting those components into a walk-forward optimization workflow.
The feature adds that boundary.
The algorithm owns the request.
The provider owns execution.
The optimizer owns parameter search.
The result contract carries the selected parameters and backtest evidence back to the algorithm.
That is the Scarab boundary:
Expose walk-forward optimization as an algorithm-level capability while keeping execution backend, optimizer mechanics, and child-run control behind explicit provider and result contracts.
Configuration and documentation
The feature introduces configuration for selecting and controlling local or cloud-backed execution.
If accepted, the LEAN documentation will need to cover:
- the new
QCAlgorithm.Optimize(...)API - scheduled walk-forward optimization behavior
- local provider configuration
- QuantConnect API-backed provider configuration
- the
walk-forward-optimization-*configuration keys - selected parameter and backtest result handling
- nested optimization restrictions
This is a feature where documentation is part of the public contract.
The API may be clear in code, but users also need to understand when optimization runs, which provider is active, and how selected parameters return to the algorithm.
Validation
The feature was validated in a local macOS environment using the .NET SDK available in the checkout.
Validation included:
dotnet build Tests/QuantConnect.Tests.csproj /p:Configuration=Debug /v:quiet
Focused walk-forward optimization coverage passed:
13 tests passed
Optimizer regression coverage passed:
186 tests passed
The branch was also checked with:
git diff --check origin/master...HEAD
The complete repository test suite was not run locally.
The validation claim is therefore limited to the build, focused walk-forward optimization tests, optimizer regression tests, and diff integrity checks listed above.
Field result
Result: bounded new-feature contribution for scheduled walk-forward optimization in LEAN.
Feature shape:
- algorithms need to schedule optimization over time
- execution may be local or cloud-backed
- algorithm code should not be coupled to one backend
- selected parameters must return to the running algorithm
- backtest evidence must be available with the result
- child optimization runs must not recursively launch new optimizations
Implementation:
- add
QCAlgorithm.Optimize(...)scheduling overloads - add a walk-forward optimization provider abstraction
- add local and QuantConnect API-backed providers
- return selected parameter sets and backtest summaries
- prevent nested child optimizations
- add focused feature and optimizer regression coverage
This contribution does not replace LEAN’s optimizer.
It adds the orchestration and provider boundary required to make scheduled walk-forward optimization available as an algorithm-level feature.
Public claim
Scarab/SDS helped shape a bounded feature contribution for QuantConnect/Lean#7031.
The PR introduces scheduled walk-forward optimization through new QCAlgorithm.Optimize(...) overloads, a provider abstraction for local and cloud-backed execution, and result plumbing for selected parameters and backtest summaries.
The design keeps algorithm code independent of the execution backend and includes safeguards against nested child optimizations.
Validation passed through the LEAN test-project build, 13 focused walk-forward optimization tests, 186 optimizer regression tests, and a clean branch diff check.
The claim is limited to the new walk-forward optimization API, provider routing, result contract, and execution safeguards. It does not claim to replace LEAN’s optimizer or redesign the broader backtesting system.
Disclosure: AI-assisted coding tools were used while preparing this contribution. The implementation, tests, technical claims, and final wording were reviewed, and responsibility for the changes and any required follow-up revisions remains with the contributor.
Top comments (0)