DEV Community

Henry Lin
Henry Lin

Posted on

Lesson 9: Time Range Testing

#datascience #testing #tutorial #machinelearning

Lesson 9: Time Range Testing

⏱ Duration: 1.5 hours
🎯 Learning Objectives: Validate strategy performance in different market environments
πŸ“š Difficulty: ⭐⭐ Backtesting practical

πŸ“– Course Overview

A strategy performing well in one time period doesn't guarantee it will perform well in other periods. This lesson will teach you how to validate strategy stability and adaptability through time range testing, avoiding overfitting traps.

9.1 Why Test Different Time Periods?

Three Core Reasons

1. Avoid Overfitting

What is Overfitting?
A strategy overfits specific characteristics of historical data, leading to poor performance on new data.

Overfitting Manifestations:

Training Period (2025-01-01 ~ 2025-06-30):
  Returns: +35%, Win Rate: 92%, Sharpe: 4.5 βœ…

Testing Period (2025-07-01 ~ 2025-09-30):
  Returns: -8%, Win Rate: 45%, Sharpe: 0.3 ❌

Conclusion: Serious overfitting!
Enter fullscreen mode Exit fullscreen mode

How to Identify Overfitting:

  • βœ… Excellent performance in training period, poor in testing period
  • βœ… Strategy parameters overly complex (> 10 conditions)
  • βœ… Strategy only works in specific time periods
  • βœ… Performance fluctuates dramatically with minor parameter adjustments

2. Validate Strategy Consistency

Definition of Consistency:
Strategy maintains relatively consistent performance across different time periods.

Consistency Metrics:

Q1 (Jan-Mar): Returns +8%, Drawdown -3%
Q2 (Apr-Jun): Returns +12%, Drawdown -4%
Q3 (Jul-Sep): Returns +9%, Drawdown -3.5%
Q4 (Oct-Dec): Returns +11%, Drawdown -3.2%

Conclusion: Strategy stable, consistent quarterly performance βœ…
Enter fullscreen mode Exit fullscreen mode

vs

Q1: Returns +45%, Drawdown -2%
Q2: Returns -15%, Drawdown -18%
Q3: Returns +30%, Drawdown -5%
Q4: Returns -20%, Drawdown -22%

Conclusion: Strategy unstable, high volatility ❌
Enter fullscreen mode Exit fullscreen mode

3. Identify Applicable Markets

Different strategies are suitable for different market conditions:

Trend Strategy vs Range Strategy:

Trend Strategy (MovingAverageCross):
  Bull Market (2025-01~03): +35% βœ…
  Range-Bound (2025-04~06): -8% ❌
  Bear Market (2025-07~09): +28% βœ…

Mean Reversion Strategy (MeanReversion):
  Bull Market: +5% ⚠️
  Range-Bound: +22% βœ…
  Bear Market: +3% ⚠️
Enter fullscreen mode Exit fullscreen mode

Conclusions:

  • Trend strategies suitable for bull and bear markets (trending markets)
  • Mean reversion strategies suitable for range-bound markets (range fluctuations)

9.2 Bull Market vs Bear Market vs Range-Bound Market

Market Type Characteristics

1. Bull Market

Features:

  • πŸ“ˆ Continuous upward trend (gain > 20%)
  • πŸ”Ί Small pullback magnitude (< 10%)
  • πŸ“Š Increased trading volume
  • πŸ’ͺ Optimistic market sentiment
  • ⏱ Duration: weeks to months

Identification Method:

# Simple judgment: MA50 rising and price above MA50
price > MA50 and MA50 > MA50.shift(30)
Enter fullscreen mode Exit fullscreen mode

Suitable Strategies:

  • βœ… Trend Following
  • βœ… Momentum
  • βœ… Breakout
  • ❌ Not suitable: Mean Reversion

Backtesting Points:

# Test bull market (assume 2025-01~03 is bull market)
freqtrade backtesting \
  -c config.json \
  --strategy MomentumTrendStrategy \
  --timerange 20250101-20250331
Enter fullscreen mode Exit fullscreen mode

2. Bear Market

Features:

  • πŸ“‰ Continuous downward trend (decline > 20%)
  • πŸ”» Weak rebounds (< 10%)
  • πŸ“Š Decreased trading volume
  • 😰 Pessimistic market sentiment
  • ⏱ Duration: weeks to months

Identification Method:

# Simple judgment: MA50 falling and price below MA50
price < MA50 and MA50 < MA50.shift(30)
Enter fullscreen mode Exit fullscreen mode

Response Strategies:

  • ⚠️ Reduce position size (50% or lower)
  • ⚠️ Raise stop loss standards
  • ⚠️ Decrease trading frequency
  • πŸ›‘ Consider suspending trading

Suitable Strategies:

  • βœ… Conservative strategies
  • βœ… Short selling strategies (if supported)
  • ❌ Not suitable: Aggressive momentum-chasing strategies

Backtesting Points:

# Test bear market (assume 2025-07~09 is bear market)
freqtrade backtesting \
  -c config.json \
  --strategy Strategy001 \
  --timerange 20250701-20250930

# Focus on drawdown and stop loss performance
Enter fullscreen mode Exit fullscreen mode

3. Range-Bound Market

Features:

  • ↔️ Sideways consolidation (fluctuation < 10%)
  • πŸ”„ Range-bound oscillation
  • πŸ“Š Moderate trading volume
  • 😐 Neutral market sentiment
  • ⏱ Duration: days to weeks

Identification Method:

# Simple judgment: price oscillating within range
price_range = (high_30d - low_30d) / low_30d
if price_range < 0.15:  # Fluctuation < 15%
    print("Range-bound market")
Enter fullscreen mode Exit fullscreen mode

Suitable Strategies:

  • βœ… Mean Reversion
  • βœ… Grid Trading
  • βœ… RSI overbought/oversold strategies
  • ❌ Not suitable: Trend Following

Backtesting Points:

# Test range-bound market
freqtrade backtesting \
  -c config.json \
  --strategy MeanReversionStrategy \
  --timerange 20250401-20250630
Enter fullscreen mode Exit fullscreen mode

Market Identification in Practice

Using TradingView for Identification:

  1. Open TradingView
  2. View BTC/USDT daily chart
  3. Add MA50 and MA200 indicators
  4. Observe the last 6-12 months of trends

Judgment Criteria:

Price > MA50 > MA200, and MA50 rising β†’ Bull Market
Price < MA50 < MA200, and MA50 falling β†’ Bear Market
Price fluctuating around MA50, MA50 flat β†’ Range-Bound Market
Enter fullscreen mode Exit fullscreen mode

9.3 Out-of-Sample Testing

What is Out-of-Sample Testing?

Definition:
Divide data into two parts:

  • Training Set (In-Sample): Used for strategy development and optimization
  • Testing Set (Out-of-Sample): Used to validate strategy effectiveness

Importance:

  • βœ… Validate strategy generalization ability
  • βœ… Avoid overfitting
  • βœ… Simulate real trading scenarios

Time Splitting Methods

Method 1: 70/30 Split

Standard Division:

Total Data: 2024-07-01 ~ 2025-09-30 (15 months)

Training Set: 2024-07-01 ~ 2025-03-31 (9 months, 60%)
Testing Set: 2025-04-01 ~ 2025-09-30 (6 months, 40%)
Enter fullscreen mode Exit fullscreen mode

Backtesting Commands:

# Training period backtest
freqtrade backtesting \
  -c config.json \
  --strategy Strategy001 \
  --timerange 20240701-20250331

# Testing period backtest
freqtrade backtesting \
  -c config.json \
  --strategy Strategy001 \
  --timerange 20250401-20250930
Enter fullscreen mode Exit fullscreen mode

Method 2: Multi-Period Rolling Test (Walk-Forward)

Principle:

Data: 12 months

Period 1: Train (Jan-Jun) β†’ Test (Jul-Aug)
Period 2: Train (Mar-Aug) β†’ Test (Sep-Oct)
Period 3: Train (May-Oct) β†’ Test (Nov-Dec)
Enter fullscreen mode Exit fullscreen mode

Advantages:

  • More comprehensive validation
  • Closer to live trading scenarios
  • Reduce single-period bias

Implementation Script:

#!/bin/bash

STRATEGY="Strategy001"
CONFIG="config.json"

echo "=== Walk-Forward Testing ==="

# Period 1
echo "Period 1: Train 2024-07~Dec, Test 2025-01~Feb"
freqtrade backtesting -c $CONFIG --strategy $STRATEGY --timerange 20240701-20241231
freqtrade backtesting -c $CONFIG --strategy $STRATEGY --timerange 20250101-20250228

# Period 2
echo "Period 2: Train 2024-Sep~2025-Feb, Test 2025-Mar~Apr"
freqtrade backtesting -c $CONFIG --strategy $STRATEGY --timerange 20240901-20250228
freqtrade backtesting -c $CONFIG --strategy $STRATEGY --timerange 20250301-20250430

# Period 3
echo "Period 3: Train 2024-Nov~2025-Apr, Test 2025-May~Jun"
freqtrade backtesting -c $CONFIG --strategy $STRATEGY --timerange 20241101-20250430
freqtrade backtesting -c $CONFIG --strategy $STRATEGY --timerange 20250501-20250630
Enter fullscreen mode Exit fullscreen mode

Method 3: Quarterly Split Testing

Principle:

Q1 (Jan-Mar): Test
Q2 (Apr-Jun): Test
Q3 (Jul-Sep): Test
Q4 (Oct-Dec): Test

Compare consistency of quarterly performance
Enter fullscreen mode Exit fullscreen mode

Backtesting Commands:

# Q1
freqtrade backtesting -c config.json --strategy Strategy001 --timerange 20250101-20250331

# Q2
freqtrade backtesting -c config.json --strategy Strategy001 --timerange 20250401-20250630

# Q3
freqtrade backtesting -c config.json --strategy Strategy001 --timerange 20250701-20250930

# Q4
freqtrade backtesting -c config.json --strategy Strategy001 --timerange 20251001-20251231
Enter fullscreen mode Exit fullscreen mode

Out-of-Sample Testing Evaluation Criteria

Qualified Standards:

Training Period Returns: +20%
Testing Period Returns: +15% (β‰₯ 75% of training period)

Training Period Drawdown: -5%
Testing Period Drawdown: -7% (≀ 150% of training period)

Training Period Sharpe: 3.0
Testing Period Sharpe: 2.3 (β‰₯ 75% of training period)

Conclusion: βœ… Strategy stable, can be used
Enter fullscreen mode Exit fullscreen mode

Warning Signals:

Training Period Returns: +30%
Testing Period Returns: +2% (only 6.7% of training period) ❌

Training Period Drawdown: -3%
Testing Period Drawdown: -15% (5x training period) ❌

Training Period Sharpe: 4.5
Testing Period Sharpe: 0.5 (only 11% of training period) ❌

Conclusion: ❌ Serious overfitting, cannot be used
Enter fullscreen mode Exit fullscreen mode

9.4 Avoiding Overfitting

Overfitting Identification Signals

Signal 1: Perfect Training Period, Poor Testing Period

Case:

Training Period (6 months):
  Trade Count: 150
  Win Rate: 95%
  Total Returns: +45%
  Sharpe: 5.2

Testing Period (3 months):
  Trade Count: 72
  Win Rate: 48%
  Total Returns: -12%
  Sharpe: -0.3

Diagnosis: Typical overfitting!
Enter fullscreen mode Exit fullscreen mode

Signal 2: Extremely Complex Strategy Parameters

Overfitting Strategy Example:

def populate_entry_trend(self, dataframe, metadata):
    dataframe.loc[
        (
            # 10 condition combinations
            (dataframe['ema5'] > dataframe['ema10']) &
            (dataframe['ema10'] > dataframe['ema20']) &
            (dataframe['rsi'] > 52.3) &  # Too precise
            (dataframe['rsi'] < 57.8) &  # Too precise
            (dataframe['macd'] > 0.0023) &  # Too precise
            (dataframe['volume'] > dataframe['volume'].shift(1) * 1.234) &  # Too precise
            (dataframe['close'] > dataframe['bb_lowerband'] * 1.012) &
            (dataframe['adx'] > 23.7) &
            (dataframe['cci'] < 87.3) &
            (dataframe['mfi'] > 42.1)
        ),
        'enter_long'] = 1
Enter fullscreen mode Exit fullscreen mode

Problems:

  • ❌ Parameters too precise (e.g., 52.3, 57.8)
  • ❌ Too many conditions (10 conditions)
  • ❌ May only suit specific historical data

Signal 3: Dramatic Performance Changes with Minor Parameter Adjustments

Test:

RSI Threshold = 30 β†’ Returns +25%
RSI Threshold = 31 β†’ Returns +2%
RSI Threshold = 29 β†’ Returns -5%

Conclusion: Strategy extremely sensitive to parameters, overfitting!
Enter fullscreen mode Exit fullscreen mode

Overfitting Prevention Methods

1. Simplify Strategy

Principles:

  • βœ… Condition count ≀ 5
  • βœ… Use integer parameters (e.g., 30, not 30.3)
  • βœ… Clear and understandable logic

Improvement Example:

# Simplified strategy
def populate_entry_trend(self, dataframe, metadata):
    dataframe.loc[
        (
            # Keep only 3 core conditions
            (dataframe['ema20'] > dataframe['ema50']) &  # Trend confirmation
            (dataframe['rsi'] > 30) &  # Not oversold
            (dataframe['volume'] > 0)  # Has volume
        ),
        'enter_long'] = 1
Enter fullscreen mode Exit fullscreen mode

2. Increase Testing Period

Recommendations:

Short-term strategies (5m-15m): At least 3 months of data
Medium-term strategies (1h-4h): At least 6 months of data
Long-term strategies (1d): At least 12 months of data
Enter fullscreen mode Exit fullscreen mode

3. Multi-Market Testing

Validation Checklist:

βœ… Bull market testing
βœ… Bear market testing
βœ… Range-bound market testing
βœ… High volatility period testing
βœ… Low volatility period testing
Enter fullscreen mode Exit fullscreen mode

4. Parameter Stability Testing

Method:
Fine-tune parameters, observe result changes

# Test RSI threshold stability
# Modify RSI threshold in strategy: 25, 30, 35
freqtrade backtesting -c config.json --strategy StrategyRSI25 --timerange 20250701-20250930
freqtrade backtesting -c config.json --strategy StrategyRSI30 --timerange 20250701-20250930
freqtrade backtesting -c config.json --strategy StrategyRSI35 --timerange 20250701-20250930

# If three versions have similar results, strategy is stable
Enter fullscreen mode Exit fullscreen mode

5. Use Regularization Techniques

Methods:

  • Limit maximum open positions
  • Set reasonable ROI gradients
  • Use conservative stop losses

Example:

# Conservative settings to prevent overfitting
stoploss = -0.10  # 10% stop loss (not too tight)
max_open_trades = 3  # Limit position count
minimal_roi = {
    "0": 0.10,     # Reasonable target (not too high)
    "120": 0.05,
    "240": 0.02
}
Enter fullscreen mode Exit fullscreen mode

πŸ’‘ Practical Tasks

Task 1: Different Time Period Comparison Testing

Choose a strategy (recommend Strategy001), test 3 different time periods:

# Period 1: 2024-10-01 ~ 2024-12-31 (Q4 2024)
freqtrade backtesting -c config.json --strategy Strategy001 --timerange 20241001-20241231 --timeframe 15m

# Period 2: 2025-01-01 ~ 2025-03-31 (Q1 2025)
freqtrade backtesting -c config.json --strategy Strategy001 --timerange 20250101-20250331 --timeframe 15m

# Period 3: 2025-04-01 ~ 2025-06-30 (Q2 2025)
freqtrade backtesting -c config.json --strategy Strategy001 --timerange 20250401-20250630 --timeframe 15m

# Period 4: 2025-07-01 ~ 2025-09-30 (Q3 2025)
freqtrade backtesting -c config.json --strategy Strategy001 --timerange 20250701-20250930 --timeframe 15m
Enter fullscreen mode Exit fullscreen mode

Task 2: Create Time Period Comparison Table

Time Period Trade Count Win Rate% Total Return% Max Drawdown% Sharpe Market Type
2024 Q4 ? ? ? ? ? ?
2025 Q1 ? ? ? ? ? ?
2025 Q2 ? ? ? ? ? ?
2025 Q3 ? ? ? ? ? ?
Average ? ? ? ? ? -
Std Dev ? ? ? ? ? -

Task 3: Out-of-Sample Testing 

# Training period: 2024-07-01 ~ 2025-03-31 (9 months)
freqtrade backtesting -c config.json --strategy Strategy001 --timerange 20240701-20250331 --timeframe 15m

# Testing period: 2025-04-01 ~ 2025-09-30 (6 months)
freqtrade backtesting -c config.json --strategy Strategy001 --timerange 20250401-20250930 --timeframe 15m
Enter fullscreen mode Exit fullscreen mode

Comparative Analysis:

Training Period Performance:
  Total Returns: ___________%
  Win Rate: ___________%
  Max Drawdown: ___________%
  Sharpe: ___________

Testing Period Performance:
  Total Returns: ___________% (____% of training period)
  Win Rate: ___________% (____% difference)
  Max Drawdown: ___________% (____% of training period)
  Sharpe: ___________ (____% of training period)

Conclusion:
  ☐ Strategy stable, testing period performance close to training period (β‰₯ 75%)
  ☐ Strategy unstable, testing period performance significantly decreased (< 50%)
  ☐ Possible overfitting
Enter fullscreen mode Exit fullscreen mode

Task 4: Judge Strategy Stability

Based on your test results, answer the following questions:

  1. Is the standard deviation of quarterly performance large?

    • Return standard deviation < 5% β†’ Stable βœ…
    • Return standard deviation > 10% β†’ Unstable ❌

  2. How is out-of-sample testing performance?

    • Testing period β‰₯ 75% of training period β†’ Stable βœ…
    • Testing period < 50% of training period β†’ Overfitting ❌

  3. Can it profit in different market conditions?

    • 3/4 quarters profitable β†’ Strong adaptability βœ…
    • Only 1/4 quarters profitable β†’ Poor adaptability ❌

  4. Final Judgment:

   ☐ Strategy stable and reliable, can enter simulation trading phase
   ☐ Strategy needs optimization, return to strategy adjustment phase
   ☐ Strategy seriously overfitted, abandon this strategy
Enter fullscreen mode Exit fullscreen mode

πŸ“š Knowledge Check

Basic Questions

  1. What is overfitting?
  2. What is the purpose of out-of-sample testing?
  3. How to judge if a strategy is stable?

Answers

  1. Strategy overfits specific characteristics of historical data, leading to poor performance on new data
  2. Validate strategy generalization ability, ensure strategy doesn't only work in specific time periods
  3. Consistent performance across different time periods, testing period performance close to training period (β‰₯ 75%)

Advanced Questions

  1. Why need to test strategies in bull, bear, and range-bound markets?
  2. Good performance in training period, poor in testing period, what does it indicate?
  3. How to prevent overfitting?

Thought Questions

  1. If a strategy performs excellently in all historical periods, does it mean live trading will also be good?
  2. What proportion of total data is most appropriate for out-of-sample testing?
  3. What advantages does Walk-Forward testing have over simple 70/30 split?

πŸ”— Reference Materials

Supporting Documentation

Recommended Reading

πŸ“Œ Key Points Summary

  1. Different time period testing is key to validating strategy stability
  2. Out-of-sample testing > In-sample testing: Testing period performance is more important
  3. Overfitting is the biggest trap in quantitative trading
  4. Strategies should be tested in bull, bear, and range-bound markets
  5. Consistency > Returns: Consistent performance is more important than occasional high returns
  6. Simple strategy > Complex strategy: Fewer conditions reduce overfitting risk

➑️ Next Lesson Preview

Lesson 10: Trading Pair Selection and Testing

In the next lesson, we will:

  • Learn how to select suitable trading pairs
  • Evaluate liquidity and volatility of trading pairs
  • Test strategy performance on different trading pairs
  • Build multi-pair portfolios

Preparation:

  • βœ… Download data for multiple pairs like BTC/USDT, ETH/USDT, BNB/USDT
  • βœ… Select a strategy with stable performance
  • βœ… Understand the difference between mainstream and altcoins

🎯 Learning Check Standards:

  • βœ… Can independently conduct out-of-sample testing
  • βœ… Can judge if a strategy is overfitted
  • βœ… Understand the impact of different market conditions on strategies
  • βœ… Can evaluate strategy stability

After completing these tasks, you have mastered the core skills of strategy validation! Ready to move on to trading pair selection learning! πŸš€

Top comments (0)