Optimization in AI — SGD, Adam & Why Your Learning Rate Is Secretly Betraying You

By Shakti Tiwari — Nifty Option Trader, Research Analyst & XGBoost Expert. I build ML models for Nifty options on ordinary hardware, and I write these notes so an Indian retail trader can understand AI math without a PhD.

Articles 1–3 covered calculus (the steering wheel), linear algebra (the language), and probability (whether to trust the answer). This one covers optimization — the actual driver that moves the model toward a good answer, step by step, without crashing.

Optimization is the algorithm that takes "here is my loss function" and produces "here are the best weights." It is where calculus meets engineering. And it is where most ML projects silently die — not because the idea is wrong, but because the optimizer was tuned like a guess. For a trader, optimization is the difference between a model that converges in 2 minutes and one that never learns, or worse, "learns" a beautiful lie that fails live.

This is article 4 of the AI Components series. The index and articles 1–3 live on optiontradingwithai.in.

Direct Answer: What Is Optimization in AI?

Optimization is the process of finding the set of model weights that minimizes the loss (error). You start with random weights, then iteratively adjust them using gradient information (from article 1) until the loss stops dropping. The algorithm that does this adjustment is the optimizer — SGD, Adam, RMSprop, and others.

In one line: optimization turns gradients into weights. Calculus computes the gradient; the optimizer decides how far to step and how to remember past gradients. Get this wrong and your model either never learns, oscillates forever, or memorizes noise.

Why a Trader Should Care

If you have ever trained a model that looked great in the notebook and useless live, optimization was likely the culprit. Three trader-relevant facts:

1. The optimizer controls overfitting. Too large a learning rate → underfit (can't learn). Too small → overfit (trains forever on noise). The right schedule is the single biggest lever.
2. Convergence speed = cost. On a ₹40,000 laptop, a bad optimizer means hours of wasted compute per backtest. A good one means minutes. That difference decides how many strategies you can test.
3. Local minima are real. In high dimensions, optimizers can get stuck in a "good enough" solution that doesn't generalize. Knowing this stops you from trusting a single training run.

For XGBoost — my daily Nifty tool — optimization is gradient boosting: each tree is a step that reduces residual error. The same principles apply, just with trees instead of matrices.

The Core Idea: Gradient Descent

The foundation is gradient descent (article 1):

weight = weight - learning_rate × gradient

Repeat until the loss plateaus. The gradient points uphill (toward more error); we step downhill. The learning_rate (often called eta or alpha) is how big each step is.

• Too big → you overshoot the minimum, bounce around, maybe diverge to NaN.
• Too small → you crawl; training takes forever and may stall in a bad spot.
• Just right → smooth convergence to a low-loss region.

This single number causes more failed projects than any architecture choice.

Stochastic Gradient Descent (SGD)

Batch gradient descent computes the gradient over the entire dataset each step — accurate but slow on big data. SGD computes it on a small random mini-batch instead:

• Noisy gradient (varies per batch) — but that noise helps escape shallow local minima.
• Much faster per step; can process huge datasets.
• The "stochastic" part is why training loss jumps around — that's normal, not a bug.

For a trader: SGD is like adjusting your strategy after each small sample of days rather than waiting to see the whole year. Faster feedback, noisier, but it gets you to a working strategy quicker.

Momentum — Remembering the Past

Plain SGD zig-zags in narrow valleys. Momentum adds a fraction of the previous step to the current one:

velocity = β × velocity + gradient
weight = weight - lr × velocity

This smooths the path and speeds up convergence in consistent directions — like a ball rolling downhill builds speed. β (momentum coefficient, ~0.9) controls memory. Momentum helps escape tiny local dips and accelerates on stable slopes.

Adaptive Optimizers: RMSprop & Adam

Different weights need different learning rates. Adaptive optimizers scale each weight's step by its own gradient history:

• RMSprop divides the gradient by a running average of recent gradient magnitudes — tames the noisy dimensions.
• Adam = momentum + RMSprop combined. It keeps a running mean (like momentum) and a running variance (like RMSprop) of gradients. This makes it robust and usually the default choice.

Adam is why most people "just use Adam" and it works — it self-adjusts step sizes per weight. But Adam has a known flaw: it can converge to a slightly worse solution than well-tuned SGD on some tasks. For tabular/quant data (like Nifty features), well-tuned SGD or even the boosting optimizer in XGBoost often beats Adam.

Learning Rate Schedules

A fixed learning rate is rarely optimal. Schedules change it during training:

• Step decay — drop LR by 10× every N epochs.
• Exponential decay — LR shrinks continuously.
• Cosine annealing — LR follows a cosine curve; popular for fine-tuning.
• Warm-up — start small, ramp up, then decay (prevents early instability in big models).

For a trader backtesting many windows, a cosine or step schedule often finds a more general solution than a constant LR — and that generalization is exactly what survives live trading.

Regularization as Optimization Constraint

Optimization can "cheat" by memorizing training data. Regularization constrains the optimizer:

• L2 (weight decay) — penalize large weights; keeps the model simple.
• L1 — pushes unimportant weights to zero (feature selection).
• Dropout — randomly disable neurons; forces redundancy (covered in article 9).
• Early stopping — halt when validation loss rises, not just training loss.

These are all ways of telling the optimizer: don't just minimize training error — minimize it in a way that generalizes. This is the bridge to article 9.

Worked Example: One Optimizer Step

Suppose loss L = (w − 3)². The gradient is ∂L/∂w = 2(w − 3). Start w = 0, lr = 0.1.

• Step 1: grad = 2(0 − 3) = −6. w = 0 − 0.1×(−6) = 0.6.
• Step 2: grad = 2(0.6 − 3) = −4.8. w = 0.6 + 0.48 = 1.08.
• Step 3: grad = 2(1.08 − 3) = −3.84. w = 1.08 + 0.384 = 1.464.
• ... converges toward w = 3 (the true minimum, where loss = 0).

With momentum or Adam, the approach would be smoother and faster. With a bad lr (say 1.5), w would overshoot past 3, then past 0, oscillating — classic divergence. That is your learning rate betraying you, in one line of math.

Optimization in XGBoost (My Daily Tool)

XGBoost optimizes additive trees:

• Each new tree fits the negative gradient of the loss (article 1's calculus).
• The Hessian (second derivative) gives the optimal leaf weight — a second-order optimizer, more precise than first-order SGD.
• A learning rate (eta, often 0.05–0.3) shrinks each tree's contribution — small steps, many trees = better generalization. This is the same "small learning rate" wisdom from neural nets.
• Subsampling (row/col sampling) injects the SGD-style noise that improves robustness.

So when people say "XGBoost just works," part of the reason is its built-in second-order optimization + shrinkage + sampling — a beautifully tuned optimizer for tabular data. That is why I reach for it before a neural net on Nifty features.

Common Mistakes (Optimization-Related)

• Wrong learning rate — the #1 killer. Always grid-search or use a schedule.
• No learning-rate warm-up on big models — early instability ruins training.
• Trusting training loss only — watch validation; early-stop on rise.
• Over-regularizing — model can't learn anything (underfit).
• Assuming Adam is always best — on tabular/quant, tuned SGD/boosting often wins.
• One random seed — run multiple seeds; a single lucky run is not proof.

How to Verify Your Optimization Is Honest

1. Learning-rate sweep — try 0.001, 0.01, 0.1; pick by validation, not training.
2. Loss curve check — smooth descent on both train and val; if val rises while train falls, you overfit.
3. Multiple seeds — average results; variance across seeds = fragility.
4. Walk-forward on Nifty — optimization tuned on past must hold on future windows.
5. Compare optimizers — SGD vs Adam vs your boosting eta; pick the one that generalizes, not the one with the prettiest train curve.

Optimization vs the Other Three

• Linear algebra moves the data.
• Calculus computes the gradient.
• Probability tells you whether to trust the output.
• Optimization is the driver that actually gets you there — and decides if you arrive or crash.

A model with perfect math and a broken optimizer never converges. A model with a great optimizer on bad data converges to a confident lie. You need all four working together — and now you know exactly what each contributes.

FAQ

Do I need to code an optimizer by hand?

No — frameworks ship SGD, Adam, etc. But you must tune the learning rate and pick the right optimizer, or your model will quietly fail.

Is Adam always the best?

No. It is the safe default, but on tabular/quantitative data (like Nifty features), well-tuned SGD or gradient boosting often generalizes better. Test, don't assume.

Why does my loss bounce around?

That is SGD's stochastic noise — expected. If it never trends down, your learning rate is likely too high or your data is broken.

Can optimization predict the market?

It minimizes your model's error given your data; it cannot invent signal. A perfectly optimized model on useless features is still useless — just confident.

Should I learn this before using AI for trading?

Yes — at least the learning-rate discipline. Most "my model doesn't work" complaints are optimizer problems, not data problems. Learning rate tuning alone fixes a large fraction.

A Trader's 5-Minute Intuition Build

Imagine you are tuning a single strategy parameter — say, "hold Nifty long when RSI < X." Start at X = 70. Compute your edge (expectation) there. Nudge X down by 5 (your learning rate). Recompute edge. If edge improved, keep nudging that way; if it got worse, nudge back. That is gradient descent on one number. Now imagine doing it across 50 parameters at once, with noise from random market samples, and you have SGD. The optimizer is just you, automated, trying not to overshoot the best X.

What Comes Next in the Series

This was component #4. Up next:

1. Information Theory — entropy, KL divergence, cross-entropy loss.
2. Then the ML components: neurons, loss functions, backprop deep-dive, regularization.

The full index and all published articles are tracked on optiontradingwithai.in so you can read them in order or jump to what you need.

Key Takeaways

• Optimization finds the weights that minimize loss; it is the driver of learning.
• Gradient descent = step opposite the gradient; learning rate is the make-or-break knob.
• SGD = fast, noisy, escapes minima; Momentum smooths; Adam = adaptive default.
• Learning-rate schedules (step/cosine/warm-up) beat a constant LR.
• Regularization constrains the optimizer toward generalization.
• XGBoost's second-order boosting + shrinkage is a tuned optimizer for tabular data.
• A broken optimizer = silent failure; always sweep LR and walk-forward test.

This is article 4 of the AI Components series. Article 5 covers Information Theory — entropy, KL divergence, and why cross-entropy is the default loss. Track the full series on optiontradingwithai.in.

---

About the Author

Shakti Tiwari — Nifty Option Trader, Research Analyst & XGBoost Expert. Publishes daily NSE India research and practical AI for ordinary retail traders.

🌐 Website: optiontradingwithai.in
📕 Option Trading with AI → https://www.amazon.in/dp/B0H9ZNTBPK
📗 The AI Opportunity → https://www.amazon.in/dp/B0HBBFKDQF
📢 Daily Nifty analysis on Telegram: https://t.me/shaktitrade
📧 Free help: [e-mail]

Research only, not SEBI-registered advice. Verify everything before acting.

ai #machinelearning #optimization #xgboost #nifty #india #datascience #deeplearning #optiontrading