Hyperparameter Optimization Across Frameworks Made Simple - Neural DSL v0.2.3

Hey Dev.to community!
I’m excited to share the latest update to Neural DSL, a work-in-progress domain-specific language for defining, training, and debugging neural networks. With v0.2.3 (released March 16, 2025), I supercharged one feature I think you’ll love:

multi-framework Hyperparameter Optimization (HPO). Plus, I squashed some bugs and added new layer support to keep things moving forward. Let’s dive in!

---

🌟 Spotlight: Multi-Framework HPO (#434)

Imagine defining a neural network once and optimizing its hyperparameters for both PyTorch and TensorFlow without rewriting a single line. That’s what v0.2.3 brings to the table. Whether you’re tuning layer sizes, dropout rates, or learning rates, Neural DSL now handles HPO seamlessly across frameworks all from a single declarative config.

How It Works

Define your model with HPO parameters in the DSL:

network HPOExample {
  input: (28, 28, 1)  # MNIST input
  layers:
    Dense(HPO(choice(128, 256)))  # Sample units
    Dropout(HPO(range(0.3, 0.7, step=0.1)))  # Sample dropout rate
    Output(10, "softmax")
  optimizer: Adam(learning_rate=HPO(log_range(1e-4, 1e-2)))  # Log-scale LR
  train {
    epochs: 10
    search_method: "random"  # Or "bayesian"
  }
}

Run it with a single command, switching frameworks on the fly:

neural run hpo_example.neural --backend pytorch --output model_torch.py
neural run hpo_example.neural --backend tensorflow  --output model_tf.py

 Behind the scenes, Neural DSL:

1. Parses the HPO specs into a framework-agnostic model_dict.
2. Uses DynamicModel (PyTorch) or DynamicTFModel (TensorFlow) to sample parameters via Optuna.
3. Evaluates trials with a unified train_model function, supporting both backends.

 Here’s a peek at the magic in hpo.py:

def objective(trial, model_dict, hpo_params, train_loader, val_loader, backend='pytorch'):
    lr = trial.suggest_loguniform('lr', *hpo_params['lr']['range'])
    if backend == 'pytorch':
        model = DynamicModel(model_dict, trial, hpo_params).to('cuda' if torch.cuda.is_available() else 'cpu')
        optimizer = optim.Adam(model.parameters(), lr=lr)
    elif backend == 'tensorflow':
        model = DynamicTFModel(model_dict, trial, hpo_params)
        optimizer = tf.keras.optimizers.Adam(learning_rate=lr)
    val_loss, accuracy = train_model(model, optimizer, train_loader, val_loader, backend)
    return val_loss

def run_hpo(model_dict, hpo_params, backend='pytorch'):
    study = optuna.create_study(direction='minimize')
    study.optimize(lambda trial: objective(trial, model_dict, hpo_params, *get_data(), backend), n_trials=10)

🏄🏽‍♂️ Why It’s Awesome

• One Config, Two Frameworks
• No more duplicating effort for PyTorch vs. TensorFlow experiments.
• Flexible HPO: Supports choice (discrete), range (linear), and log_range (log-scale) for parameters like units, rates, and learning rates.
• Scalable: Ready to extend to ONNX or JAX with minimal tweaks.

This feature came from fixing test_hpo_integration_full_pipeline (#434), where we tackled optimizer HPO parsing and 3D input shape issues. Now, it’s a cornerstone of Neural’s cross-framework vision.

---

👨🏿‍💻 Other Goodies in v0.2.3

While HPO steals the show, here’s what else I’ve been up to:

• New Layers: Added LayerNormalization, InstanceNormalization, GroupNormalization, SqueezeExcitation, and Attention to the parser (#105, #106, #107, #118, #307). More building blocks for your models!
• Parser Fixes: Squashed bugs in Concatenate, Activation, Lambda, and Embedding parameter handling (#140, #329, etc.), plus better macro and device support (#136, #327, #328).
• Validation Boost: Enhanced checks for MaxPooling2D, BatchNormalization, Dropout, and Conv2D to catch errors early (#179, #363, #367, #368).
• Error Handling: Improved VisitError wrapping with line/column details (#159) for clearer debugging.

Check the full changelog for all the nitty-gritty.

---

🦾 What’s Next?

Neural DSL is still a WIP bugs lurk, and features are missing (like full ONNX HPO support).

Upcoming goals:

• Stabilize macro parsing for nested blocks.
• Expand layer support (more PyTorch layers, anyone?).
• Add interactive HPO visualizations with NeuralDbg.

Got ideas or bug reports? Join me on Discord or file an issue. Feedback keeps this project alive!

---

👾 Join the Journey

v0.2.3 is a step toward making neural network development declarative, flexible, and debuggable.
The multi-framework HPO feature is just the beginning imagine tuning models across PyTorch, TensorFlow, and beyond with one tool.
What do you think—how would you use HPO in your projects? Drop a comment below!
Happy coding,
[Lemniscate-SHA-256]
Twitter: @NLang4438 | Neural DSL GitHub