DEV Community: wanqiuxu

How to Turn Per-Spectrum Fitting Into NumPy Batch Fitting — SoftwareX Accepted It in 17 Days

wanqiuxu — Thu, 05 Nov 2026 03:48:42 +0000

Lab code that fits spectra one by one in a Python loop gets slow once you have thousands of curves. SPECTROview, an open-source Python desktop app for spectroscopic processing and fitting, replaces that loop with a vectorized batch fitting (VBF) engine built on NumPy. On an Intel Core i5-8365U laptop with 16 GB of RAM, the authors report 16×–62× speedups versus lmfit one-by-one fitting, depending on dataset size and model complexity. The largest test used 16,384 spectra, with peak memory of 845 MB for under a second before returning to about 80 MB after convergence. The SoftwareX paper that describes the software was received 2026-06-17, accepted 2026-07-04, and published 2026-07-08 — a 17-day path from receipt to acceptance in CrushSCI’s SoftwareX records.

What SoftwareX is

SoftwareX is published by Elsevier (ISSN 2352-7110) as an open-access (OA) journal. CrushSCI labels for the journal include: SCIE, OA, 2026 Rising Star Zone 3, 2025 CAS Zone 4, 2026 JCR Q3, with 479 articles published in 2025.

Scope

Per the journal’s Aims and Scope, SoftwareX is meant to give research software academic recognition: peer-reviewed software that can be cited, inspected, and reused, with credit for developers. It is multidisciplinary — mathematics and physics, environment, medicine and biology, humanities and social sciences — and also takes software in areas such as citizen science, cybersecurity, the digital economy, and energy. Domain-independent building blocks are welcome: math libraries, image-processing libraries, visualization tools, data-management tools, and similar.

A submission has two parts: a short descriptive paper of at most 4,000 words, plus an open-source software release with supporting materials. The paper is framed as documentation for software readers and potential users. Article types include Original software publication and Software update.

For developers, that means the code must be open, installable, and usable; the paper should explain what problem the software solves, how it is designed, and how others can use it — not a long algorithm-theory treatise.

The 17-day paper

In CrushSCI’s SoftwareX records, the shortest review cycle is 17 days:

Received: 2026-06-17
Accepted: 2026-07-04
Published: 2026-07-08

Those dates match the paper PDF: Received 17 June 2026, Received in revised form 28 June 2026, Accepted 4 July 2026, Available online 8 July 2026.

What the paper covers

Title: SPECTROview: An open-source application for interactive spectroscopic data processing, fitting, and visualization. Sole author: Van Hoan Le (CEA / LETI, Grenoble). Published in SoftwareX volume 35, article 102862.

Problem. Raman, photoluminescence, and similar experiments produce large numbers of spectra — a 2D scan or wafer map can hold tens of thousands. A common Python approach is lmfit curve fitting inside a per-spectrum loop. That becomes slow at scale, and Python’s GIL limits CPU-bound parallelism.

Software. SPECTROview is Python with a PySide6 UI and three workspaces: Spectra (single spectrum), Maps (2D maps), Graphs (statistical plots). The pipeline covers import from multiple instrument formats, baseline correction, multi-peak fitting, and export of publication-ready figures. It is GPL-3.0 on GitHub and installable with pip install spectroview.

Method: vectorized batch fitting (VBF). Instead of fitting spectra one by one, the engine stacks N spectra into an N×M matrix (M = points per spectrum), builds the Levenberg–Marquardt Jacobian as an N×M×K tensor (K = number of fit parameters), and solves all equations in one vectorized LAPACK call. Two further choices: closed-form analytic Jacobians for each peak shape (not numerical differences), and a convergence mask that drops already-converged spectra from later iterations.

The paper also states why the authors did not reuse batch LM implementations from PyTorch or TensorFlow: those rely on autodiff aimed at neural-net training, not closed-form peak shapes such as Gaussians and Lorentzians, and they do not conveniently handle spectra that converge on different iteration steps.

Results. On an Intel Core i5-8365U, 16 GB laptop, four representative datasets were run 10 times each. Versus lmfit one-by-one fitting, VBF speedups were between 16× and 62×, depending on data scale and model complexity. The largest dataset had 16,384 spectra; peak memory was 845 MB for under a second, then about 80 MB after convergence.

Limits. The authors note that VBF uses one fit model for every spectrum in a batch; with many overlapping peaks (for example more than 20–30), results grow more sensitive to initial parameters and constraints. For inhomogeneous data such as composites or layered structures, the user must define a single model that covers every peak that may appear. Speedups were measured on one laptop against lmfit one-by-one fitting only. Full vectorization trades memory for speed: memory grows linearly with N×M×K.

Takeaway for developers. The contribution is not a new mathematical method; it is rewriting a common slow path (nonlinear fitting in a Python loop) as batched tensor work, then shipping it as an installable tool with a UI and documentation — the kind of research software SoftwareX’s scope describes.

Source

Receipt, acceptance, and publication dates above come from CrushSCI’s paper records for SoftwareX and were checked against the Received / Accepted / Available online dates on the paper PDF. SoftwareX on CrushSCI: https://www.crushsci.com/journal/26

VM Placement Is a Moving Target: Reading the IDEAL SDN Paper

wanqiuxu — Tue, 06 Oct 2026 04:31:24 +0000

A VM scheduler changes the system it is trying to optimize. Place one workload, and the remaining CPU capacity, host utilization, and available network paths all change. The next placement decision now has a different input.

That feedback loop is the central idea in IDEAL: Impact-Aware Virtual Data Center Embedding With Incremental Analysis for SDN-Controlled Networks, by N. Preetham, Sourav Kanti Addya, T. G. Keerthan Kumar, and Saumya Hegde. The paper appears in IEEE Access; its DOI record identifies the published article. The authors' public manuscript provides the algorithm, experimental setup, and results discussed here.

The request includes a network

Virtual Data Center Embedding, or VDCE, allocates both compute and communication resources.

A request contains virtual machines plus virtual links between them. Each VM needs compute capacity. Each virtual link needs bandwidth along a physical route. A placement is successful when the infrastructure can satisfy both sets of demands.

This coupling matters. Two hosts can have enough spare compute capacity while the path between them lacks the bandwidth required by the application. A compute-only placement decision can therefore leave an otherwise promising request impossible to complete.

The paper represents compute demand using Computational Resource Blocks, or CRBs. In its model, one CRB corresponds to one CPU core and 512 MB of memory. Virtual links carry explicit bandwidth demands in Mbps.

Evaluate the change caused by a placement

IDEAL evaluates candidate hosts using two quantities:

Marginal energy increment: the additional host-side energy associated with placing the next VM on a candidate host.
Incremental load-balance distortion: the change in the distribution of residual resources across active hosts after that tentative placement.

The second quantity helps explain why maximizing consolidation alone can produce poor allocations. Concentrating workloads can reduce host activation, while leaving an uneven distribution of spare capacity for later requests. Spreading workloads widely can improve balance while increasing the number of active hosts.

Consider a hypothetical choice between an already active host and an idle host. The active host has enough capacity for the next VM, but little capacity would remain afterward. The idle host offers more room but adds activation overhead. The scheduler needs to compare the consequences of both choices against the current infrastructure state.

IDEAL uses the Analytic Hierarchy Process, or AHP, to weight its criteria and VIKOR to rank the candidate hosts. These are decision-support tools. The repeated calculation of placement impact, followed by immediate state updates, gives the ranking its current meaning.

After a placement, the scheduler evaluates the candidates again for the next VM.

Treat admission as a transaction

The embedding procedure has two stages: place the VMs, then map their virtual links onto physical paths.

Its control flow can be read as a resource transaction:

Identify feasible hosts for the next VM.
Estimate each candidate's energy and balance impact using the current state.
Rank the candidates, place the VM, and reserve its compute resources.
Repeat the evaluation for the next VM using the updated state.
Map the virtual links using bandwidth-feasible paths and reserve their bandwidth.
Accept the complete request, or release its reservations if any VM or link fails to map.

For link mapping, physical links with insufficient residual bandwidth are excluded from the candidate graph. The implementation uses a Dijkstra-based K-shortest-path search with K = 5.

The rollback behavior is especially useful to examine as an engineer. A failed request should leave the infrastructure state as it was before the admission attempt. IDEAL keeps a request-level reservation log covering placed VMs and reserved physical links, then restores those resources on failure.

That makes the bookkeeping part of the algorithm. A good ranking function still needs correct reservation and release behavior to produce reliable subsequent decisions.

What the experiment actually measures

The authors evaluate IDEAL in a Mininet and Ryu SDN emulation environment using Python and OpenFlow. The physical network has 54 hosts and 222 physical links, arranged as three spine switches, six leaf switches per spine, and three hosts per leaf.

Each virtual data center request contains 2-10 VMs. Five workload scenarios contain 200, 400, 600, 800, and 1,000 requests, with ten runs per scenario.

The comparison uses five heuristic baselines: CEVNE, DROI, SCA-R, LitE, and First-Fit. The authors report an average 48.70% reduction in active-host energy consumption across those baseline comparisons.

The accounting boundary matters here. The manuscript reports a much smaller reduction, around 3-4%, for total physical-network host energy, because that calculation includes the baseline power consumed by idle hosts. Consolidating workloads changes active-host energy more dramatically than it changes the modeled energy of the whole host fleet.

For infrastructure work, both measurements are useful: one describes the placement policy's effect on the active set, and the other describes its effect on all modeled hosts.

The ablation shows the trade-off

Table 6 of the manuscript compares energy-only selection, balance-only selection, and the full framework. The following values are reported means and standard deviations across the evaluated load scenarios:

Metric	IDEAL-E: energy only	IDEAL-B: balance only	Full IDEAL
Active-host energy, Wh	3,180 ± 126	4,875 ± 184	2,605 ± 96
Total physical-network host energy, Wh	9,038 ± 142	9,526 ± 168	8,790 ± 121
Hosts used	23.6 ± 1.4	31.4 ± 1.8	13.8 ± 0.9
Load-balancing efficiency	0.41 ± 0.03	0.68 ± 0.04	0.73 ± 0.03

Energy-only selection consumes less active-host energy than balance-only selection, but its load-balancing score is lower. The full framework combines the two criteria and achieves better values on both metrics in this experiment.

The evaluation covers a 54-host topology with synthetic arrivals and heuristic baselines. Testing the approach on a larger or production-oriented workload means measuring the repeated ranking cost, routing cost, and resulting allocations under that workload.

What to carry into a scheduler design

Three engineering ideas are particularly useful beyond this paper:

Re-evaluate after reservation. An accepted placement changes the inputs to the next decision.
Keep compute and network feasibility connected. Host capacity and path bandwidth belong to the same admission decision.
Make resource changes reversible. A reservation log supports complete rollback when a multi-resource request fails.

These ideas also suggest practical checks: a host's rank should be allowed to change after a reservation; a bandwidth failure should release the request's compute allocations; and an energy comparison should state whether idle hosts are included.

For the publication trail, the IEEE Access paper records on CrushSCI include this title and its DOI and publisher links, alongside the recorded receipt and acceptance dates. I maintain CrushSCI; the technical discussion here is based on the authors' manuscript.

This research explainer was prepared with AI assistance. Numerical results are attributed to the authors' experiments.