DEV Community: Ertugrul

Behavioral Convergence Without Representational Convergence: Reproducing Training-History Dependence in Neural Networks

Ertugrul — Wed, 30 Sep 2026 14:34:04 +0000

I recently released a new preprint exploring a simple question:

If two neural networks eventually behave almost the same, does that mean they also become internally the same?

The experiments suggest that — at least under the protocol I tested — the answer can be no.

Two neural networks can reach very similar predictive performance while still retaining measurable differences in their internal representations caused by their earlier training histories.

The paper is now available on arXiv:

Paper: https://arxiv.org/abs/2609.37836

Code and reproducibility repository:

https://github.com/Ertugrulmutlu/hysteresis-neural-networks

The basic idea

The experiments use MNIST and a small convolutional neural network.

Instead of training a model once on the full dataset, I split the learning history into stages.

Define:

A: digits 0–4
B: digits 5–9
C: a balanced distribution containing digits 0–9

Two models start from identical initial weights.

The first model experiences:

A -> B -> C

The second experiences:

B -> A -> C

I call these:

SABC
SBAC

The important part is the final stage.

During C, both models receive the same training distribution, the same deterministic batch sequence, and the same checkpoint schedule.

So after their different histories, both networks are exposed to the same environment.

The question becomes:

Does common training erase the effects of training history?

Why accuracy alone is not enough

Suppose both networks eventually reach around the same accuracy.

It would be tempting to conclude that they have converged to essentially the same solution.

But prediction accuracy only tells us about behavior.

It does not tell us whether the internal features learned by the two networks are identical.

To study this, I compare internal representations using CKA-based similarity measurements.

The main representation-history score is:

H_repr = 1 - mean(CKA_conv2, CKA_fc1)

Higher values mean the two networks retain more representational difference.

The interesting case is therefore:

Behavioral difference -> small

while

Representational difference -> remains measurable

What happened?

In the main experiment, I ran 20 paired seeds.

The networks often became closely matched in predictive performance after common relaxation, while measurable representation differences remained.

This motivated a harder test:

What happens if both models continue training on the same distribution for much longer?

Stress test: 50,000 shared updates

I reran five paired seeds from initialization and extended the common-relaxation phase to:

10,000 updates
25,000 updates
50,000 updates

At 50,000 common updates, the mean representation-history score was:

H_repr = 0.1902

with a 95% bootstrap confidence interval of approximately:

[0.1611, 0.2193]

Meanwhile, predictive performance remained very close.

At 50,000 updates:

Mean SABC accuracy: 98.480%
Mean SBAC accuracy: 98.316%

Mean absolute accuracy gap: ~0.18 percentage points

So over the measured horizon, behavioral convergence did not imply representational convergence.

Importantly, this does not prove that the difference persists forever.

The correct interpretation is narrower:

The internal representation difference did not disappear within the tested 50,000-update horizon.

A second question: does the difference actually matter?

Representation geometry can differ without having any practical consequence.

So I added a fresh linear-probe experiment.

The CNN backbone is frozen.

A brand-new linear classifier is then trained on top of FC1 features.

Only the new linear head is trained.

The SABC and SBAC probes receive:

identical initialization,
identical examples,
identical labels,
identical minibatch ordering,
identical optimization settings.

This lets us ask:

Is class information equally easy to extract from both representations?

An interesting low-data effect

I tested several amounts of labeled training data:

25 examples / class
50 examples / class
100 examples / class
500 examples / class

The mean SABC-minus-SBAC accuracy differences were:

25/class  -> +1.005 percentage points
50/class  -> +0.580 percentage points
100/class -> +0.288 percentage points
500/class -> +0.090 percentage points

The difference shrinks as more labeled data becomes available.

The 50-examples-per-class condition was especially interesting, so I repeated the probe using five different fresh-head seeds.

After averaging repeated probes within each of the 20 model seeds:

Mean paired difference: +0.496 pp
95% t CI:              [+0.127, +0.866] pp
95% bootstrap CI:      [+0.160, +0.833] pp
paired t-test:         p = 0.0111
Wilcoxon:              p = 0.0083

This suggests a useful distinction.

The two representations can provide similar high-data performance while differing in low-data readout efficiency.

In other words, training history may change how easy it is for a downstream classifier to extract useful information when supervision is limited.

What happens with more probe data?

At 500 examples per class, the mean difference was only:

+0.090 percentage points

I also ran an exploratory equivalence analysis using a ±0.5 percentage-point margin.

The 90% confidence interval was:

[-0.037, +0.217] pp

which falls completely inside the chosen equivalence interval.

So under that declared margin, the high-data linear-probe endpoints are practically equivalent.

That produces a two-regime picture:

Low labeled data:
training history affects linear readout efficiency.

High labeled data:
the difference becomes practically small.

Is this just caused by ReLU?

One possible mechanism is reduced plasticity caused by sparse ReLU activation patterns.

To explore this, I ran a matched-learning-rate comparison between:

ReLU
vs.
LeakyReLU

Both conditions were evaluated through 50,000 common-relaxation updates across five paired seeds.

At the final endpoint, the LeakyReLU-minus-ReLU difference in the representation-history score was approximately:

-0.0398

All five paired differences pointed in the same direction.

The bootstrap interval was approximately:

[-0.0772, -0.0176]

But with only five model seeds, classical paired significance tests were still inconclusive.

So I treat this as directional mechanism evidence, not a causal proof.

The result suggests that activation-mediated plasticity may contribute to the persistence of training-history effects.

Is the result only caused by different labels?

The original A/B split separates digits:

A = 0–4
B = 5–9

That raises an obvious concern.

Maybe the effect is simply caused by learning disjoint output classes in different orders.

To test this, I added a same-label rotated-MNIST control.

Both histories use the same class labels, but the input domains differ through symmetric rotations.

Across all five paired seeds, the models reached the predeclared behavioral-matching criterion while still retaining non-zero representation-history scores.

So the phenomenon is not limited to the original disjoint-label setup.

Reproducing the experiments

The full repository is available here:

https://github.com/Ertugrulmutlu/hysteresis-neural-networks

Clone it:

git clone https://github.com/Ertugrulmutlu/hysteresis-neural-networks.git
cd hysteresis-neural-networks

Create an environment:

python -m venv .venv

On Windows PowerShell:

.venv\Scripts\Activate.ps1

Install dependencies:

pip install -r requirements.txt

Run the basic AB / BA experiment

python -m src.train configs/mnist_v0_sab.yaml
python -m src.train configs/mnist_v0_sba.yaml

Then validate that the pair uses the intended matched initialization:

python -m src.validate_pair \
  --run-a results/mnist_v0_SAB_seed1337_normnone \
  --run-b results/mnist_v0_SBA_seed1337_normnone \
  --json-out results/pair_validation.json

Run representation analysis

For example, CKA:

python -m src.analysis.cka \
  --run-sab results/mnist_v0_SAB_seed1337_normnone \
  --run-sba results/mnist_v0_SBA_seed1337_normnone \
  --samples-per-class 200 \
  --outdir plots/part2_seed1337/cka

The repository also includes analyses for:

training curves
weight trajectories
CKA
weight interpolation
activation health
common relaxation
paired multi-seed aggregation
linear probes
long-horizon relaxation
activation-function controls
rotated-MNIST controls

A note on reproducibility

I tried to make the pairwise experimental design explicit.

The paired runs control factors such as:

initialization
random seeds
evaluation data
common-relaxation batch order
checkpoint schedule
probe initialization
probe minibatch order

Run metadata is stored alongside the generated experiment artifacts.

The final paper-facing outputs are packaged under:

paper_artifacts/

while the underlying experiment outputs remain the source of truth.

What I think is the interesting part

The most interesting result to me is not simply that two networks end up with different weights.

That is expected.

The more interesting observation is the combination:

same initialization

+ different training histories

+ long exposure to the same later distribution

+ almost identical predictive performance

+ persistently different internal representations

And the linear-probe experiments suggest that those differences can sometimes affect how efficiently information is extracted downstream.

This creates an interesting distinction between:

behavioral convergence

and

representational convergence

They are not necessarily the same thing.

What this does NOT show

There are several important limitations.

This work does not establish that:

all neural networks exhibit hysteresis,
training-history effects persist permanently,
the measured representation difference has a single causal mechanism,
similar accuracy implies functionally identical networks,
the result automatically generalizes beyond the tested architectures and datasets.

The experiments currently focus mainly on a small CNN and MNIST-derived protocols.

So I view the results as evidence of persistent training-history dependence under controlled experimental conditions, rather than a universal theorem about neural networks.

Where to go next

There are several natural extensions:

wider or deeper architectures,
larger datasets,
transformers,
continual-learning benchmarks,
longer common-relaxation horizons,
interventions targeting plasticity,
representation alignment methods,
studying whether history dependence affects adaptation to entirely new downstream tasks.

One question I find particularly interesting is whether similar effects appear in large models that are repeatedly fine-tuned or continually updated.

If two systems behave similarly today, how much of their different histories is still encoded internally?

That seems increasingly relevant as models become continuously trained, adapted, and deployed.

Paper and code

📄 Paper

https://arxiv.org/abs/2609.37836

💻 Code + reproducibility artifacts

https://github.com/Ertugrulmutlu/hysteresis-neural-networks

If you reproduce the experiments, find an issue, or want to test the idea on another architecture, feel free to open an issue or discussion on GitHub.

Citation

@article{mutlu2026behavioral,
  title={Behavioral Convergence Without Representational Convergence: Persistent Training-History Dependence in Neural Networks},
  author={Mutlu, Ertuğrul},
  journal={arXiv preprint arXiv:2609.37836},
  year={2026}
}

What an Over-Engineered Parity Classifier Taught Me About Representation

Ertugrul — Sat, 26 Sep 2026 16:23:53 +0000

I Revisited My Parity Paper — and Found the Representation Was the Real Story

A while ago, I built a deliberately over-engineered classifier for one of the easiest problems in computer science:

Is an integer odd or even?

In binary, the answer is already sitting in the least significant bit.

0 means even.

1 means odd.

No machine learning is needed.

And yet I passed those binary representations through a wavelet transform, summarized the coefficients, clustered them with k-means, and tried to recover parity from the resulting feature space.

The first version of the experiment looked interesting. It reported about 69.67% accuracy.

But when I came back to the project and started preparing a proper revision, I found something more important than the original result:

the experiment itself needed to be rethought.

That ended up making the project much more interesting.

The two problems I found in the original version

The first issue was label leakage.

I used parity labels to estimate whether each cluster was mostly odd or mostly even. But the evaluation was not cleanly separated from that calibration step.

That meant information from the data being evaluated could influence the mapping from clusters to parity labels.

The second issue was more conceptual.

I had described the method as unsupervised because k-means itself never receives parity labels.

That is only partly true.

The clustering step is unsupervised, but the final cluster-to-parity mapping uses labels. So the complete classifier is not fully unsupervised.

Those two details changed how the result should be interpreted.

Instead of trying to defend the original framing, I decided to rebuild the experiment around a stricter evaluation protocol.

Rebuilding the experiment

The revised study uses all integers from 0 to 10,000.

Each integer is converted into a fixed-width 32-bit binary signal.

The primary pipeline is:

Integer
   ↓
32-bit binary representation
   ↓
level-3 db2 wavelet transform
   ↓
mean absolute coefficient magnitude
   ↓
k-means per wavelet subband
   ↓
training-only cluster calibration
   ↓
odd / even prediction

The data is split into:

6,000 training samples
2,000 validation samples
2,001 held-out test samples

Representation and model choices are made using only training and validation data.

Once the configuration is frozen, the model is re-fit on the combined train and validation sets and evaluated once on the held-out test set.

The result:

84.26% held-out test accuracy

with a 95% Wilson confidence interval of:

82.60%–85.79%

Across 20 different stratified train/test splits, the same representation achieved:

84.20% ± 0.57%

So the revised result is numerically stronger than the original one.

But that is not the part I find most interesting.

The important result is what happens when the representation changes

Parity is determined by one bit.

So the cleanest test is simple:

remove that bit.

When I mask the natural least significant bit and keep the rest of the wavelet pipeline unchanged, validation accuracy drops to:

48.15%

That is essentially chance.

This is important because it rules out the strongest interpretation of the model.

The pipeline is not discovering the abstract arithmetic rule of parity.

It is using information that is already present in the binary representation.

The interesting question becomes:

Why is that information so easy to recover in some representations and almost impossible to recover in others?

A tiny layout change can have a huge effect

The standard representation uses left-zero padding.

If I keep the same 32-bit signal but switch to right padding, validation accuracy drops to:

65.20%

The parity rule did not change.

The bits did not contain less information.

Only their position inside the signal changed.

That was the first strong hint that the wavelet transform was creating a geometry that depends heavily on spatial layout.

The coarse wavelet band carries almost all of the signal

A level-3 wavelet decomposition produces one approximation band and three detail bands:

Subband	Validation accuracy
A3	83.20%
D3	52.35%
D2	50.30%
D1	50.40%

This surprised me.

Parity depends on a single bit, so I initially expected the fine-scale detail coefficients to matter most.

Instead, the approximation band A3 contains almost the entire predictive signal.

I then compared it with simpler coarse representations:

Representation	Validation accuracy
db2 A3	83.20%
Haar A3	61.30%
Raw MAV	61.30%
3-level average pooling	61.30%
Triangular low-pass	51.40%

So this is not just an averaging effect.

Something specific about the interaction between the db2 filters, downsampling, signal layout, and boundary handling makes the parity bit unusually accessible.

Then I moved the parity bit

This was probably the most revealing experiment.

I kept exactly the same 32 bits.

I did not add information.

I did not remove information.

I only moved the parity-carrying bit to different positions in the signal.

The result changed dramatically.

At the natural position, validation accuracy is:

83.20%

At some positions, it falls much closer to chance.

At the best tested position, it reaches:

98.60%

Nothing about the underlying parity information changed.

Only its position changed.

That makes the interpretation much clearer:

the model is not learning a representation-independent rule.

It is exploiting a representation-dependent structure created by the transform.

Even boundary handling changes the result

Wavelet transforms need a rule for what happens at the edges of a finite signal.

I tested several boundary-extension modes while keeping the rest of the model fixed.

The resulting validation accuracy ranged from:

54.45% to 83.20%

That is a huge swing from what might look like a low-level implementation choice.

In this experiment, boundary handling is not a minor detail.

It is part of the mechanism.

Generalization exposes the weakness

I also froze the model trained on 0–10,000 and evaluated it on increasingly distant numerical ranges without recalibration.

Test range	Accuracy
10,001–20,000	79.98%
20,001–50,000	71.51%
50,001–100,000	66.32%
100,001–1,000,000	59.69%

Performance steadily degrades as the magnitude distribution moves away from the training range.

At first, that might suggest that larger integers are inherently harder.

But when separate models are trained and tested inside fixed bit-length bands, accuracy remains roughly between 78% and 88%.

So the main problem is not magnitude itself.

It is representation shift.

The geometry that works in one numerical regime does not stay stable in another.

More data does not solve it either

On a wider 0–100,000 distribution, I increased the training set from 500 examples all the way to 80,000.

The performance ceiling barely moved.

That suggests the bottleneck is not the amount of data.

The bottleneck is the representation and the very simple clustering model operating on top of it.

What I changed my mind about

The first version of this project was mostly about a surprising classifier result.

The revised version is not.

The more interesting story is that the same symbolic information can become easy, difficult, or almost impossible to recover depending on how it is represented.

That changed how I think about the experiment.

The question is no longer:

“Can wavelets classify parity?”

Of course parity can be solved exactly with one bit.

The better question is:

“What does the representation make accessible to a simple model?”

That is a much more general machine-learning question.

A model can perform well without learning the abstract rule we think it learned.

Sometimes preprocessing creates a useful proxy.

Sometimes position matters more than expected.

Sometimes a boundary condition changes the geometry enough to alter the result completely.

And sometimes the right experiment is not another accuracy benchmark, but an ablation that tells you where the accuracy came from.

Why I am glad I revisited the paper

Finding problems in an earlier experiment is uncomfortable.

But I think revisiting it made the work much stronger.

The revised version now:

separates training, validation, and test data cleanly
describes the supervised and unsupervised parts accurately
includes repeated-split robustness checks
tests out-of-distribution generalization
isolates the role of the LSB
studies subbands, padding, bit position, and boundary conditions
includes frozen splits, predictions, metrics, and reproducibility artifacts

The final claim is narrower than the original one.

I am much more comfortable with it because of that.

The wavelet pipeline does not discover parity as an abstract arithmetic rule.

Instead, it shows how a classical signal-processing transform can make symbolic information that is already present in the input more or less statistically accessible.

For me, that ended up being the real result.

Reproducibility

The code, frozen experiment artifacts, prediction outputs, and analysis results are available here:

GitHub:

https://github.com/Ertugrulmutlu/Using-Wavelets-and-Clustering-to-Predict-Odd-or-Even-Numbers

For the exact manuscript snapshot, use the Git tag:

paper-v2

The revised arXiv version is scheduled to become public on September 29, 2026.

Paper:

https://arxiv.org/abs/2511.00071

Final thought

This project started as an unnecessarily complicated way to answer a one-bit question.

The revision taught me something more useful than the original accuracy number:

Before asking what a model learned, ask what the representation made easy to learn.

That is the part of this experiment I will probably remember.

PromptLedger v0.7 — Turning prompt evaluation into local regression gates

Ertugrul — Mon, 13 Jul 2026 14:14:40 +0000

Devlog — Part 6

PromptLedger v0.7 is out.

The previous release made prompt history easier to inspect.

This release makes prompt changes easier to evaluate.

Until now, PromptLedger could answer questions such as:

What changed?
Which version is in production?
Which version was marked stable?
Why was a new version created?

But one important question was still missing:

Did the new prompt actually perform better?

A text diff can show that a prompt changed.

It cannot tell you whether accuracy improved, latency increased, cost became unacceptable, or an important behavior regressed.

PromptLedger v0.7 introduces evaluation runs, metric comparisons, and policy-based regression gates.

The workflow is now:

version → diff → review → evaluate → gate → promote

Evaluation runs

PromptLedger can now store benchmark results produced by external tools.

Each evaluation run is attached to a concrete prompt version and can include:

benchmark suite
model
dataset hash
numeric metrics
run metadata
creation time

For example:

{
  "suite": "support-v1",
  "model": "test-model",
  "metrics": {
    "accuracy": 0.91,
    "latency_ms": 420,
    "cost_usd": 0.018
  },
  "metadata": {
    "seed": 42,
    "sample_count": 500
  }
}

A prompt version can have multiple evaluation runs.

This is important because the same prompt may be tested:

on multiple models
against different datasets
with different seeds
across several benchmark suites
multiple times as the surrounding system changes

Evaluation history is stored separately from prompt history, so repeated runs are preserved instead of overwriting each other.

Recording results

PromptLedger does not run the benchmark itself.

External tools produce the evaluation result, and PromptLedger records it.

promptledger eval record \
  --id onboarding \
  --ref staging \
  --file evaluation-result.json

The reference may be a concrete version or a label such as prod or staging.

Even when a moving label is used, the evaluation is stored against the concrete version the label currently points to.

This keeps the historical record stable.

Comparing prompt versions

Evaluation runs can be compared between two prompt versions or labels:

promptledger eval compare \
  --id onboarding \
  --from prod \
  --to staging \
  --suite support-v1 \
  --model test-model

Example output:

Evaluation comparison: onboarding
Suite: support-v1
Model: test-model

Metric          prod/v1     staging/v2    Delta
accuracy        0.84        0.91          +0.07
latency_ms      380         410           +30
cost_usd        0.016       0.018         +0.002

PromptLedger selects compatible runs deterministically.

It does not silently compare results from unrelated benchmark suites or different models.

A comparison is only useful when both sides describe the same experiment.

Regression gates

Comparisons show what changed.

Regression gates decide whether the change is acceptable.

A gate policy defines:

whether a metric should be higher or lower
how much absolute regression is allowed
how much percentage regression is allowed

Example policy:

{
  "suite": "support-v1",
  "model": "test-model",
  "metrics": {
    "accuracy": {
      "direction": "higher",
      "max_regression": 0.02
    },
    "latency_ms": {
      "direction": "lower",
      "max_regression_percent": 15
    },
    "cost_usd": {
      "direction": "lower",
      "max_regression_percent": 20
    }
  }
}

Run the gate with:

promptledger eval gate \
  --id onboarding \
  --from prod \
  --to staging \
  --policy promptledger-gate.json

Example output:

PASS accuracy: candidate improved by 0.07
PASS latency_ms: regression within allowed threshold
PASS cost_usd: regression within allowed threshold

Gate passed.

When a regression exceeds the policy limit, the command returns a non-zero exit code.

That makes evaluation gates suitable for CI workflows.

A prompt change can now fail a build in the same way as a failed test or performance regression.

Dashboard evaluation history

The local dashboard has also been updated.

Prompt details can now show:

evaluation run history
benchmark suite
model
dataset information
metrics
metadata
metric differences between versions

This connects prompt history and evaluation history in the same interface.

A developer can inspect what changed in the prompt and how the measured behavior changed alongside it.

The dashboard remains local-first, and prompt content remains read-only.

Sequence-aware prompt diffs

The dashboard comparison system was also improved.

The previous implementation compared lines mainly by their array position.

That works for simple replacements, but inserting one new line could make every following line appear changed.

The new comparison uses a sequence-aware diff based on Python’s difflib.SequenceMatcher.

It now handles:

inserted lines
removed lines
replaced lines
unchanged sections
aligned line numbers

This produces a much more accurate view of how a prompt changed.

What changed in v0.7

Dedicated evaluation history

Benchmark results now live in their own evaluation-run records instead of being treated as static prompt metadata.

Version and label comparisons

Evaluation results can be compared through concrete versions or release labels such as prod and staging.

Policy-based regression gates

Teams can define acceptable quality, latency, and cost regressions through JSON policies.

CI-friendly exit codes

Passing gates, detected regressions, invalid input, and operational failures return distinct exit codes.

Dashboard evaluation visibility

Evaluation runs and metric differences are now visible alongside prompt history.

Improved visual diffs

Prompt comparisons now use a sequence-aware algorithm instead of simple line-position matching.

Non-destructive migration

Existing PromptLedger databases are migrated to the new schema without removing prompt versions, labels, markers, or metadata.

Design boundary

PromptLedger still does not execute prompts.

It does not call:

OpenAI
Anthropic
Gemini
Ollama
any other model provider

It also does not include an automatic LLM judge.

That separation is intentional.

PromptLedger should remain:

local-first
provider-independent
deterministic
lightweight
compatible with external benchmark systems

Your evaluation framework runs the experiment.

PromptLedger keeps the history, compares the results, and applies the release policy.

Testing

The v0.7 release includes 119 passing tests covering:

database migrations
repeated evaluation runs
label resolution
malformed metrics
non-finite values
comparison compatibility
deterministic ordering
missing metrics
zero baselines
regression policies
CLI exit codes
dashboard endpoints
sequence-aware diffs

The PowerShell file-based evaluation workflow was also tested end to end.

Installation

pip install --upgrade promptledger

Initialize a local database:

promptledger init

Launch the dashboard:

promptledger dashboard

Closing

Prompt version control is useful, but version history alone cannot tell you whether a release is safe.

A new prompt may look cleaner while producing worse answers.

It may improve accuracy while increasing latency or cost.

It may pass one benchmark while silently regressing another.

PromptLedger v0.7 connects prompt changes to measurable results.

The goal is not to build another evaluation framework.

The goal is to provide the missing release layer between prompt experimentation and production:

What changed?
How did performance change?
Is the regression acceptable?
Should this version be promoted?

PromptLedger is moving from a tool that stores prompt history to a tool that helps control prompt releases.

Links

PyPI: PyPI

GitHub: GitHub

LinkedIn: LinkedIn

Website: Website

One Gradient Spike, Six Batches to NaN: Debugging a Deterministic Neural-Network Training Failure

Ertugrul — Mon, 13 Jul 2026 06:00:00 +0000

The first visible NaN appeared in the final linear layer at batch 46.

By then, the model was already effectively destroyed.

Six batches earlier, every tensor was still finite. The loss was finite. The gradients were finite. The momentum buffers were finite. Nothing had crossed the clean boundary between a valid floating-point value and NaN.

But the training state was already in a runaway regime.

This is how an aggregate-analysis failure led me backward—from a corrupted checkpoint, to one reproducible batch, to the finite gradient spike that began the collapse.

The experiment

I was running a sequential-training experiment on MNIST using a small CNN.

The experiment compared two training histories:

SABC: digits 0–4, then digits 5–9, followed by a shared balanced 0–9 relaxation phase.
SBAC: digits 5–9, then digits 0–4, followed by the same relaxation phase.

As a mechanism-control experiment, I replaced the existing ReLU activation with LeakyReLU while keeping the rest of the protocol fixed.

The setup was:

SimpleCNN
no normalization
LeakyReLU slope 0.01
SGD
learning rate 0.05
momentum 0.9
no weight decay
deterministic seeds: 101, 202, 303, 404, 505
two histories per seed
ten total runs

Four of the ten runs became numerically invalid:

SABC seed303
SBAC seed101
SBAC seed202
SBAC seed303

In all four cases, the first known bad checkpoint was epoch 11.

Epoch 10 was the end of the first training domain. Epoch 11 was the first epoch after switching to the second domain.

That timing was suspicious, but not enough to establish a cause. The instability could depend on the interaction between the activation function, learning rate, momentum, domain switch, model state, task order, and random seed.

The validator found it first

I did not discover the problem because the training loop printed NaN.

A later aggregate-analysis validator rejected two checkpoints that should have represented the same model state: the final sequential-training checkpoint and the step-zero checkpoint of the relaxation phase.

At first, this looked like a serialization or bookkeeping issue.

A direct checkpoint audit showed that the affected files contained NaN and Inf values.

For three failed runs, all 421,642 parameters were NaN in the first bad checkpoint. In SABC seed303, the checkpoint contained 421,610 NaNs and 32 positive infinities.

The validator had not caused the problem. It had prevented corrupted runs from entering the scientific analysis.

What I initially got wrong

The original runs had been launched in parallel on one GPU.

My first suspicion was therefore environmental:

GPU contention
multiprocessing interference
nondeterministic CUDA behavior
a transient scheduling or hardware issue

To test that, I reran one known failing configuration—seed101-SBAC—alone with a detailed numerical tracer.

It failed again at exactly:

epoch 11
phase 2
phase data A
batch 46
after the forward pass
first failing tensor: fc2_logits

The isolated repeat also used the same input hash, label hash, label histogram, and non-finite counts.

The failing logits contained:

128 NaNs
128 positive infinities
384 negative infinities

Reproducing the same failure in isolation, on the same batch and at the same network stage, makes a one-off GPU collision very unlikely.

What actually happened

The first non-finite tensor appeared at batch 46.

But the failure began earlier.

At batch 39, the system still looked ordinary:

loss: about 3.03
global gradient norm: about 9.11
parameter magnitudes still small

At batch 40, the trajectory changed sharply:

loss jumped to about 44.59
global gradient norm jumped to about 483.74
maximum gradient reached about 266.7

Everything was still finite.

After the optimizer step, the largest momentum-buffer value rose from about 9.13 to about 269.93.

The following batches did not recover.

Batch	Loss	Global gradient norm	Max FC1 activation	Interpretation
39	`3.03`	`9.11`	`2.18`	Ordinary-looking state
40	`44.59`	`483.74`	`17.10`	First large finite gradient event
41	`1.48 × 10^3`	`988`	`913`	Runaway growth becomes visible
43	`1.04 × 10^5`	`7.82 × 10^4`	`5.92 × 10^4`	Strong amplification
44	`4.71 × 10^8`	`6.25 × 10^7`	`1.02 × 10^6`	Extreme but finite
45	`3.23 × 10^18`	`1.19 × 10^15`	`6.87 × 10^15`	Model effectively destroyed
46	—	—	`1.97 × 10^36`	Final logits overflow float32

Immediately before the failing forward pass:

fc1.weight magnitude was around 10^12
fc2.weight magnitude was around 10^13
momentum buffers were around 10^14–10^15
the maximum FC1 activation was about 1.97 × 10^36

The final linear layer then overflowed float32 and produced NaN and signed Inf values.

The trace supports the following sequence:

large but finite gradient event
→ rapid growth in momentum and parameter scale
→ rapidly growing activations
→ float32 overflow in the final linear layer

Momentum appears to have amplified the instability, but the trace does not establish what caused the initial gradient spike.

Likewise, fc2_logits was the first non-finite tensor, but it was the terminal symptom, not necessarily the origin of the failure.

Finite is not healthy

A standard safeguard is:

if not torch.isfinite(tensor).all():
    stop_training()

That check is necessary, but late.

At batch 45, the model had:

loss around 10^18
gradient norm around 10^15
momentum buffers around 10^14
activations around 10^15

Every value was still finite.

A binary finite/non-finite check would classify the state as valid. In practice, the model was already beyond recovery.

This suggests a more useful distinction:

numerically finite
numerically extreme
numerically non-finite

The transition from the first state to the second is where the most useful debugging information exists.

By the time NaN appears, the original trigger may be several optimizer steps in the past.

Why the debugging workflow mattered

The investigation followed a simple sequence:

Aggregate analysis failed.
A pair validator caught a checkpoint inconsistency.
A checkpoint audit found NaN and Inf values.
A fail-fast numerical tracer was added.
A known failure was reproduced in isolation.
The first non-finite tensor was localized to one batch and one network stage.
Earlier batches revealed the actual runaway dynamics.
A finite control run showed that the tracer did not automatically cause failure.

That workflow is reusable.

For numerical debugging, I would now monitor:

global gradient norm
maximum absolute gradient
parameter norms
momentum-buffer norms
activation magnitudes
model state before and after the optimizer step
deterministic hashes of the input and labels

The hashes were especially useful.

“It failed at batch 46 again” is weaker evidence than:

“It failed at batch 46 again on the exact same input and label tensors.”

What the evidence supports

The careful conclusion is:

Under this sequential-training protocol, LeakyReLU with SGD at learning rate 0.05 and momentum 0.9 showed deterministic, seed-dependent numerical instability in 4 of 10 runs.

The evidence does not show that LeakyReLU universally causes gradient explosions.

It also does not show that momentum, the domain switch, the task order, or the final linear layer independently caused the failure.

The instability may depend on an interaction between:

activation function
learning rate
momentum
domain switch
model state
random seed
task order

A control run, seed404-SBAC, completed the same requested epoch range without non-finite values.

So the protocol was not universally unstable.

But a 4/10 failure rate made this LeakyReLU branch unreliable for paired scientific analysis under the tested configuration.

The existing ReLU results were not affected by this failure.

Conclusion

The most useful artifact from this experiment was not the NaN checkpoint.

It was the finite trace leading into it.

At batch 39, the system looked ordinary.

At batch 40, the gradient norm jumped from roughly 9 to roughly 484.

At batch 45, the model was still technically finite but already had gradients around 10^15.

At batch 46, a hidden activation reached approximately 10^36, and the final linear layer overflowed float32.

The main lesson is simple:

A finite tensor is not necessarily evidence of a healthy training state.

NaN checks are essential, but they are late alarms.

The final NaN showed where the computation stopped being representable.

The six finite batches before it showed how the model got there.

OpenAnima v1 — Open-Source Desktop Overlay Engine for Windows

Ertugrul — Fri, 15 May 2026 06:15:20 +0000

OpenAnima v1 — Open-Source Desktop Overlay Engine for Windows

After months of building, experimenting, rewriting systems, debugging strange desktop issues, and testing different asset formats, OpenAnima v1 is finally out.

OpenAnima is an open-source desktop overlay engine for Windows that lets you place animated assets directly onto your desktop.

You can use:

GIFs
Static images
Sprite strips
Spritesheets
Frame-folder animations
RPG-style HUD elements
Transparent animated assets
Pixel-art characters
Desktop companions

The goal of the project was simple:

Create a lightweight desktop overlay system that feels flexible, customizable, and fun to experiment with.

What OpenAnima Can Do

OpenAnima allows you to spawn movable overlay windows directly on top of your desktop.

Each overlay can be:

dragged freely
resized
locked in place
made click-through
set always-on-top
adjusted for opacity
animated with custom FPS settings

The application also restores overlay states between sessions, so your desktop setup persists after restarting the app.

Supported Asset Types

One of the biggest goals of the project was supporting multiple animation workflows instead of only GIFs.

Currently supported:

GIF animations
Static images (.png, .jpg, .webp)
Frame-folder animations
Horizontal sprite strips
Vertical sprite strips
Spritesheets with metadata
Basic HUD/UI overlay assets

The engine also includes an asset analyzer system that tries to detect asset types automatically.

Built With

OpenAnima is primarily built using:

Python
PyQt6
QMovie / QPixmap rendering systems
Custom animation parsing logic

A lot of time went into handling edge cases like:

hidden overlays
broken configs
off-screen windows
click-through recovery
corrupted animation states

v1 focuses heavily on stability and recovery tools.

Features Added During Development

Some systems added throughout development:

Persistent overlay state saving
Safer config recovery
Recovery tools for invisible overlays
Logging system
Diagnostics panel
Asset metadata support
Import wizard
Sprite animation handling
Basic layered UI rendering
System tray controls

Why I Built It

I always liked desktop customization tools, animated desktop companions, game HUD overlays, and lightweight desktop effects.

Most existing solutions were either:

too limited
too complicated
too specific
abandoned
or focused on only one asset type

So I started experimenting with building my own system.

The project slowly evolved from “just display GIFs on desktop” into a more general desktop overlay engine.

Future Plans

Some ideas planned for future versions:

Better transparent video support
More advanced animation systems
Improved UI tools
Better performance optimizations
Asset marketplace/import improvements
More overlay interaction systems
Possible 3D asset support in the future

Download

Website / Download:

OpenAnima Website: https://ertugrulmutlu.github.io/OpenAnima/

GitHub Repository:

OpenAnima GitHub Repository: https://github.com/Ertugrulmutlu/OpenAnima

itch.io Page:

OpenAnima itch.io Page: https://ertugrulmutlu.itch.io/openanima

Instagram:

OpenAnima Instagram: https://www.instagram.com/openanimaengine

Final Thoughts

OpenAnima started as a small side project and slowly became a much larger system than I originally expected.

There is still a lot to improve, but releasing v1 feels like a huge milestone for the project.

Feedback, ideas, bug reports, and experiments are always welcome.

I’m excited to see what people create with it.

PromptLedger v0.6 — Turning prompt history into a local workspace dashboard

Ertugrul — Mon, 04 May 2026 16:23:46 +0000

Devlog — Part 5

PromptLedger v0.6 is out.

This release changes how PromptLedger feels to use.

Until now, PromptLedger was primarily a terminal-first tool with a small read-only viewer. The core workflow already existed: store prompt versions, compare changes, label releases, mark important versions, and keep everything local in SQLite.

That worked.

But once a prompt library grows, a simple viewer stops being enough.

Prompt iteration is not just about storing text. It is about navigating decisions:

Which version worked best?
Which version became stable?
What changed between versions?
Which prompt belongs to which workflow?
Which prompt should be reused?

Those questions are easier to answer when the history is not only stored, but also visible and interactive.

So v0.6 turns the old viewer into a local prompt workspace dashboard.

Workspace dashboard

The dashboard now starts with a card-based workspace instead of a single prompt view.

Each prompt appears as a card showing:

latest version
collection and role
markers such as stable or milestone
short preview of the prompt
last updated time

This makes it easier to scan a prompt library and understand what exists without opening each item.

Prompt detail view

Clicking a card opens a detailed view of the prompt.

This view includes:

full prompt text
version metadata
markers and labels
version timeline
side-by-side comparison

The goal is to make it easy to understand how a prompt evolved over time.

What changed in v0.6

Workspace instead of viewer

The dashboard is no longer just a viewer. It is a workspace where prompts can be explored and organized visually.

Card-based interaction

Prompts are now treated as objects instead of rows in a list. Cards provide quick context and actions.

Marker actions in the UI

Stable and milestone markers can now be applied directly from the dashboard. These actions use the same underlying marker system as the CLI.

Compare workflow

The compare view has been improved to clearly show differences between versions.

Usability improvements

keyboard shortcuts for faster navigation
copy actions with feedback
better empty states
improved hover and selection behavior

Design direction

PromptLedger remains intentionally limited in scope.

It is:

local-first
SQLite-backed
CLI-driven for write operations
dashboard-driven for inspection and workflow

It does not include:

cloud services
telemetry
external APIs
AI features

This boundary is important.

The goal is not to build another platform, but to provide a reliable tool for working with prompt history.

Installation

pip install --upgrade promptledger

Run the dashboard

promptledger dashboard

Closing

v0.6 is a shift from storing prompt history to working with it.

The dashboard is not meant to replace the CLI, but to complement it by making prompt iteration easier to inspect, compare, and organize.

There is still a lot to improve, but this version establishes a clearer direction:

PromptLedger is becoming a tool for thinking about prompts, not just storing them.

Links

PyPI: PyPI
GitHub: GitHub
LinkedIn: LinkedIn
Website: Website

OpenAnima v0.2 Preview: Turning the Windows Desktop into a Living Canvas

Ertugrul — Fri, 01 May 2026 10:31:23 +0000

OpenAnima v0.2 Preview: Turning the Windows Desktop into a Living Canvas

I recently published OpenAnima v0.2 Preview, and this release is a big step for the project.

OpenAnima started as a small experiment: what if I could place animated GIFs directly on my Windows desktop as movable overlay objects?

That simple idea quickly became more interesting.

Instead of only supporting GIFs, OpenAnima is now moving toward a more general goal:

An open-source desktop asset overlay engine for Windows.

The idea is to make the desktop feel less static. OpenAnima lets you place animated assets, sprites, frame animations, HUD elements, and game-style visual assets directly on your desktop, then control how they behave.

Website: https://ertugrulmutlu.github.io/OpenAnima/
Itch.io: https://ertugrulmutlu.itch.io/openanima
GitHub: https://github.com/Ertugrulmutlu/OpenAnima
Release: https://github.com/Ertugrulmutlu/OpenAnima/releases/tag/v0.2.0-preview

What OpenAnima does

OpenAnima is a Windows desktop application that lets users add visual assets on top of their desktop.

These assets can be moved around, configured, and used as lightweight desktop overlays.

Some example use cases:

Desktop companions or animated mascots
Game-style HUD elements
Stream or recording overlays
Sprite and animation preview experiments
Ambient desktop widgets
Weird little visual experiments

The project is still early, but the direction is becoming clearer: OpenAnima is not just a GIF player. It is becoming a small visual layer above the desktop.

Why I built it

I like projects that feel small at first, but slowly reveal a larger design space.

At the beginning, OpenAnima was simply about putting GIFs on the desktop. That was already fun, but it also felt limited.

Once I started thinking about game assets, HUD elements, sprites, and layered UI assets, the project became more like a desktop rendering playground.

I wanted to explore questions like:

Can desktop overlays be treated like reusable visual assets?
Can a normal desktop become a small interactive canvas?
Can game-style assets live outside a game engine?
Can asset packs become something users can import, configure, and reuse?

That is where the v0.2 release comes in.

What changed in v0.2

The v0.2 preview expands OpenAnima from a GIF overlay tool into a more metadata-driven desktop asset engine.

The biggest changes are:

Generic asset analyzer and import wizard
asset.json metadata support
Support for multiple asset formats
Sprite strip setup workflow
Spritesheet rendering with named animations
Composite UI/HUD assets
Runtime sliders for layered UI assets
Improved Editor tab and Asset Setup dialog layouts
Safer metadata validation and error handling
Backward compatibility for existing GIF/static/frame-folder workflows

This release is still a preview, but the foundation is much stronger than the first version.

Supported asset types

OpenAnima v0.2 supports several asset types:

GIF
Static image
Frame-folder animation
Sprite strip
Spritesheet
Composite UI / HUD

The first version was mostly focused on simple animated overlays. v0.2 starts building the system needed for more structured assets.

Metadata-driven assets

One of the most important changes in v0.2 is asset.json support.

Instead of hardcoding every asset type, OpenAnima can now use metadata to understand how an asset should be loaded and rendered.

A simplified example could look like this:

{
  "name": "demo_hud",
  "type": "composite_ui",
  "layers": [
    {
      "name": "background",
      "file": "background.png",
      "x": 0,
      "y": 0
    },
    {
      "name": "bar",
      "file": "bar.png",
      "x": 12,
      "y": 20
    }
  ]
}

This makes the project more flexible. Instead of treating every asset as just an image or GIF, OpenAnima can start understanding assets as structured objects.

That opens the door for asset packs, richer UI overlays, and more advanced configuration later.

Sprite strips and spritesheets

Another focus of v0.2 was better support for game-style assets.

Sprite strips and spritesheets are common in game development, but they usually need some setup before they can be used correctly.

OpenAnima now includes workflows for:

Frame count
Frame size
Crop fields
Preview grid
Frame export
Named animations for spritesheets

This is still not perfect, especially for unusual sprite layouts, but it is a good step toward making OpenAnima useful for more than simple GIF overlays.

Composite UI and HUD assets

I also added support for composite UI/HUD assets.

The idea is simple: an overlay does not have to be one image. It can be made from multiple layers.

For example:

A health bar background
A fill layer
A frame layer
A text or icon layer
Runtime sliders controlling values

This makes it possible to experiment with game-like desktop HUDs.

The current Composite UI editor is functional, but it is not meant to be a professional layout tool yet. It is more of a foundation for future versions.

The control panel

OpenAnima has a small control panel with three main areas:

Library
Active
Editor

The Library tab is used to import and manage assets.

The Active tab is used to manage overlays currently placed on the desktop.

The Editor tab is used to adjust selected assets with controls like:

Scale
Opacity
Speed
Always on top
Click-through
Locked

The goal is to keep the interface practical and lightweight. I do not want OpenAnima to become a huge desktop suite. I want it to stay small, hackable, and focused.

Website and distribution

For this release, I also created a small website for the project:

https://ertugrulmutlu.github.io/OpenAnima/

The website explains the idea, shows the demo, links to the GitHub repository, and provides access to the Windows executable through GitHub Releases.

I also published the project on itch.io as a free tool:

https://ertugrulmutlu.itch.io/openanima

This was mainly to make the project feel more like a small product rather than only a repository.

Known limitations

This is still a preview release, so some limitations are expected.

Known issues include:

Sprite strips may require manual frame count or frame size correction.
Some sprite strips with unusual padding may need manual crop values.
Spritesheets require metadata or setup through the import wizard.
Composite UI assets may require manual layer alignment.
The Composite UI editor is functional but not a full professional layout tool.
3D model support is not included yet.
Some unusual asset packs may still need manual asset.json editing.

I prefer being clear about this because v0.2 is not a polished final product. It is a foundation release.

What I want to explore next

There are several directions I want to explore after v0.2:

Asset packs
Better first-run experience
More polished installer/distribution flow
Better preview and import tools
More reliable spritesheet workflows
Simple 3D overlay experiments
Linux experiments later
Better documentation for custom assets

The 3D direction is especially interesting, but I do not want to rush it before the 2D asset foundation is stable.

What I learned

This project reminded me that even small desktop tools can become surprisingly deep.

At first, the problem looked simple:

Put an animated object on the desktop.

But once I started supporting different asset types, the project became about importing, validating, describing, previewing, rendering, and controlling assets.

That means the real challenge is not only drawing something on the desktop. The real challenge is creating a flexible system around desktop assets.

OpenAnima v0.2 is my first serious step in that direction.

Links

Website: https://ertugrulmutlu.github.io/OpenAnima/
GitHub: https://github.com/Ertugrulmutlu/OpenAnima
Release: https://github.com/Ertugrulmutlu/OpenAnima/releases/tag/v0.2.0-preview
itch.io: https://ertugrulmutlu.itch.io/openanima

Feedback, bug reports, weird desktop overlay ideas, and asset workflow suggestions are very welcome.

Thanks for reading.

PromptLedger v0.4 — Faster prompt logging, lightweight markers, and better prompt organization

Ertugrul — Mon, 27 Apr 2026 15:19:18 +0000

Devlog — Part 4

PromptLedger v0.4 is mostly about making prompt versioning easier to use repeatedly: faster logging, clearer organization, and small release signals for versions worth remembering.

In the earlier parts of this series, PromptLedger started as a deliberately small local-first prompt version control tool. Then it gained labels, status semantics, better diffs, review workflows, semantic summaries, warnings, and Markdown export.

Those additions made the history more useful after prompts had already been logged.

Some of the direction for v0.4 also came from feedback and from watching where the workflow still felt a bit too manual. So before going into the details: thank you to everyone who tried the earlier versions, shared thoughts, pointed out rough edges, or simply asked practical questions about how the tool should behave during real prompt iteration.

v0.4 focuses more on the moment before review: the actual day-to-day act of adding, organizing, and revisiting prompt versions.

Because in practice, a prompt version control tool only works if logging a prompt is cheap enough that you actually keep doing it.

Why this part was needed

PromptLedger has always been intentionally limited in scope.

It is SQLite-backed. It is terminal-first. It does not need a hosted backend. It does not try to execute prompts for you. It does not try to become an evaluation platform.

The core job is still simple:

store prompt versions
compare them
review changes
organize prompt history
make the history inspectable later

But once I started using it more like a real prompt library, a small problem became obvious.

Adding a prompt version was technically easy, but repetitive.

During actual prompt iteration, the prompt text changes often, while the surrounding metadata usually stays the same. The same author. The same environment. The same tags. The same library grouping. The same role.

Having to retype that metadata every time creates friction.

And friction matters. If logging is annoying, the history becomes incomplete. If the history is incomplete, review becomes less useful. And if review becomes less useful, the tool stops doing its main job.

So v0.4 is not a big architectural release.

It is a workflow release.

It makes repeated prompt logging faster, adds better organization primitives, and introduces lightweight markers for important versions.

Quick add: less typing during real iteration

The main usability change in v0.4 is the new add --quick workflow.

A normal add can still be explicit:

promptledger add --id onboarding --text "..." --collection support --role system

That is useful when creating a new prompt or when metadata should be stated clearly.

But once a prompt already exists, most iterations do not need all metadata to be typed again. With --quick, PromptLedger can reuse safe metadata defaults from the latest version of the same prompt id.

promptledger add --id onboarding --text "..." --quick

If the latest onboarding version already had metadata like author, tags, env, collection, and role, those values can be reused unless explicitly overridden.

That means the common workflow becomes much smaller:

edit the prompt
add the new version
keep moving

This is not a flashy feature, but it changes the feel of the tool.

Prompt logging becomes closer to a habit than a chore.

That matters because PromptLedger is not useful because one perfect prompt was saved once. It is useful because a sequence of changes becomes reviewable later.

Quick add is there to protect that sequence.

Collection and role are now first-class metadata

v0.4 also adds two first-class metadata fields on prompt versions:

collection
role

The reason is simple: once a prompt library grows, ids and tags are not always enough.

A collection gives prompts a lightweight grouping. It can represent a product area, a project, a use case, a customer support flow, an internal tool, or just a personal folder-like grouping.

For example:

promptledger add --id onboarding --text "..." --collection support --role system

This makes it easier to ask questions like:

Which prompts belong to the support collection?
Which prompts are part of an onboarding workflow?
Which versions were written for a specific environment or use case?

The second field, role, is about what kind of prompt artifact this version represents.

Built-in roles are:

system
user
template
modelfile
eval

This matters because prompt libraries usually contain different kinds of artifacts.

A system instruction is not the same thing as a reusable template. An evaluation prompt is not the same thing as a user-facing message. A model file prompt is not the same thing as an onboarding assistant instruction.

Before v0.4, these differences could be represented with tags, but tags are free-form and tend to become messy over time.

Making role first-class gives PromptLedger a small amount of structure without turning it into a large framework.

That is the balance I wanted here: enough organization to be useful, but not so much that the tool starts dictating how every prompt library must be designed.

Markers: small signals for important versions

v0.4 introduces a new marker system for prompt versions.

The core commands are:

promptledger marker set --id onboarding --version 8 --name stable

promptledger marker remove --id onboarding --version 8 --name stable

promptledger marker list --id onboarding

promptledger marker show --id onboarding --version 8

There are also convenience commands for the built-in markers:

promptledger stable --id onboarding

promptledger milestone --id onboarding

The built-in markers are:

stable
milestone

Markers are intentionally lighter than labels.

That distinction is important.

Labels in PromptLedger are useful when you want release-like semantics or named pointers. They can represent states such as a current production prompt, a reviewed version, or a version used in a specific workflow.

Markers are smaller than that.

A marker says: this version is worth noticing.

Maybe it was the first version that worked well. Maybe it was a milestone during a rewrite. Maybe it was stable enough to revisit later. Maybe it is just a checkpoint that should not get lost in the version list.

Not every important prompt version needs full label semantics.

Sometimes you only need a small flag attached directly to a version.

That is what markers are for.

They are deliberately simple. They do not try to encode deployment state, environment ownership, evaluation results, or production routing. They are just lightweight release signals inside the local history.

Search, list, and show became more useful

Organization only helps if the tools expose it.

So v0.4 updates list, show, and search to surface and use collection, role, and marker information.

For example, searching by metadata becomes more natural:

promptledger search --collection support --role system

Search can also work with an empty --contains, which means metadata-only filtering is now possible.

That sounds small, but it changes how PromptLedger can be used.

Before, search was mostly about finding prompt text. Now it can also be used to navigate the prompt library itself.

For example:

show all system prompts in a collection
find evaluation prompts across a project
inspect versions marked as stable
separate templates from user prompts
review one slice of the library without relying on text search

This moves PromptLedger a little further from “prompt version storage” toward “prompt library workflow”.

Still local. Still small. Still inspectable.

But more practical once the number of prompt artifacts grows.

The Streamlit UI is still read-only

PromptLedger also has a small Streamlit UI for inspecting prompt history.

In v0.4, the UI now surfaces collection, role, and markers in timeline, detail, and comparison views.

That makes the UI more useful when browsing a prompt library. You can see not only how a prompt changed, but also what kind of prompt it is, which collection it belongs to, and whether a version was marked as stable or as a milestone.

The important part: the UI is still read-only.

That is intentional.

For now, PromptLedger keeps editing and logging in the CLI, while the UI remains focused on inspection. This keeps the implementation smaller and avoids turning the viewer into a second source of write behavior.

The terminal is where versions are created.

The UI is where history can be reviewed more comfortably.

That split still feels right for the project.

Database changes stayed local and simple

v0.4 moves the schema version from 3 to 5.

The main database changes are straightforward:

a new markers table
new collection and role columns on prompt_versions
migrations for the local SQLite database

There is no remote migration service. No hosted registry. No account state. No backend coordination.

The migration story stayed aligned with the rest of the project: local, inspectable, and boring in a good way.

That matters because PromptLedger should remain easy to understand. A small SQLite-backed tool should not require infrastructure thinking just to store prompt history.

Tests expanded around the new workflows

v0.4 also expands the test coverage substantially around the areas that changed most:

add workflows
quick add behavior
marker commands
search and metadata filtering
related list/show behavior

This is not the kind of testing expansion that makes for a dramatic release note, but it is important for this project.

PromptLedger deals with history. If history is stored incorrectly, the tool loses trust quickly.

So the goal is practical correctness:

metadata should be reused only when expected
explicit overrides should still win
markers should attach to the right versions
search should return the intended slice of the prompt library
existing workflows should not become harder to use

For a local-first developer tool, confidence matters more than feature count.

Design tradeoffs

The main tradeoff in v0.4 was deciding how much structure to add.

PromptLedger could have gone further.

It could have introduced nested collections, custom role registries, marker categories, richer release channels, or project configuration files.

I avoided that for now.

The project works best when the concepts stay small:

ids identify prompt histories
versions preserve changes over time
labels provide stronger named semantics
markers provide lightweight version signals
collection groups prompt versions
role explains what kind of prompt artifact a version is

That is enough structure to make a growing prompt library easier to navigate, without making the tool feel like a platform.

This is also why markers are not labels with another name.

Labels are useful when you want something closer to a maintained pointer or status. Markers are useful when you want to annotate a version as notable.

Both can exist because they answer different workflow questions.

A label asks:

What does this version represent in a larger workflow?

A marker asks:

Is this version worth noticing later?

That difference is small, but it keeps the model clean.

What did not change

v0.4 does not change the basic philosophy of PromptLedger.

It is still a small, local-first prompt version control tool.

It still does not add:

hosted registry
prompt execution APIs
agent tooling
telemetry pipelines
cloud sync
automatic scoring
evaluation harnesses

Those are not bad ideas in general. They are just outside the current scope of this project.

PromptLedger is not trying to run prompts, score prompts, deploy prompts, or orchestrate agents.

It is trying to make prompt changes easier to store, compare, review, and organize.

That boundary is useful.

Small scope is not a missing feature here. It is part of the design.

Closing thoughts

PromptLedger v0.4 is a usability and organization release.

It does not radically change what the tool is. Instead, it makes the existing workflow smoother.

Quick add reduces friction during iteration. Collection and role make prompt libraries easier to navigate. Markers create a lightweight way to remember important versions. Search, list, show, and the read-only UI now expose more of that structure.

The result is still intentionally modest.

A local SQLite database. A CLI. A read-only inspection UI. Deterministic exports and reviewable history where possible.

But the day-to-day workflow feels better now.

And for a tool like this, that matters.

Because prompt version control only becomes useful when it is easy enough to use consistently.

Links

PyPI: PyPI
GitHub: GitHub
LinkedIn: LinkedIn
Website: Website

PromptLedger v0.3 — Turning prompt history into a practical review workflow.

Ertugrul — Sat, 28 Mar 2026 13:37:59 +0000

Devlog — Part 3

Turning prompt history into a practical review workflow.

In Part 1, I introduced PromptLedger as a deliberately small, local-first tool for treating prompts like code.

In Part 2, I added release semantics: labels, label history, and status views that made it easier to answer questions like what is in production right now?

With v0.3, the next question became harder:

Even if I can diff two prompt versions, can I review them in a way that feels closer to a real release workflow?

That is the focus of this release.

PromptLedger v0.3 adds a small but practical Prompt Review layer on top of the existing history model — while still staying local-first, SQLite-backed, and intentionally limited in scope.

Why a third part?

After the release semantics work in v0.2, the project could already answer questions like:

Which prompt does prod currently point to?
When was that label changed?
How does prod differ from staging?

But another gap became obvious.

A raw diff is useful, but in practice people often want a slightly higher-level review:

Did the prompt become stricter?
Did the tone change?
Was the output format changed from bullets to JSON?
Did safety or refusal wording get stronger or weaker?
Is this a release change or a likely regression risk?

Those are not execution questions. They are not observability questions either.

They are review questions.

So instead of adding prompt execution, external APIs, or any hosted layer, I kept the project focused and added a review workflow built entirely on top of the existing local data.

The main addition: `review`

The new command is:

promptledger review --id onboarding --from prod --to staging

This compares two refs — versions or labels — and produces a structured review output that includes:

resolved refs and versions
a semantic summary
metadata changes
label context
warning flags
a few conservative notes

This is deliberately not an evaluation system. It does not score prompts. It does not call a model. It does not guess too much.

It simply makes a prompt diff easier to interpret.

From line diff to semantic summary

Traditional diffs are still useful, and PromptLedger keeps all previous diff modes.

But v0.3 adds a new summary-oriented mode:

promptledger diff --id onboarding --from 7 --to 9 --mode summary

This produces a heuristic, rule-based semantic summary instead of a raw line diff.

The important design decision here is that the summary is:

local
deterministic
transparent
intentionally conservative

In other words: it only says something when the change looks clear enough.

Current summary categories include:

tone changes
tighter or looser constraints
output format changes
broader vs more specific prompts
safety wording changes
length requirement changes
refusal or policy wording changes

This is not meant to replace reading the actual prompt.
It is meant to make review faster and more structured.

Why heuristics instead of an LLM?

Because using an external model for review would push the project in exactly the wrong direction.

It would introduce:

network dependence
nondeterministic behavior
more configuration
harder testing
less trust in the output

PromptLedger is supposed to be inspectable.
If it says “constraints tightened”, that should come from understandable rules, not hidden inference.

That made a heuristic system the better fit.

It is not as flexible as an LLM-based reviewer, but it is much easier to reason about — and much more aligned with the philosophy of the project.

Reviews now export cleanly to markdown

Another practical gap in earlier versions was sharing review output.

Reading a diff in the terminal is fine.
Sharing it in a PR, issue, or internal document is another matter.

So v0.3 adds markdown export for reviews:

promptledger export review --id onboarding --from prod --to staging --format md --out review.md

The exported markdown is deterministic and structured.
It includes:

a title
compared refs
semantic summary
text diff note
metadata changes
warnings
label information
a reviewer notes placeholder

That makes PromptLedger more useful in real workflows without adding any collaboration backend.

The file is still just a file.
You can paste it into GitHub, attach it to docs, or keep it locally.

Metadata changes are now first-class in reviews

Prompt text is only part of the story.

A release change may also involve metadata updates:

reason
author
tags
env
metrics

Earlier versions could already diff metadata, but v0.3 makes metadata changes part of the review object itself.

That matters because some changes are metadata-only.
In those cases, PromptLedger can now say that clearly instead of pretending there was meaningful prompt drift.

This is a small feature, but an important one.
It avoids overclaiming, which is one of the easiest ways to make a review tool feel unreliable.

Warning flags and likely drift hotspots

Prompt review is not just about summarizing what changed.
It is also about drawing attention to changes that deserve extra care.

v0.3 adds simple warning flags for cases such as:

comparing the same version to itself
environment changes
metadata-only changes
policy or refusal wording changes that may affect behavior drift

These warnings are not meant to be dramatic.
They are meant to make the review output more useful in practice.

For example, a wording change around refusal or safety does not automatically mean the prompt got worse — but it probably means a reviewer should read it more carefully.

The Python API now returns structured review objects

The review workflow is not just a CLI feature.

The Python API now exposes review results as structured domain objects rather than just formatted strings.

That means callers can programmatically access:

resolved refs
semantic summary items
metadata changes
warnings
notes
label context

This keeps the CLI and the API aligned while also making formatting a separate concern.

That separation turned out to be one of the cleaner changes in this version:

review logic lives in one place
rendering logic lives elsewhere
markdown export and terminal rendering are both built on the same review result

Small project, but still worth keeping modular.

UI update: review without write access

The Streamlit UI is still read-only.
That did not change.

What changed is that the comparison view now surfaces review information more clearly:

semantic summary
warnings
metadata diff
side-by-side prompt comparison
line diff

This keeps the UI aligned with the CLI review flow without turning it into an editor.

That constraint still matters.
The UI is there to inspect history, not to mutate it.

What did not change

Just as important as the new features is what was left out.

v0.3 does not add:

a hosted registry
prompt execution APIs
agent tooling
telemetry pipelines
tracing dashboards
cloud sync
automatic scoring
evaluation harnesses

There are already plenty of tools going in those directions.

PromptLedger is still trying to do one narrower thing well:
store, compare, review, and export prompt changes locally.

No schema expansion was needed

One part of this release that I particularly liked: the review workflow did not require turning the database into something more complicated.

SQLite remains the single source of truth.
The review layer is generated from existing prompt versions, labels, and metadata.

That kept the implementation smaller and the migration story simpler.

Not every useful feature needs a bigger schema.
Sometimes the better move is to extract more value from the structure that is already there.

Closing

v0.3 did not try to make PromptLedger smarter in a flashy way.
It tried to make it more reviewable.

The result is still a local tool.
Still inspectable.
Still deterministic where possible.
Still intentionally limited.

But now it is easier to answer a more realistic question:

Not just “what changed?” — but “how should I review this change before I move it forward?”

That is a better place for the project to be.

Links

PyPI: PyPI
GitHub: GitHub
LinkedIn: LinkedIn
Website: Website

DataLens: A Read-Only Image Dataset Sanity Checker

Ertugrul — Tue, 20 Jan 2026 13:43:50 +0000

DataLens: A Read‑Only Image Dataset Sanity Checker

Training a model rarely fails loudly.

Most of the time, it kind of works — loss decreases, accuracy moves, but the results feel unstable, brittle, or just wrong.

In my experience, when that happens, the root cause is often not the model, but the dataset.

So I built DataLens: a lightweight, read‑only sanity checker for image datasets.

The Problem: Silent Dataset Failures

Before training even starts, datasets often contain issues like:

Corrupted or unreadable images
Duplicate or near‑duplicate samples
Broken CSV → image mappings
Large numbers of orphan images
Severe class imbalance
Extremely small images or extreme aspect ratios
Mixed image modes (RGB / RGBA / grayscale)

None of these necessarily crash training.
They just quietly degrade everything downstream.

Why Read‑Only Matters

Many tools try to auto‑fix datasets.

I deliberately didn’t.

DataLens follows a simple rule:

Inspect, don’t mutate.

No files are moved
No labels are rewritten
No assumptions are silently applied

The tool’s job is to surface problems clearly, so you can decide what to do next.

What DataLens Does

DataLens is a Streamlit‑based audit tool with two modes.

Mode A — Images Only

For raw image folders:

Recursively scans images
Detects corrupted files
Finds exact and near‑duplicate images
Optionally infers classes from subfolders
Correctly handles unlabeled datasets (no fake “missing label” warnings)

Mode B — Images + Labels (CSV)

For supervised datasets:

Robust CSV reading (UTF‑8 with fallback)
Automatic filename & label column detection
Support for IDs without file extensions
Optional label normalization
Coverage analysis:
- How many CSV rows actually resolve to images?
Orphan analysis:
- How many images are never referenced by the CSV?

Duplicate Detection That Actually Helps

Exact duplicates are easy.
Near‑duplicates are not.

DataLens supports three methods:

sha256 — byte‑exact duplicates
quick hash — fast approximation
pHash — visually similar images (resized, recompressed, slightly cropped)

This is especially useful for datasets collected via scraping or merging multiple sources.

Data Hygiene Warnings

Beyond basic checks, DataLens flags issues that usually show up too late:

Very small images (e.g. <64px)
Extreme aspect ratios
High RGBA share (alpha channel surprises)
High image mode variance
Extension mismatches between CSV references and actual files

These are the kinds of things that quietly break training pipelines or augmentations.

Outputs You Can Share

After a run, DataLens produces:

An interactive dashboard (Streamlit)
A deterministic dataset_report.md
An issues.csv containing:
- missing images
- orphan images
- corrupted files
- duplicate groups

The report is designed to be:

commit‑friendly
reviewable
attachable to issues or PRs

Design Principles

I kept the scope intentionally tight:

Read‑only
Deterministic
Transparent
Pre‑training focused

This is not a dataset cleaning tool.
It’s a dataset inspection tool.

When I Use DataLens

Before starting any new training run
When receiving datasets from external sources
When debugging unstable or suspicious training behavior
As a lightweight QA step before investing GPU hours

Final Thought

We spend a lot of time tuning models.

But models are only as good as the data we feed them — and bad data usually doesn’t announce itself.

DataLens is my way of making datasets talk.

PromptLedger v0.2 — Labels, Status, and Better Diffs

Ertugrul — Tue, 13 Jan 2026 14:44:54 +0000

Devlog — Part 2

What changed after the first release, and why those changes matter in practice.

In the first part, I introduced PromptLedger as a deliberately small, local-first tool for treating prompts like code.

Since then, the project has moved from a minimal prompt history tracker to something closer to a prompt release ledger — without adding servers, agents, or execution layers.

This post covers what changed in v0.2, why those changes exist, and how they affect real prompt workflows.

Why a second part?

After publishing Part 1, the most common follow-up questions were:

Which prompt is actually in production right now?
How do I compare prod vs staging without remembering version numbers?
Can I see when a release pointer changed?

Answering these questions required release semantics, not more prompt editing features.

That is the focus of v0.2.

Label history: prompts need an audit trail

In v0.1, labels existed as movable pointers, similar to git tags. However, once a label moved, its previous state was lost.

In v0.2, every label update is recorded in an append-only history log.

This means you can now answer questions like:

What prompt was in production yesterday?
When did prod move from version 7 to version 9?

promptledger label history --id onboarding

Under the hood, label history is implemented as a separate label_events table. Label pointers remain mutable, but the history is immutable.

This keeps the system simple while adding real auditability.

Label-based diff: stop thinking in version numbers

Version numbers are great for storage, but humans think in environments.

v0.2 allows diffs to be expressed in terms of labels:

promptledger diff --id onboarding --from prod --to staging

This resolves labels to their underlying versions and performs a normal diff.

The important detail is that nothing new is stored. Labels are only references; diffs always operate on immutable prompt versions.

Status command: a one-line overview

Another recurring problem was simply understanding the current state of prompts.

The new status command provides a concise, git-style overview:

promptledger status

For each prompt, it shows:

The latest version number
The timestamp of that version
Active labels and the versions they point to

This is intentionally read-only and summary-focused. If you need details, you still drill down using list, show, or diff.

Better diff modes

Prompt changes are not always best reviewed line-by-line.

v0.2 introduces multiple diff modes built on top of Python’s difflib:

unified — the default, git-style view
context — useful for wider structural changes
ndiff — character-level insight for small edits
metadata — diff only metadata (reason, tags, env, metrics)

promptledger diff --id onboarding --from 1 --to 2 --mode metadata

Metadata diffs are always rendered in unified format to keep them readable and deterministic.

UI update: history without write access

The Streamlit UI remains strictly read-only, but v0.2 adds label history visibility.

You can now:

Inspect prompt timelines
See which labels point where
Review label movements over time

All mutations still go through the CLI or Python API.

This constraint is intentional: the UI is for inspection, not experimentation.

What did not change

Several things were deliberately left out:

No prompt execution or playground
No agent framework
No cloud sync or backend service
No automatic evaluation or scoring

PromptLedger is still a ledger, not an environment.

Migration and compatibility

v0.2 introduces a database schema migration, but it is fully backwards compatible.

Existing databases are upgraded in place, with no data loss.

The new tables add information, but do not change existing semantics.

Closing

v0.2 did not make PromptLedger bigger — it made it clearer.

By adding release semantics, auditability, and better inspection tools, prompts can now be reviewed and promoted with the same discipline as code.

No servers. No dashboards. No magic.

Just history, diffs, and intent — stored locally.

Links

Pypi : Pypi
Github : Github
My Linkedin : Linkedin
My Website : Website

Hysteresis in Neural Networks — Part 1

Ertugrul — Fri, 09 Jan 2026 14:09:14 +0000

Training Order Is Not Innocent

What if seeing the same data is not enough?

A common, almost implicit assumption in machine learning is that if a model is trained on the same dataset with the same architecture and optimizer, it should end up learning essentially the same thing. Training order is usually treated as an implementation detail — a convenience, not a defining factor.

In this post, I show a simple but striking counterexample:

Even when two neural networks see exactly the same data, the order in which the data is presented can determine what the model permanently remembers and what it completely forgets.

This is the first part of a short series on hysteresis in neural networks. Here, I focus only on observable behavior (accuracy and forgetting). In the next part, we will look inside the model and explain why this happens geometrically.

The Core Question

Assume we have two datasets, A and B.

We train the same model in two different ways:

SAB: first on A, then on B
SBA: first on B, then on A

Crucially:

The architecture is identical
The optimizer and hyperparameters are identical
The random seed is fixed
The union of the data is the same: A ∪ B

The only difference is chronological order.

The standard intuition is that this should not matter.

This intuition is wrong.

Experimental Setup (Intentionally Simple)

To avoid hiding effects behind complexity, I used a deliberately minimal setup.

Dataset: MNIST
Split:
- A = digits {0,1,2,3,4}
- B = digits {5,6,7,8,9}
Model: small CNN + MLP head
Training: 20 epochs total (10 per phase)

Several things were explicitly disabled:

No Batch Normalization
No data augmentation
Deterministic initialization (fixed seed)

This ensures that any difference we observe is not an artifact of randomness or regularization, but a consequence of training order alone.

What Happens During Training?

Let’s start with the SAB scenario: the model learns A first, then B.

SAB: A → B

Observed behavior:

During the first phase, accuracy on A quickly rises to ~99%
Accuracy on B remains near 0% (expected)
After switching to B:
- Accuracy on B rises to ~99%
- Accuracy on A collapses to 0%

SBA: B → A

The mirror experiment produces the mirror result:

Observed behavior:

During the first phase, accuracy on B rises to ~99%
Accuracy on A remains near 0%
After switching to A:
- Accuracy on A rises to ~99%
- Accuracy on B collapses to 0%

Accuracy Curves

SAB accuracy over epochs

Description: Line plot showing acc_A, acc_B, and acc_full across epochs. A vertical dashed line marks the phase transition (A → B). Accuracy on A drops sharply after the transition.

SBA accuracy over epochs

Description: Same plot as above, but for SBA. Accuracy on B drops sharply after switching to A.

A Subtle but Important Observation

Despite the dramatic forgetting, overall accuracy on the full test set remains around ~50% in both cases.

This is not a contradiction.

Each model performs extremely well on half of the classes and completely fails on the other half. Aggregated metrics hide this asymmetry.

Two models can have similar overall accuracy while representing fundamentally different worlds internally.

Quantifying the Effect: Hysteresis Loss

We can define a simple, order-dependent metric:

[\mathcal{L}_{\text{hyst}}(A) = |\text{Acc}_A(SAB) - \text{Acc}_A(SBA)|]

In this experiment:

Acc_A(SAB) ≈ 0.00
Acc_A(SBA) ≈ 0.99

This yields a hysteresis loss close to 1.0, i.e. the maximum possible difference.

The same holds symmetrically for dataset B.

Hysteresis summary

Description: Bar chart showing absolute accuracy differences between SAB and SBA.
Hysteresis for the individual subsets A and B is near-maximal, while hysteresis in the aggregate (full test accuracy) is close to zero and visually compressed due to the shared scale.

This highlights a key point: global performance metrics can remain almost invariant, even when class-conditional behavior is maximally path-dependent.

What This Means

This result shows that training order is not just an optimization detail.

The network does not converge to a single, order-independent solution
Learning leaves path-dependent traces in weight space
Once the model commits to one subset, returning to a balanced representation is not trivial

This behavior is strongly reminiscent of hysteresis in physical systems, where the final state depends on the path taken, not only on the endpoint.

What This Post Does Not Explain (Yet)

This post only shows that hysteresis exists.

It does not explain:

Whether the two final models lie in different basins of attraction
Whether their internal representations are aligned or incompatible
Whether one can smoothly interpolate between them without loss spikes

These questions require looking at the geometry of the weight space, not just accuracy curves.

That is exactly what Part 2 will address.

Reproducibility

All experiments were run with a fixed configuration and deterministic setup. The code used for this post (FAZ 1) is available on GitHub:

Hysteresis in Neural Networks

The geometric analysis (weight trajectories, representation similarity, interpolation barriers) will be released together with the next post.

Closing Thoughts

If training order alone can erase entire subsets of knowledge, then:

Curriculum learning has hidden costs
"Same data" does not imply "same model"
Optimization is not just minimization — it is a history-dependent process

In the next post, we will open the model and examine how these irreversible choices are encoded in the geometry of neural manifolds.

Part 2: Inside the Weight Space — Geometry of Hysteresis

Links

Linkedin: Linkedin
Github: Github
Website: Website